Running cluster checkups

OKD Virtualization 4.11 includes a diagnostic framework to run predefined checkups that can be used for cluster maintenance and troubleshooting.

The OKD cluster checkup framework is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

About the OKD cluster checkup framework

A checkup is an automated test workload that allows you to verify if a specific cluster functionality works as expected. The cluster checkup framework uses native Kubernetes resources to configure and execute the checkup.

By using predefined checkups, cluster administrators can improve cluster maintainability, troubleshoot unexpected behavior, minimize errors, and save time. They can also review the results of the checkup and share them with experts for further analysis. Vendors can write and publish checkups for features or services that they provide and verify that their customer environments are configured correctly.

Running a predefined checkup in the cluster involves setting up the namespace and service account for the framework, creating the ClusterRole and ClusterRoleBinding objects for the service account, enabling permissions for the checkup, and creating the input config map and the checkup job. You can run a checkup multiple times.

Checking network connectivity and latency for virtual machines on a secondary network

As a cluster administrator, you use a predefined checkup to verify network connectivity and measure latency between virtual machines (VMs) that are attached to a secondary network interface.

To run a checkup for the first time, follow the steps in the procedure.

If you have previously run a checkup, skip to step 5 of the procedure because the steps to install the framework and enable permissions for the checkup are not required.

Prerequisites

  • You installed the OpenShift CLI (oc).

  • You logged in to the cluster as a user with the cluster-admin role.

  • The cluster has at least two worker nodes.

  • The Multus Container Network Interface (CNI) plug-in is installed on the cluster.

  • You configured a network attachment definition for a namespace.

Procedure

  1. Create a configuration file that contains the resources to set up the framework. This includes a namespace and service account for the framework, and the ClusterRole and ClusterRoleBinding objects to define permissions for the service account.

    Example framework manifest file

    1. ---
    2. apiVersion: v1
    3. kind: Namespace
    4. metadata:
    5. name: kiagnose
    6. ---
    7. apiVersion: v1
    8. kind: ServiceAccount
    9. metadata:
    10. name: kiagnose
    11. namespace: kiagnose
    12. ---
    13. apiVersion: rbac.authorization.k8s.io/v1
    14. kind: ClusterRole
    15. metadata:
    16. name: kiagnose
    17. rules:
    18. - apiGroups: [ "" ]
    19. resources: [ "configmaps" ]
    20. verbs:
    21. - get
    22. - list
    23. - create
    24. - update
    25. - patch
    26. - apiGroups: [ "" ]
    27. resources: [ "namespaces" ]
    28. verbs:
    29. - get
    30. - list
    31. - create
    32. - delete
    33. - watch
    34. - apiGroups: [ "" ]
    35. resources: [ "serviceaccounts" ]
    36. verbs:
    37. - get
    38. - list
    39. - create
    40. - apiGroups: [ "rbac.authorization.k8s.io" ]
    41. resources:
    42. - roles
    43. - rolebindings
    44. - clusterrolebindings
    45. verbs:
    46. - get
    47. - list
    48. - create
    49. - delete
    50. - apiGroups: [ "rbac.authorization.k8s.io" ]
    51. resources:
    52. - clusterroles
    53. verbs:
    54. - get
    55. - list
    56. - create
    57. - bind
    58. - apiGroups: [ "batch" ]
    59. resources: [ "jobs" ]
    60. verbs:
    61. - get
    62. - list
    63. - create
    64. - delete
    65. - watch
    66. ---
    67. apiVersion: rbac.authorization.k8s.io/v1
    68. kind: ClusterRoleBinding
    69. metadata:
    70. name: kiagnose
    71. roleRef:
    72. apiGroup: rbac.authorization.k8s.io
    73. kind: ClusterRole
    74. name: kiagnose
    75. subjects:
    76. - kind: ServiceAccount
    77. name: kiagnose
    78. namespace: kiagnose
    79. ...
  2. Apply the framework manifest:

    1. $ oc apply -f <framework_manifest>.yaml
  3. Create a configuration file that contains the ClusterRole and Role objects with permissions that the checkup requires for cluster access:

    Example cluster role manifest file

    1. apiVersion: rbac.authorization.k8s.io/v1
    2. kind: ClusterRole
    3. metadata:
    4. name: kubevirt-vm-latency-checker
    5. rules:
    6. - apiGroups: ["kubevirt.io"]
    7. resources: ["virtualmachineinstances"]
    8. verbs: ["get", "create", "delete"]
    9. - apiGroups: ["subresources.kubevirt.io"]
    10. resources: ["virtualmachineinstances/console"]
    11. verbs: ["get"]
    12. - apiGroups: ["k8s.cni.cncf.io"]
    13. resources: ["network-attachment-definitions"]
    14. verbs: ["get"]
  4. Apply the checkup roles manifest:

    1. $ oc apply -f <latency_roles>.yaml
  5. Create a ConfigMap manifest that contains the input parameters for the checkup. The config map provides the input for the framework to run the checkup and also stores the results of the checkup.

    Example input config map

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4. name: kubevirt-vm-latency-checkup
    5. namespace: kiagnose
    6. data:
    7. spec.image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup:v4.11.0
    8. spec.timeout: 10m
    9. spec.clusterRoles: |
    10. kubevirt-vmis-manager
    11. spec.param.network_attachment_definition_namespace: "default" (1)
    12. spec.param.network_attachment_definition_name: "bridge-network" (2)
    13. spec.param.max_desired_latency_milliseconds: "10" (3)
    14. spec.param.sample_duration_seconds: "5" (4)
    1The namespace where the NetworkAttachmentDefinition object resides.
    2The name of the NetworkAttachmentDefinition object.
    3Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the check fails.
    4Optional: The duration of the latency check, in seconds.
  6. Create the config map in the framework’s namespace:

    1. $ oc apply -f <latency_config_map>.yaml
  7. Create a Job object to run the checkup:

    Example job manifest

    1. apiVersion: batch/v1
    2. kind: Job
    3. metadata:
    4. name: kubevirt-vm-latency-checkup
    5. namespace: kiagnose
    6. spec:
    7. backoffLimit: 0
    8. template:
    9. spec:
    10. serviceAccount: kiagnose
    11. restartPolicy: Never
    12. containers:
    13. - name: framework
    14. image: registry.redhat.io/container-native-virtualization/checkup-framework:v4.11.0
    15. env:
    16. - name: CONFIGMAP_NAMESPACE
    17. value: kiagnose
    18. - name: CONFIGMAP_NAME
    19. value: kubevirt-vm-latency-checkup
  8. Apply the Job manifest. The checkup uses the ping utility to verify connectivity and measure latency.

    1. $ oc apply -f <latency_job>.yaml
  9. Wait for the job to complete:

    1. $ oc wait --for=condition=complete --timeout=10m job.batch/kubevirt-vm-latency-checkup -n kiagnose
  10. Review the results of the latency checkup by retrieving the status of the ConfigMap object. If the measured latency is greater than the value of the spec.param.max_desired_latency_milliseconds attribute, the checkup fails and returns an error.

    1. $ oc get configmap kubevirt-vm-latency-checkup -n kiagnose -o yaml

    Example output config map (success)

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4. name: kubevirt-vm-latency-checkup
    5. namespace: kiagnose
    6. ...
    7. status.succeeded: "true"
    8. status.failureReason: ""
    9. status.result.minLatencyNanoSec: 2000
    10. status.result.maxLatencyNanoSec: 3000
    11. status.result.avgLatencyNanoSec: 2500
    12. status.results.measurementDurationSec: 300
    13. ...
  11. Delete the framework and checkup resources that you previously created. This includes the job, config map, cluster role, and framework manifest files.

    Do not delete the framework and cluster role manifest files if you plan to run another checkup.

    1. $ oc delete -f <file_name>.yaml

Additional resources