Kernel Module Management Operator

Learn about the Kernel Module Management (KMM) Operator and how you can use it to deploy out-of-tree kernel modules and device plugins on OKD clusters.

About the Kernel Module Management Operator

The Kernel Module Management (KMM) Operator manages, builds, signs, and deploys out-of-tree kernel modules and device plugins on OKD clusters.

KMM adds a new Module CRD which describes an out-of-tree kernel module and its associated device plugin. You can use Module resources to configure how to load the module, define ModuleLoader images for kernel versions, and include instructions for building and signing modules for specific kernel versions.

KMM is designed to accommodate multiple kernel versions at once for any kernel module, allowing for seamless node upgrades and reduced application downtime.

Installing the Kernel Module Management Operator

As a cluster administrator, you can install the Kernel Module Management (KMM) Operator by using the OpenShift CLI or the web console.

The KMM Operator is supported on OKD 4.12 and later. Installing KMM on version 4.11 does not require specific additional steps. For details on installing KMM on version 4.10 and earlier, see the section “Installing the Kernel Module Management Operator on earlier versions of OKD”.

Installing the Kernel Module Management Operator using the web console

As a cluster administrator, you can install the Kernel Module Management (KMM) Operator using the OKD web console.

Procedure

  1. Log in to the OKD web console.

  2. Install the Kernel Module Management Operator:

    1. In the OKD web console, click Operators → OperatorHub.

    2. Select Kernel Module Management Operator from the list of available Operators, and then click Install.

    3. From the Installed Namespace list, select the openshift-kmm namespace.

    4. Click Install.

Verification

To verify that KMM Operator installed successfully:

  1. Navigate to the Operators → Installed Operators page.

  2. Ensure that Kernel Module Management Operator is listed in the openshift-kmm project with a Status of InstallSucceeded.

    During installation, an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Troubleshooting

  1. To troubleshoot issues with Operator installation:

    1. Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.

    2. Navigate to the Workloads → Pods page and check the logs for pods in the openshift-kmm project.

Installing the Kernel Module Management Operator by using the CLI

As a cluster administrator, you can install the Kernel Module Management (KMM) Operator by using the OpenShift CLI.

Prerequisites

  • You have a running OKD cluster.

  • You installed the OpenShift CLI (oc).

  • You are logged into the OpenShift CLI as a user with cluster-admin privileges.

Procedure

  1. Install KMM in the openshift-kmm namespace:

    1. Create the following Namespace CR and save the YAML file, for example, kmm-namespace.yaml:

      1. apiVersion: v1
      2. kind: Namespace
      3. metadata:
      4.   name: openshift-kmm

    2. Create the following OperatorGroup CR and save the YAML file, for example, kmm-op-group.yaml:

      1. apiVersion: operators.coreos.com/v1
      2. kind: OperatorGroup
      3. metadata:
      4.   name: kernel-module-management
      5.   namespace: openshift-kmm

    3. Create the following Subscription CR and save the YAML file, for example, kmm-sub.yaml:

      1. apiVersion: operators.coreos.com/v1alpha1
      2. kind: Subscription
      3. metadata:
      4.   name: kernel-module-management
      5.   namespace: openshift-kmm
      6. spec:
      7.   channel: release-1.0
      8.   installPlanApproval: Automatic
      9.   name: kernel-module-management
      10.   source: redhat-operators
      11.   sourceNamespace: openshift-marketplace
      12.   startingCSV: kernel-module-management.v1.0.0

    4. Create the subscription object by running the following command:

      1. $ oc create -f kmm-sub.yaml

Verification

  • To verify that the Operator deployment is successful, run the following command:

    1. $ oc get -n openshift-kmm deployments.apps kmm-operator-controller-manager

    Example output

    1. NAME                              READY UP-TO-DATE  AVAILABLE AGE
    2. kmm-operator-controller-manager   1/1   1           1         97s

    The Operator is available.

Installing the Kernel Module Management Operator on earlier versions of OKD

The KMM Operator is supported on OKD 4.12 and later. For version 4.10 and earlier, you must create a new SecurityContextConstraint object and bind it to the Operator’s ServiceAccount. As a cluster administrator, you can install the Kernel Module Management (KMM) Operator by using the OpenShift CLI.

Prerequisites

  • You have a running OKD cluster.

  • You installed the OpenShift CLI (oc).

  • You are logged into the OpenShift CLI as a user with cluster-admin privileges.

Procedure

  1. Install KMM in the openshift-kmm namespace:

    1. Create the following Namespace CR and save the YAML file, for example, kmm-namespace.yaml file:

      1. apiVersion: v1
      2. kind: Namespace
      3. metadata:
      4.   name: openshift-kmm

    2. Create the following SecurityContextConstraint object and save the YAML file, for example, kmm-security-constraint.yaml:

      1. allowHostDirVolumePlugin: false
      2. allowHostIPC: false
      3. allowHostNetwork: false
      4. allowHostPID: false
      5. allowHostPorts: false
      6. allowPrivilegeEscalation: false
      7. allowPrivilegedContainer: false
      8. allowedCapabilities:
      9.   - NET_BIND_SERVICE
      10. apiVersion: security.openshift.io/v1
      11. defaultAddCapabilities: null
      12. fsGroup:
      13.   type: MustRunAs
      14. groups: []
      15. kind: SecurityContextConstraints
      16. metadata:
      17.   name: restricted-v2
      18. priority: null
      19. readOnlyRootFilesystem: false
      20. requiredDropCapabilities:
      21.   - ALL
      22. runAsUser:
      23.   type: MustRunAsRange
      24. seLinuxContext:
      25.   type: MustRunAs
      26. seccompProfiles:
      27.   - runtime/default
      28. supplementalGroups:
      29.   type: RunAsAny
      30. users: []
      31. volumes:
      32.   - configMap
      33.   - downwardAPI
      34.   - emptyDir
      35.   - persistentVolumeClaim
      36.   - projected
      37.   - secret

    3. Bind the SecurityContextConstraint object to the Operator’s ServiceAccount by running the following commands:

      1. $ oc apply -f kmm-security-constraint.yaml
      1. $ oc adm policy add-scc-to-user kmm-security-constraint -z kmm-operator-controller-manager -n openshift-kmm

    4. Create the following OperatorGroup CR and save the YAML file, for example, kmm-op-group.yaml:

      1. apiVersion: operators.coreos.com/v1
      2. kind: OperatorGroup
      3. metadata:
      4.   name: kernel-module-management
      5.   namespace: openshift-kmm

    5. Create the following Subscription CR and save the YAML file, for example, kmm-sub.yaml:

      1. apiVersion: operators.coreos.com/v1alpha1
      2. kind: Subscription
      3. metadata:
      4.   name: kernel-module-management
      5.   namespace: openshift-kmm
      6. spec:
      7.   channel: release-1.0
      8.   installPlanApproval: Automatic
      9.   name: kernel-module-management
      10.   source: redhat-operators
      11.   sourceNamespace: openshift-marketplace
      12.   startingCSV: kernel-module-management.v1.0.0

    6. Create the subscription object by running the following command:

      1. $ oc create -f kmm-sub.yaml

Verification

  • To verify that the Operator deployment is successful, run the following command:

    1. $ oc get -n openshift-kmm deployments.apps kmm-operator-controller-manager

    Example output

    1. NAME                              READY UP-TO-DATE  AVAILABLE AGE
    2. kmm-operator-controller-manager   1/1   1           1         97s

    The Operator is available.

Uninstalling the Kernel Module Management Operator

Use one of the following procedures to uninstall the Kernel Module Management (KMM) Operator, depending on how the KMM Operator was installed.

Uninstalling a Red Hat catalog installation

Use this procedure if KMM was installed from the Red Hat catalog.

Procedure

Use the following method to uninstall the KMM Operator:

  • Use the OpenShift console under Operators -→ Installed Operators to locate and uninstall the Operator.

Alternatively, you can delete the Subscription resource in the KMM namespace.

Uninstalling a CLI installation

Use this command if the KMM Operator was installed using the OpenShift CLI.

Procedure

  • Run the following command to uninstall the KMM Operator:

    1. $ oc delete -k https://github.com/rh-ecosystem-edge/kernel-module-management/config/default

    Using this command deletes the Module CRD and all Module instances in the cluster.

Kernel module deployment

For each Module resource, Kernel Module Management (KMM) can create a number of DaemonSet resources:

  • One ModuleLoader DaemonSet per compatible kernel version running in the cluster.

  • One device plugin DaemonSet, if configured.

The module loader daemon set resources run ModuleLoader images to load kernel modules. A module loader image is an OCI image that contains the .ko files and both the modprobe and sleep binaries.

When the module loader pod is created, the pod runs modprobe to insert the specified module into the kernel. It then enters a sleep state until it is terminated. When that happens, the ExecPreStop hook runs modprobe -r to unload the kernel module.

If the .spec.devicePlugin attribute is configured in a Module resource, then KMM creates a device plugin daemon set in the cluster. That daemon set targets:

  • Nodes that match the .spec.selector of the Module resource.

  • Nodes with the kernel module loaded (where the module loader pod is in the Ready condition).

The Module custom resource definition

The Module custom resource definition (CRD) represents a kernel module that can be loaded on all or select nodes in the cluster, through a module loader image. A Module custom resource (CR) specifies one or more kernel versions with which it is compatible, and a node selector.

The compatible versions for a Module resource are listed under .spec.moduleLoader.container.kernelMappings. A kernel mapping can either match a literal version, or use regexp to match many of them at the same time.

The reconciliation loop for the Module resource runs the following steps:

  1. List all nodes matching .spec.selector.

  2. Build a set of all kernel versions running on those nodes.

  3. For each kernel version:

    1. Go through .spec.moduleLoader.container.kernelMappings and find the appropriate container image name. If the kernel mapping has build or sign defined and the container image does not already exist, run the build, the signing job, or both, as needed.

    2. Create a module loader daemon set with the container image determined in the previous step.

    3. If .spec.devicePlugin is defined, create a device plugin daemon set using the configuration specified under .spec.devicePlugin.container.

  4. Run garbage-collect on:

    1. Existing daemon set resources targeting kernel versions that are not run by any node in the cluster.

    2. Successful build jobs.

    3. Successful signing jobs.

Set soft dependencies between kernel modules

Some configurations require that several kernel modules be loaded in a specific order to work properly, even though the modules do not directly depend on each other through symbols. These are called soft dependencies. depmod is usually not aware of these dependencies, and they do not appear in the files it produces. For example, if mod_a has a soft dependency on mod_b, modprobe mod_a will not load mod_b.

You can resolve these situations by declaring soft dependencies in the Module Custom Resource Definition (CRD) using the modulesLoadingOrder field.

  1. # ...
  2. spec:
  3.   moduleLoader:
  4.     container:
  5.       modprobe:
  6.         moduleName: mod_a
  7.         dirName: /opt
  8.         firmwarePath: /firmware
  9.         parameters:
  10.           - param=1
  11.         modulesLoadingOrder:
  12.           - mod_a
  13.           - mod_b

In the configuration above:

  • The loading order is mod_b, then mod_a.

  • The unloading order is mod_a, then mod_b.

The first value in the list, to be loaded last, must be equivalent to the moduleName.

Security and permissions

Loading kernel modules is a highly sensitive operation. After they are loaded, kernel modules have all possible permissions to do any kind of operation on the node.

ServiceAccounts and SecurityContextConstraints

Kernel Module Management (KMM) creates a privileged workload to load the kernel modules on nodes. That workload needs ServiceAccounts allowed to use the privileged SecurityContextConstraint (SCC) resource.

The authorization model for that workload depends on the namespace of the Module resource, as well as its spec.

  • If the .spec.moduleLoader.serviceAccountName or .spec.devicePlugin.serviceAccountName fields are set, they are always used.

  • If those fields are not set, then:

    • If the Module resource is created in the operator’s namespace (openshift-kmm by default), then KMM uses its default, powerful ServiceAccounts to run the daemon sets.

    • If the Module resource is created in any other namespace, then KMM runs the daemon sets as the namespace’s default ServiceAccount. The Module resource cannot run a privileged workload unless you manually enable it to use the privileged SCC.

openshift-kmm is a trusted namespace.

When setting up RBAC permissions, remember that any user or ServiceAccount creating a Module resource in the openshift-kmm namespace results in KMM automatically running privileged workloads on potentially all nodes in the cluster.

To allow any ServiceAccount to use the privileged SCC and therefore to run module loader or device plugin pods, use the following command:

  1. $ oc adm policy add-scc-to-user privileged -z "${serviceAccountName}" [ -n "${namespace}" ]

Pod security standards

OpenShift runs a synchronization mechanism that sets the namespace Pod Security level automatically based on the security contexts in use. No action is needed.

Additional resources

Replacing in-tree modules with out-of-tree modules

You can use Kernel Module Management (KMM) to build kernel modules that can be loaded or unloaded into the kernel on demand. These modules extend the functionality of the kernel without the need to reboot the system. Modules can be configured as built-in or dynamically loaded.

Dynamically loaded modules include in-tree modules and out-of-tree (OOT) modules. In-tree modules are internal to the Linux kernel tree, that is, they are already part of the kernel. Out-of-tree modules are external to the Linux kernel tree. They are generally written for development and testing purposes, such as testing the new version of a kernel module that is shipped in-tree, or to deal with incompatibilities.

Some modules loaded by KMM could replace in-tree modules already loaded on the node. To unload an in-tree module before loading your module, set the .spec.moduleLoader.container.inTreeModuleToRemove field. The following is an example for module replacement for all kernel mappings:

  1. # ...
  2. spec:
  3.   moduleLoader:
  4.     container:
  5.       modprobe:
  6.         moduleName: mod_a
  8.       inTreeModuleToRemove: mod_b

In this example, the moduleLoader pod uses inTreeModuleToRemove to unload the in-tree mod_b before loading mod_a from the moduleLoader image. When the moduleLoader`pod is terminated and `mod_a is unloaded, mod_b is not loaded again.

The following is an example for module replacement for specific kernel mappings:

  1. # ...
  2. spec:
  3.   moduleLoader:
  4.     container:
  5.       kernelMappings:
  6.         - literal: 6.0.15-300.fc37.x86_64
  7.           containerImage: some.registry/org/my-kmod:6.0.15-300.fc37.x86_64
  8.           inTreeModuleToRemove: <module_name>

Additional resources

Example Module CR

The following is an annotated Module example:

  1. apiVersion: kmm.sigs.x-k8s.io/v1beta1
  2. kind: Module
  3. metadata:
  4.   name: <my_kmod>
  5. spec:
  6.   moduleLoader:
  7.     container:
  8.       modprobe:
  9.         moduleName: <my_kmod> (1)
  10.         dirName: /opt (2)
  11.         firmwarePath: /firmware (3)
  12.         parameters:  (4)
  13.           - param=1
  14.       kernelMappings:  (5)
  15.         - literal: 6.0.15-300.fc37.x86_64
  16.           containerImage: some.registry/org/my-kmod:6.0.15-300.fc37.x86_64
  17.         - regexp: '^.+\fc37\.x86_64$' (6)
  18.           containerImage: "some.other.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
  19.         - regexp: '^.+$' (7)
  20.           containerImage: "some.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
  21.           build:
  22.             buildArgs:  (8)
  23.               - name: ARG_NAME
  24.                 value: <some_value>
  25.             secrets:
  26.               - name: <some_kubernetes_secret>  (9)
  27.             baseImageRegistryTLS: (10)
  28.               insecure: false
  29.               insecureSkipTLSVerify: false (11)
  30.             dockerfileConfigMap:  (12)
  31.               name: <my_kmod_dockerfile>
  32.           sign:
  33.             certSecret:
  34.               name: <cert_secret>  (13)
  35.             keySecret:
  36.               name: <key_secret>  (14)
  37.             filesToSign:
  38.               - /opt/lib/modules/${KERNEL_FULL_VERSION}/<my_kmod>.ko
  39.           registryTLS: (15)
  40.             insecure: false (16)
  41.             insecureSkipTLSVerify: false
  42.     serviceAccountName: <sa_module_loader>  (17)
  43.   devicePlugin:  (18)
  44.     container:
  45.       image: some.registry/org/device-plugin:latest  (19)
  46.       env:
  47.         - name: MY_DEVICE_PLUGIN_ENV_VAR
  48.           value: SOME_VALUE
  49.       volumeMounts:  (20)
  50.         - mountPath: /some/mountPath
  51.           name: <device_plugin_volume>
  52.     volumes:  (21)
  53.       - name: <device_plugin_volume>
  54.         configMap:
  55.           name: <some_configmap>
  56.     serviceAccountName: <sa_device_plugin> (22)
  57.   imageRepoSecret:  (23)
  58.     name: <secret_name>
  59.   selector:
  60.     node-role.kubernetes.io/worker: ""
1Required.
2Optional.
3Optional: Copies /firmware/* into /var/lib/firmware/ on the node.
4Optional.
5At least one kernel item is required.
6For each node running a kernel matching the regular expression, KMM creates a DaemonSet resource running the image specified in containerImage with ${KERNEL_FULL_VERSION} replaced with the kernel version.
7For any other kernel, build the image using the Dockerfile in the my-kmod ConfigMap.
8Optional.
9Optional: A value for some-kubernetes-secret can be obtained from the build environment at /run/secrets/some-kubernetes-secret.
10Optional: Avoid using this parameter. If set to true, the build is allowed to pull the image in the Dockerfile FROM instruction using plain HTTP.
11Optional: Avoid using this parameter. If set to true, the build will skip any TLS server certificate validation when pulling the image in the Dockerfile FROM instruction using plain HTTP.
12Required.
13Required: A secret holding the public secureboot key with the key ‘cert’.
14Required: A secret holding the private secureboot key with the key ‘key’.
15Optional: Avoid using this parameter. If set to true, KMM will be allowed to check if the container image already exists using plain HTTP.
16Optional: Avoid using this parameter. If set to true, KMM will skip any TLS server certificate validation when checking if the container image already exists.
17Optional.
18Optional.
19Required: If the device plugin section is present.
20Optional.
21Optional.
22Optional.
23Optional: Used to pull module loader and device plugin images.

Using a ModuleLoader image

Kernel Module Management (KMM) works with purpose-built module loader images. These are standard OCI images that must satisfy the following requirements:

  • .ko files must be located in /opt/lib/modules/${KERNEL_VERSION}.

  • modprobe and sleep binaries must be defined in the $PATH variable.

Running depmod

If your module loader image contains several kernel modules and if one of the modules depends on another module, it is best practice to run depmod at the end of the build process to generate dependencies and map files.

You must have a Red Hat subscription to download the kernel-devel package.

Procedure

  1. To generate modules.dep and .map files for a specific kernel version, run depmod -b /opt ${KERNEL_VERSION}.

Example Dockerfile

If you are building your image on OKD, consider using the Driver Tool Kit (DTK).

For further information, see using an entitled build.

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: kmm-ci-dockerfile
  5. data:
  6.   dockerfile: |
  7.     ARG DTK_AUTO
  8.     FROM ${DTK_AUTO} as builder
  9.     ARG KERNEL_VERSION
  10.     WORKDIR /usr/src
  11.     RUN ["git", "clone", "https://github.com/rh-ecosystem-edge/kernel-module-management.git"]
  12.     WORKDIR /usr/src/kernel-module-management/ci/kmm-kmod
  13.     RUN KERNEL_SRC_DIR=/lib/modules/${KERNEL_VERSION}/build make all
  14.     FROM registry.redhat.io/ubi9/ubi-minimal
  15.     ARG KERNEL_VERSION
  16.     RUN microdnf install kmod
  17.     COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_a.ko /opt/lib/modules/${KERNEL_VERSION}/
  18.     COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_b.ko /opt/lib/modules/${KERNEL_VERSION}/
  19.     RUN depmod -b /opt ${KERNEL_VERSION}

Additional resources

Building in the cluster

KMM can build module loader images in the cluster. Follow these guidelines:

  • Provide build instructions using the build section of a kernel mapping.

  • Copy the Dockerfile for your container image into a ConfigMap resource, under the dockerfile key.

  • Ensure that the ConfigMap is located in the same namespace as the Module.

KMM checks if the image name specified in the containerImage field exists. If it does, the build is skipped.

Otherwise, KMM creates a Build resource to build your image. After the image is built, KMM proceeds with the Module reconciliation. See the following example.

  1. # ...
  2. - regexp: '^.+$'
  3.   containerImage: "some.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
  4.   build:
  5.     buildArgs:  (1)
  6.       - name: ARG_NAME
  7.         value: <some_value>
  8.     secrets: (2)
  9.       - name: <some_kubernetes_secret> (3)
  10.     baseImageRegistryTLS:
  11.       insecure: false (4)
  12.       insecureSkipTLSVerify: false (5)
  13.     dockerfileConfigMap:  (6)
  14.       name: <my_kmod_dockerfile>
  15.   registryTLS:
  16.     insecure: false (7)
  17.     insecureSkipTLSVerify: false (8)
1Optional.
2Optional.
3Will be mounted in the build pod as /run/secrets/some-kubernetes-secret.
4Optional: Avoid using this parameter. If set to true, the build will be allowed to pull the image in the Dockerfile FROM instruction using plain HTTP.
5Optional: Avoid using this parameter. If set to true, the build will skip any TLS server certificate validation when pulling the image in the Dockerfile FROM instruction using plain HTTP.
6Required.
7Optional: Avoid using this parameter. If set to true, KMM will be allowed to check if the container image already exists using plain HTTP.
8Optional: Avoid using this parameter. If set to true, KMM will skip any TLS server certificate validation when checking if the container image already exists.

Additional resources

Using the Driver Toolkit

The Driver Toolkit (DTK) is a convenient base image for building build module loader images. It contains tools and libraries for the OpenShift version currently running in the cluster.

Procedure

Use DTK as the first stage of a multi-stage Dockerfile.

  1. Build the kernel modules.

  2. Copy the .ko files into a smaller end-user image such as ubi-minimal.

  3. To leverage DTK in your in-cluster build, use the DTK_AUTO build argument. The value is automatically set by KMM when creating the Build resource. See the following example.

    1. ARG DTK_AUTO
    2. FROM ${DTK_AUTO} as builder
    3. ARG KERNEL_VERSION
    4. WORKDIR /usr/src
    5. RUN ["git", "clone", "https://github.com/rh-ecosystem-edge/kernel-module-management.git"]
    6. WORKDIR /usr/src/kernel-module-management/ci/kmm-kmod
    7. RUN KERNEL_SRC_DIR=/lib/modules/${KERNEL_VERSION}/build make all
    8. FROM registry.redhat.io/ubi9/ubi-minimal
    9. ARG KERNEL_VERSION
    10. RUN microdnf install kmod
    11. COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_a.ko /opt/lib/modules/${KERNEL_VERSION}/
    12. COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_b.ko /opt/lib/modules/${KERNEL_VERSION}/
    13. RUN depmod -b /opt ${KERNEL_VERSION}

Additional resources

Using signing with Kernel Module Management (KMM)

On a Secure Boot enabled system, all kernel modules (kmods) must be signed with a public/private key-pair enrolled into the Machine Owner’s Key (MOK) database. Drivers distributed as part of a distribution should already be signed by the distribution’s private key, but for kernel modules build out-of-tree, KMM supports signing kernel modules using the sign section of the kernel mapping.

For more details on using Secure Boot, see Generating a public and private key pair

Prerequisites

  • A public private key pair in the correct (DER) format.

  • At least one secure-boot enabled node with the public key enrolled in its MOK database.

  • Either a pre-built driver container image, or the source code and Dockerfile needed to build one in-cluster.

Adding the keys for secureboot

To use KMM Kernel Module Management (KMM) to sign kernel modules, a certificate and private key are required. For details on how to create these, see Generating a public and private key pair.

For details on how to extract the public and private key pair, see Signing kernel modules with the private key. Use steps 1 through 4 to extract the keys into files.

Procedure

  1. Create the sb_cert.cer file that contains the certificate and the sb_cert.priv file that contains the private key:

    1. $ openssl req -x509 -new -nodes -utf8 -sha256 -days 36500 -batch -config configuration_file.config -outform DER -out my_signing_key_pub.der -keyout my_signing_key.priv

  2. Add the files by using one of the following methods:

    • Add the files as secrets directly:

      1. $ oc create secret generic my-signing-key --from-file=key=<my_signing_key.priv>
      1. $ oc create secret generic my-signing-key-pub --from-file=key=<my_signing_key_pub.der>

    • Add the files by base64 encoding them:

      1. $ cat sb_cert.priv | base64 -w 0 > my_signing_key2.base64
      1. $ cat sb_cert.cer | base64 -w 0 > my_signing_key_pub.base64

  3. Add the encoded text to a YAML file:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: my-signing-key-pub
    5.   namespace: default (1)
    6. type: Opaque
    7. data:
    8.   cert: <base64_encoded_secureboot_public_key>
    10. ---
    11. apiVersion: v1
    12. kind: Secret
    13. metadata:
    14.   name: my-signing-key
    15.   namespace: default (1)
    16. type: Opaque
    17. data:
    18.   key: <base64_encoded_secureboot_private_key>
    1namespace - Replace default with a valid namespace.

  4. Apply the YAML file:

    1. $ oc apply -f <yaml_filename>

Checking the keys

After you have added the keys, you must check them to ensure they are set correctly.

Procedure

  1. Check to ensure the public key secret is set correctly:

    1. $ oc get secret -o yaml <certificate secret name> | awk '/cert/{print $2; exit}' | base64 -d  | openssl x509 -inform der -text

    This should display a certificate with a Serial Number, Issuer, Subject, and more.

  2. Check to ensure the private key secret is set correctly:

    1. $ oc get secret -o yaml <private key secret name> | awk '/key/{print $2; exit}' | base64 -d

    This should display the key enclosed in the -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- lines.

Signing a pre-built driver container

Use this procedure if you have a pre-built image, such as an image either distributed by a hardware vendor or built elsewhere.

The following YAML file adds the public/private key-pair as secrets with the required key names - key for the private key, cert for the public key. The cluster then pulls down the unsignedImage image, opens it, signs the kernel modules listed in filesToSign, adds them back, and pushes the resulting image as containerImage.

Kernel Module Management (KMM) should then deploy the DaemonSet that loads the signed kmods onto all the nodes that match the selector. The driver containers should run successfully on any nodes that have the public key in their MOK database, and any nodes that are not secure-boot enabled, which ignore the signature. They should fail to load on any that have secure-boot enabled but do not have that key in their MOK database.

Prerequisites

  • The keySecret and certSecret secrets have been created.

Procedure

  1. Apply the YAML file:

    1. ---
    2. apiVersion: kmm.sigs.x-k8s.io/v1beta1
    3. kind: Module
    4. metadata:
    5.   name: example-module
    6. spec:
    7.   moduleLoader:
    8.     serviceAccountName: default
    9.     container:
    10.       modprobe: (1)
    11.         moduleName: '<your module name>'
    12.       kernelMappings:
    13.         # the kmods will be deployed on all nodes in the cluster with a kernel that matches the regexp
    14.         - regexp: '^.*\.x86_64$'
    15.           # the container to produce containing the signed kmods
    16.           containerImage: <image name e.g. quay.io/myuser/my-driver:<kernelversion>-signed>
    17.           sign:
    18.             # the image containing the unsigned kmods (we need this because we are not building the kmods within the cluster)
    19.             unsignedImage: <image name e.g. quay.io/myuser/my-driver:<kernelversion> >
    20.             keySecret: # a secret holding the private secureboot key with the key 'key'
    21.               name: <private key secret name>
    22.             certSecret: # a secret holding the public secureboot key with the key 'cert'
    23.               name: <certificate secret name>
    24.             filesToSign: # full path within the unsignedImage container to the kmod(s) to sign
    25.               - /opt/lib/modules/4.18.0-348.2.1.el8_5.x86_64/kmm_ci_a.ko
    26.   imageRepoSecret:
    27.     # the name of a secret containing credentials to pull unsignedImage and push containerImage to the registry
    28.     name: repo-pull-secret
    29.   selector:
    30.     kubernetes.io/arch: amd64
1modprobe - The name of the kmod to load.

Building and signing a ModuleLoader container image

Use this procedure if you have source code and must build your image first.

The following YAML file builds a new container image using the source code from the repository. The image produced is saved back in the registry with a temporary name, and this temporary image is then signed using the parameters in the sign section.

The temporary image name is based on the final image name and is set to be <containerImage>:<tag>-<namespace>_<module name>_kmm_unsigned.

For example, using the following YAML file, Kernel Module Management (KMM) builds an image named example.org/repository/minimal-driver:final-default_example-module_kmm_unsigned containing the build with unsigned kmods and push it to the registry. Then it creates a second image named example.org/repository/minimal-driver:final that contains the signed kmods. It is this second image that is loaded by the DaemonSet object and deploys the kmods to the cluster nodes.

After it is signed, the temporary image can be safely deleted from the registry. It will be rebuilt, if needed.

Prerequisites

  • The keySecret and certSecret secrets have been created.

Procedure

  1. Apply the YAML file:

    1. ---
    2. apiVersion: v1
    3. kind: ConfigMap
    4. metadata:
    5.   name: example-module-dockerfile
    6.   namespace: default (1)
    7. data:
    8.   Dockerfile: |
    9.     ARG DTK_AUTO
    10.     ARG KERNEL_VERSION
    11.     FROM ${DTK_AUTO} as builder
    12.     WORKDIR /build/
    13.     RUN git clone -b main --single-branch https://github.com/rh-ecosystem-edge/kernel-module-management.git
    14.     WORKDIR kernel-module-management/ci/kmm-kmod/
    15.     RUN make
    16.     FROM registry.access.redhat.com/ubi9/ubi:latest
    17.     ARG KERNEL_VERSION
    18.     RUN yum -y install kmod && yum clean all
    19.     RUN mkdir -p /opt/lib/modules/${KERNEL_VERSION}
    20.     COPY --from=builder /build/kernel-module-management/ci/kmm-kmod/*.ko /opt/lib/modules/${KERNEL_VERSION}/
    21.     RUN /usr/sbin/depmod -b /opt
    22. ---
    23. apiVersion: kmm.sigs.x-k8s.io/v1beta1
    24. kind: Module
    25. metadata:
    26.   name: example-module
    27.   namespace: default (1)
    28. spec:
    29.   moduleLoader:
    30.     serviceAccountName: default (2)
    31.     container:
    32.       modprobe:
    33.         moduleName: simple_kmod
    34.       kernelMappings:
    35.         - regexp: '^.*\.x86_64$'
    36.           containerImage: < the name of the final driver container to produce>
    37.           build:
    38.             dockerfileConfigMap:
    39.               name: example-module-dockerfile
    40.           sign:
    41.             keySecret:
    42.               name: <private key secret name>
    43.             certSecret:
    44.               name: <certificate secret name>
    45.             filesToSign:
    46.               - /opt/lib/modules/4.18.0-348.2.1.el8_5.x86_64/kmm_ci_a.ko
    47.   imageRepoSecret: (3)
    48.     name: repo-pull-secret
    49.   selector: # top-level selector
    50.     kubernetes.io/arch: amd64
1namespace - Replace default with a valid namespace.
2serviceAccountName - The default serviceAccountName does not have the required permissions to run a module that is privileged. For information on creating a service account, see “Creating service accounts” in the “Additional resources” of this section.
3imageRepoSecret - Used as imagePullSecrets in the DaemonSet object and to pull and push for the build and sign features.

Additional resources

For information on creating a service account, see Creating service accounts.

KMM hub and spoke

In hub and spoke scenarios, many spoke clusters are connected to a central, powerful hub cluster. Kernel Module Management (KMM) depends on Red Hat Advanced Cluster Management (RHACM) to operate in hub and spoke environments.

KMM is compatible with hub and spoke environments through decoupling KMM features. A ManagedClusterModule Custom Resource Definition (CRD) is provided to wrap the existing Module CRD and extend it to select Spoke clusters. Also provided is KMM-Hub, a new standalone controller that builds images and signs modules on the hub cluster.

In hub and spoke setups, spokes are focused, resource-constrained clusters that are centrally managed by a hub cluster. Spokes run the single-cluster edition of KMM, with those resource-intensive features disabled. To adapt KMM to this environment, you should reduce the workload running on the spokes to the minimum, while the hub takes care of the expensive tasks.

Building kernel module images and signing the .ko files, should run on the hub. The scheduling of the Module Loader and Device Plugin DaemonSets can only happen on the spokes.

Additional resources

KMM-Hub

The KMM project provides KMM-Hub, an edition of KMM dedicated to hub clusters. KMM-Hub monitors all kernel versions running on the spokes and determines the nodes on the cluster that should receive a kernel module.

KMM-Hub runs all compute-intensive tasks such as image builds and kmod signing, and prepares the trimmed-down Module to be transferred to the spokes through RHACM.

KMM-Hub cannot be used to load kernel modules on the hub cluster. Install the regular edition of KMM to load kernel modules.

Additional resources

Installing KMM-Hub

You can use one of the following methods to install KMM-Hub:

  • Using the Operator Lifecycle Manager (OLM)

  • Creating KMM resources

Additional resources

Installing KMM-Hub using the Operator Lifecycle Manager

Use the Operators section of the OpenShift console to install KMM-Hub.

Installing KMM-Hub by creating KMM resources

Procedure

  • If you want to install KMM-Hub programmatically, you can use the following resources to create the Namespace, OperatorGroup and Subscription resources:
  1. ---
  2. apiVersion: v1
  3. kind: Namespace
  4. metadata:
  5.   name: openshift-kmm-hub
  6. ---
  7. apiVersion: operators.coreos.com/v1
  8. kind: OperatorGroup
  9. metadata:
  10.   name: kernel-module-management-hub
  11.   namespace: openshift-kmm-hub
  12. ---
  13. apiVersion: operators.coreos.com/v1alpha1
  14. kind: Subscription
  15. metadata:
  16.   name: kernel-module-management-hub
  17.   namespace: openshift-kmm-hub
  18. spec:
  19.   channel: stable
  20.   installPlanApproval: Automatic
  21.   name: kernel-module-management-hub
  22.   source: redhat-operators
  23.   sourceNamespace: openshift-marketplace

Using the ManagedClusterModule CRD

Use the ManagedClusterModule Custom Resource Definition (CRD) to configure the deployment of kernel modules on spoke clusters. This CRD is cluster-scoped, wraps a Module spec and adds the following additional fields:

  1. apiVersion: hub.kmm.sigs.x-k8s.io/v1beta1
  2. kind: ManagedClusterModule
  3. metadata:
  4.   name: <my-mcm>
  5.   # No namespace, because this resource is cluster-scoped.
  6. spec:
  7.   moduleSpec: (1)
  8.     selector: (2)
  9.       node-wants-my-mcm: 'true'
  11.   spokeNamespace: <some-namespace> (3)
  13.   selector: (4)
  14.     wants-my-mcm: 'true'
1moduleSpec: Contains moduleLoader and devicePlugin sections, similar to a Module resource.
2Selects nodes within the ManagedCluster.
3Specifies in which namespace the Module should be created.
4Selects ManagedCluster objects.

If build or signing instructions are present in .spec.moduleSpec, those pods are run on the hub cluster in the operator’s namespace.

When the .spec.selector matches one or more ManagedCluster resources, then KMM-Hub creates a ManifestWork resource in the corresponding namespace(s). ManifestWork contains a trimmed-down Module resource, with kernel mappings preserved but all build and sign subsections are removed. containerImage fields that contain image names ending with a tag are replaced with their digest equivalent.

Running KMM on the spoke

After installing KMM on the spoke, no further action is required. Create a ManagedClusterModule object from the hub to deploy kernel modules on spoke clusters.

Procedure

You can install KMM on the spokes cluster through a RHACM Policy object. In addition to installing KMM from the Operator hub and running it in a lightweight spoke mode, the Policy configures additional RBAC required for the RHACM agent to be able to manage Module resources.

  • Use the following RHACM policy to install KMM on spoke clusters:

    1. ---
    2. apiVersion: policy.open-cluster-management.io/v1
    3. kind: Policy
    4. metadata:
    5.   name: install-kmm
    6. spec:
    7.   remediationAction: enforce
    8.   disabled: false
    9.   policy-templates:
    10.     - objectDefinition:
    11.         apiVersion: policy.open-cluster-management.io/v1
    12.         kind: ConfigurationPolicy
    13.         metadata:
    14.           name: install-kmm
    15.         spec:
    16.           severity: high
    17.           object-templates:
    18.           - complianceType: mustonlyhave
    19.             objectDefinition:
    20.               apiVersion: v1
    21.               kind: Namespace
    22.               metadata:
    23.                 name: openshift-kmm
    24.           - complianceType: mustonlyhave
    25.             objectDefinition:
    26.               apiVersion: operators.coreos.com/v1
    27.               kind: OperatorGroup
    28.               metadata:
    29.                 name: kmm
    30.                 namespace: openshift-kmm
    31.               spec:
    32.                 upgradeStrategy: Default
    33.           - complianceType: mustonlyhave
    34.             objectDefinition:
    35.               apiVersion: operators.coreos.com/v1alpha1
    36.               kind: Subscription
    37.               metadata:
    38.                 name: kernel-module-management
    39.                 namespace: openshift-kmm
    40.               spec:
    41.                 channel: stable
    42.                 config:
    43.                   env:
    44.                     - name: KMM_MANAGED
    45.                       value: "1"
    46.                 installPlanApproval: Automatic
    47.                 name: kernel-module-management
    48.                 source: redhat-operators
    49.                 sourceNamespace: openshift-marketplace
    50.           - complianceType: mustonlyhave
    51.             objectDefinition:
    52.               apiVersion: rbac.authorization.k8s.io/v1
    53.               kind: ClusterRole
    54.               metadata:
    55.                 name: kmm-module-manager
    56.               rules:
    57.                 - apiGroups: [kmm.sigs.x-k8s.io]
    58.                   resources: [modules]
    59.                   verbs: [create, delete, get, list, patch, update, watch]
    60.           - complianceType: mustonlyhave
    61.             objectDefinition:
    62.               apiVersion: rbac.authorization.k8s.io/v1
    63.               kind: ClusterRoleBinding
    64.               metadata:
    65.                 name: klusterlet-kmm
    66.               subjects:
    67.               - kind: ServiceAccount
    68.                 name: klusterlet-work-sa
    69.                 namespace: open-cluster-management-agent
    70.               roleRef:
    71.                 kind: ClusterRole
    72.                 name: kmm-module-manager
    73.                 apiGroup: rbac.authorization.k8s.io
    74. ---
    75. apiVersion: apps.open-cluster-management.io/v1
    76. kind: PlacementRule
    77. metadata:
    78.   name: all-managed-clusters
    79. spec:
    80.   clusterSelector: (1)
    81.     matchExpressions: []
    82. ---
    83. apiVersion: policy.open-cluster-management.io/v1
    84. kind: PlacementBinding
    85. metadata:
    86.   name: install-kmm
    87. placementRef:
    88.   apiGroup: apps.open-cluster-management.io
    89.   kind: PlacementRule
    90.   name: all-managed-clusters
    91. subjects:
    92.   - apiGroup: policy.open-cluster-management.io
    93.     kind: Policy
    94.     name: install-kmm
    1The spec.clusterSelector field can be customized to target select clusters only.

Customizing upgrades for kernel modules

Use this procedure to upgrade the kernel module while running maintenance operations on the node, including rebooting the node, if needed. To minimize the impact on the workloads running in the cluster, run the kernel upgrade process sequentially, one node at a time.

This procedure requires knowledge of the workload utilizing the kernel module and must be managed by the cluster administrator.

Prerequisites

  • Before upgrading, set the kmm.node.kubernetes.io/version-module.<module_namespace>.<module_name>=$moduleVersion label on all the nodes that are used by the kernel module.

  • Terminate all user application workloads on the node or move them to another node.

  • Unload the currently loaded kernel module.

  • Ensure that the user workload (the application running in the cluster that is accessing kernel module) is not running on the node prior to kernel module unloading and that the workload is back running on the node after the new kernel module version has been loaded.

Procedure

  1. Ensure that the device plugin managed by KMM on the node is unloaded.

  2. Update the following fields in the Module custom resource (CR):

    • containerImage (to the appropriate kernel version)

    • version

      The update should be atomic; that is, both the containerImage and version fields must be updated simultaneously.

  3. Terminate any workload using the kernel module on the node being upgraded.

  4. Remove the kmm.node.kubernetes.io/version-module.<module_namespace>.<module_name> label on the node. Run the following command to unload the kernel module from the node:

    1. $ oc label node/<node_name> kmm.node.kubernetes.io/version-module.<module_namespace>.<module_name>-

  5. If required, as the cluster administrator, perform any additional maintenance required on the node for the kernel module upgrade.

    If no additional upgrading is needed, you can skip Steps 3 through 6 by updating the kmm.node.kubernetes.io/version-module.<module-namespace>.<module-name> label value to the new $moduleVersion as set in the Module.

  6. Run the following command to add the kmm.node.kubernetes.io/version-module.<module_namespace>.<module_name>=$moduleVersion label to the node. The $moduleVersion must be equal to the new value of the version field in the Module CR.

    1. $ oc label node/<node_name> kmm.node.kubernetes.io/version-module.<module_namespace>.<module_name>=<desired_version>

    Because of Kubernetes limitations in label names, the combined length of Module name and namespace must not exceed 39 characters.

  7. Restore any workload that leverages the kernel module on the node.

  8. Reload the device plugin managed by KMM on the node.

Day 1 kernel module loading

Kernel Module Management (KMM) is typically a Day 2 Operator. Kernel modules are loaded only after the complete initialization of a Linux (RHCOS) server. However, in some scenarios the kernel module must be loaded at an earlier stage. Day 1 functionality allows you to use the Machine Config Operator (MCO) to load kernel modules during the Linux systemd initialization stage.

Additional resources

Day 1 supported use cases

The Day 1 functionality supports a limited number of use cases. The main use case is to allow loading out-of-tree (OOT) kernel modules prior to NetworkManager service initialization. It does not support loading kernel module at the initramfs stage.

The following are the conditions needed for Day 1 functionality:

  • The kernel module is not loaded in the kernel.

  • The in-tree kernel module is loaded into the kernel, but can be unloaded and replaced by the OOT kernel module. This means that the in-tree module is not referenced by any other kernel modules.

  • In order for Day 1 functionlity to work, the node must have a functional network interface, that is, an in-tree kernel driver for that interface. The OOT kernel module can be a network driver that will replace the functional network driver.

OOT kernel module loading flow

The loading of the out-of-tree (OOT) kernel module leverages the Machine Config Operator (MCO). The flow sequence is as follows:

Procedure

  1. Apply a MachineConfig resource to the existing running cluster. In order to identify the necessary nodes that need to be updated, you must create an appropriate MachineConfigPool resource.

  2. MCO applies the reboots node by node. On any rebooted node, two new systemd services are deployed: pull service and load service.

  3. The load service is configured to run prior to the NetworkConfiguration service. The service tries to pull a predefined kernel module image and then, using that image, to unload an in-tree module and load an OOT kernel module.

  4. The pull service is configured to run after NetworkManager service. The service checks if the preconfigured kernel module image is located on the node’s filesystem. If it is, the service exists normally, and the server continues with the boot process. If not, it pulls the image onto the node and reboots the node afterwards.

The kernel module image

The Day 1 functionality uses the same DTK based image leveraged by Day 2 KMM builds. The out-of-tree kernel module should be located under /opt/lib/modules/${kernelVersion}.

Additional resources

In-tree module replacement

The Day 1 functionality always tries to replace the in-tree kernel module with the OOT version. If the in-tree kernel module is not loaded, the flow is not affected; the service proceeds and loads the OOT kernel module.

MCO yaml creation

KMM provides an API to create an MCO YAML manifest for the Day 1 functionality:

  1. ProduceMachineConfig(machineConfigName, machineConfigPoolRef, kernelModuleImage, kernelModuleName string) (string, error)

The returned output is a string representation of the MCO YAML manifest to be applied. It is up to the customer to apply this YAML.

The parameters are:

machineConfigName

The name of the MCO YAML manifest. This parameter is set as the name parameter of the metadata of the MCO YAML manifest.

machineConfigPoolRef

The MachineConfigPool name used to identify the targeted nodes.

kernelModuleImage

The name of the container image that includes the OOT kernel module.

kernelModuleName

The name of the OOT kernel module. This parameter is used both to unload the in-tree kernel module (if loaded into the kernel) and to load the OOT kernel module.

The API is located under pkg/mcproducer package of the KMM source code. The KMM operator does not need to be running to use the Day 1 functionality. You only need to import the pkg/mcproducer package into their operator/utility code, call the API, and apply the produced MCO YAML to the cluster.

The MachineConfigPool

The MachineConfigPool identifies a collection of nodes that are affected by the applied MCO.

  1. kind: MachineConfigPool
  2. metadata:
  3.   name: sfc
  4. spec:
  5.   machineConfigSelector: (1)
  6.     matchExpressions:
  7.       - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker, sfc]}
  8.   nodeSelector: (2)
  9.     matchLabels:
  10.       node-role.kubernetes.io/sfc: ""
  11.   paused: false
  12.   maxUnavailable: 1
1Matches the labels in the MachineConfig.
2Matches the labels on the node.

There are predefined MachineConfigPools in the OCP cluster:

  • worker: Targets all worker nodes in the cluster

  • master: Targets all master nodes in the cluster

Define the following MachineConfig to target the master MachineConfigPool:

  1. metadata:
  2.   labels:
  3.     machineconfiguration.opensfhit.io/role: master

Define the following MachineConfig to target the worker MachineConfigPool:

  1. metadata:
  2.   labels:
  3.     machineconfiguration.opensfhit.io/role: worker

Additional resources

Debugging and troubleshooting

If the kmods in your driver container are not signed or are signed with the wrong key, then the container can enter a PostStartHookError or CrashLoopBackOff status. You can verify by running the oc describe command on your container, which displays the following message in this scenario:

  1. modprobe: ERROR: could not insert '<your_kmod_name>': Required key not available

KMM firmware support

Kernel modules sometimes need to load firmware files from the file system. KMM supports copying firmware files from the ModuleLoader image to the node’s file system.

The contents of .spec.moduleLoader.container.modprobe.firmwarePath are copied into the /var/lib/firmware path on the node before running the modprobe command to insert the kernel module.

All files and empty directories are removed from that location before running the modprobe -r command to unload the kernel module, when the pod is terminated.

Additional resources

Configuring the lookup path on nodes

On OKD nodes, the set of default lookup paths for firmwares does not include the /var/lib/firmware path.

Procedure

  1. Use the Machine Config Operator to create a MachineConfig custom resource (CR) that contains the /var/lib/firmware path:

    1. apiVersion: machineconfiguration.openshift.io/v1
    2. kind: MachineConfig
    3. metadata:
    4.   labels:
    5.     machineconfiguration.openshift.io/role: worker (1)
    6.   name: 99-worker-kernel-args-firmware-path
    7. spec:
    8.   kernelArguments:
    9.     - 'firmware_class.path=/var/lib/firmware'
    1You can configure the label based on your needs. In the case of single-node OpenShift, use either control-pane or master objects.

  2. By applying the MachineConfig CR, the nodes are automatically rebooted.

Additional resources

Building a ModuleLoader image

Procedure

  • In addition to building the kernel module itself, include the binary firmware in the builder image:

    1. FROM registry.redhat.io/ubi9/ubi-minimal as builder
    3. # Build the kmod
    5. RUN ["mkdir", "/firmware"]
    6. RUN ["curl", "-o", "/firmware/firmware.bin", "https://artifacts.example.com/firmware.bin"]
    8. FROM registry.redhat.io/ubi9/ubi-minimal
    10. # Copy the kmod, install modprobe, run depmod
    12. COPY --from=builder /firmware /firmware

Tuning the Module resource

Procedure

  • Set .spec.moduleLoader.container.modprobe.firmwarePath in the Module custom resource (CR):

    1. apiVersion: kmm.sigs.x-k8s.io/v1beta1
    2. kind: Module
    3. metadata:
    4.   name: my-kmod
    5. spec:
    6.   moduleLoader:
    7.     container:
    8.       modprobe:
    9.         moduleName: my-kmod  # Required
    11.         firmwarePath: /firmware (1)
    1Optional: Copies /firmware/* into /var/lib/firmware/ on the node.

Troubleshooting KMM

When troubleshooting KMM installation issues, you can monitor logs to determine at which stage issues occur. Then, retrieve diagnostic data relevant to that stage.

Using the must-gather tool

The oc adm must-gather command is the preferred way to collect a support bundle and provide debugging information to Red Hat Support. Collect specific information by running the command with the appropriate arguments as described in the following sections.

Additional resources

Gathering data for KMM

Procedure

  1. Gather the data for the KMM Operator controller manager:

    1. Set the MUST_GATHER_IMAGE variable:

      1. $ export MUST_GATHER_IMAGE=$(oc get deployment -n openshift-kmm kmm-operator-controller-manager -ojsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env[?(@.name=="RELATED_IMAGES_MUST_GATHER")].value}')

      Use the -n <namespace> switch to specify a namespace if you installed KMM in a custom namespace.

    2. Run the must-gather tool:

      1. $ oc adm must-gather --image="${MUST_GATHER_IMAGE}" -- /usr/bin/gather

  2. View the Operator logs:

    1. $ oc logs -fn openshift-kmm deployments/kmm-operator-controller-manager

    Example output

    1. I0228 09:36:37.352405       1 request.go:682] Waited for 1.001998746s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/machine.openshift.io/v1beta1?timeout=32s
    2. I0228 09:36:40.767060       1 listener.go:44] kmm/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"="127.0.0.1:8080"
    3. I0228 09:36:40.769483       1 main.go:234] kmm/setup "msg"="starting manager"
    4. I0228 09:36:40.769907       1 internal.go:366] kmm "msg"="Starting server" "addr"={"IP":"127.0.0.1","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
    5. I0228 09:36:40.770025       1 internal.go:366] kmm "msg"="Starting server" "addr"={"IP":"::","Port":8081,"Zone":""} "kind"="health probe"
    6. I0228 09:36:40.770128       1 leaderelection.go:248] attempting to acquire leader lease openshift-kmm/kmm.sigs.x-k8s.io...
    7. I0228 09:36:40.784396       1 leaderelection.go:258] successfully acquired lease openshift-kmm/kmm.sigs.x-k8s.io
    8. I0228 09:36:40.784876       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1beta1.Module"
    9. I0228 09:36:40.784925       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.DaemonSet"
    10. I0228 09:36:40.784968       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.Build"
    11. I0228 09:36:40.785001       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.Job"
    12. I0228 09:36:40.785025       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.Node"
    13. I0228 09:36:40.785039       1 controller.go:193] kmm "msg"="Starting Controller" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module"
    14. I0228 09:36:40.785458       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PodNodeModule" "controllerGroup"="" "controllerKind"="Pod" "source"="kind source: *v1.Pod"
    15. I0228 09:36:40.786947       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1beta1.PreflightValidation"
    16. I0228 09:36:40.787406       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1.Build"
    17. I0228 09:36:40.787474       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1.Job"
    18. I0228 09:36:40.787488       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1beta1.Module"
    19. I0228 09:36:40.787603       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="NodeKernel" "controllerGroup"="" "controllerKind"="Node" "source"="kind source: *v1.Node"
    20. I0228 09:36:40.787634       1 controller.go:193] kmm "msg"="Starting Controller" "controller"="NodeKernel" "controllerGroup"="" "controllerKind"="Node"
    21. I0228 09:36:40.787680       1 controller.go:193] kmm "msg"="Starting Controller" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation"
    22. I0228 09:36:40.785607       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "source"="kind source: *v1.ImageStream"
    23. I0228 09:36:40.787822       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="preflightvalidationocp" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidationOCP" "source"="kind source: *v1beta1.PreflightValidationOCP"
    24. I0228 09:36:40.787853       1 controller.go:193] kmm "msg"="Starting Controller" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream"
    25. I0228 09:36:40.787879       1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="preflightvalidationocp" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidationOCP" "source"="kind source: *v1beta1.PreflightValidation"
    26. I0228 09:36:40.787905       1 controller.go:193] kmm "msg"="Starting Controller" "controller"="preflightvalidationocp" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidationOCP"
    27. I0228 09:36:40.786489       1 controller.go:193] kmm "msg"="Starting Controller" "controller"="PodNodeModule" "controllerGroup"="" "controllerKind"="Pod"

Gathering data for KMM-Hub

Procedure

  1. Gather the data for the KMM Operator hub controller manager:

    1. Set the MUST_GATHER_IMAGE variable:

      1. $ export MUST_GATHER_IMAGE=$(oc get deployment -n openshift-kmm-hub kmm-operator-hub-controller-manager -ojsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env[?(@.name=="RELATED_IMAGES_MUST_GATHER")].value}')

      Use the -n <namespace> switch to specify a namespace if you installed KMM in a custom namespace.

    2. Run the must-gather tool:

      1. $ oc adm must-gather --image="${MUST_GATHER_IMAGE}" -- /usr/bin/gather -u

  2. View the Operator logs:

    1. $ oc logs -fn openshift-kmm-hub deployments/kmm-operator-hub-controller-manager

    Example output

    1. I0417 11:34:08.807472       1 request.go:682] Waited for 1.023403273s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/tuned.openshift.io/v1?timeout=32s
    2. I0417 11:34:12.373413       1 listener.go:44] kmm-hub/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"="127.0.0.1:8080"
    3. I0417 11:34:12.376253       1 main.go:150] kmm-hub/setup "msg"="Adding controller" "name"="ManagedClusterModule"
    4. I0417 11:34:12.376621       1 main.go:186] kmm-hub/setup "msg"="starting manager"
    5. I0417 11:34:12.377690       1 leaderelection.go:248] attempting to acquire leader lease openshift-kmm-hub/kmm-hub.sigs.x-k8s.io...
    6. I0417 11:34:12.378078       1 internal.go:366] kmm-hub "msg"="Starting server" "addr"={"IP":"127.0.0.1","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
    7. I0417 11:34:12.378222       1 internal.go:366] kmm-hub "msg"="Starting server" "addr"={"IP":"::","Port":8081,"Zone":""} "kind"="health probe"
    8. I0417 11:34:12.395703       1 leaderelection.go:258] successfully acquired lease openshift-kmm-hub/kmm-hub.sigs.x-k8s.io
    9. I0417 11:34:12.396334       1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1beta1.ManagedClusterModule"
    10. I0417 11:34:12.396403       1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.ManifestWork"
    11. I0417 11:34:12.396430       1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.Build"
    12. I0417 11:34:12.396469       1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.Job"
    13. I0417 11:34:12.396522       1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.ManagedCluster"
    14. I0417 11:34:12.396543       1 controller.go:193] kmm-hub "msg"="Starting Controller" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule"
    15. I0417 11:34:12.397175       1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "source"="kind source: *v1.ImageStream"
    16. I0417 11:34:12.397221       1 controller.go:193] kmm-hub "msg"="Starting Controller" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream"
    17. I0417 11:34:12.498335       1 filter.go:196] kmm-hub "msg"="Listing all ManagedClusterModules" "managedcluster"="local-cluster"
    18. I0417 11:34:12.498570       1 filter.go:205] kmm-hub "msg"="Listed ManagedClusterModules" "count"=0 "managedcluster"="local-cluster"
    19. I0417 11:34:12.498629       1 filter.go:238] kmm-hub "msg"="Adding reconciliation requests" "count"=0 "managedcluster"="local-cluster"
    20. I0417 11:34:12.498687       1 filter.go:196] kmm-hub "msg"="Listing all ManagedClusterModules" "managedcluster"="sno1-0"
    21. I0417 11:34:12.498750       1 filter.go:205] kmm-hub "msg"="Listed ManagedClusterModules" "count"=0 "managedcluster"="sno1-0"
    22. I0417 11:34:12.498801       1 filter.go:238] kmm-hub "msg"="Adding reconciliation requests" "count"=0 "managedcluster"="sno1-0"
    23. I0417 11:34:12.501947       1 controller.go:227] kmm-hub "msg"="Starting workers" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "worker count"=1
    24. I0417 11:34:12.501948       1 controller.go:227] kmm-hub "msg"="Starting workers" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "worker count"=1
    25. I0417 11:34:12.502285       1 imagestream_reconciler.go:50] kmm-hub "msg"="registered imagestream info mapping" "ImageStream"={"name":"driver-toolkit","namespace":"openshift"} "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "dtkImage"="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df42b4785a7a662b30da53bdb0d206120cf4d24b45674227b16051ba4b7c3934" "name"="driver-toolkit" "namespace"="openshift" "osImageVersion"="412.86.202302211547-0" "reconcileID"="e709ff0a-5664-4007-8270-49b5dff8bae9"