Post-installation node tasks

After installing OKD, you can further expand and customize your cluster to your requirements through certain node tasks.

Adding FCOS compute machines to an OKD cluster

You can add more Fedora CoreOS (FCOS) compute machines to your OKD cluster on bare metal.

Before you add more compute machines to a cluster that you installed on bare metal infrastructure, you must create FCOS machines for it to use. You can either use an ISO image or network PXE booting to create the machines.

Prerequisites

• You installed a cluster on bare metal.

• You have installation media and Fedora CoreOS (FCOS) images that you used to create your cluster. If you do not have these files, you must obtain them by following the instructions in the installation procedure.

Creating more FCOS machines using an ISO image

You can create more Fedora CoreOS (FCOS) compute machines for your bare metal cluster by using an ISO image to create the machines.

Prerequisites

• Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.

Procedure

1. Use the ISO file to install FCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.

   • Use ISO redirection with a LOM interface.

2. After the instance boots, press the TAB or E key to edit the kernel command line.

3. Add the parameters to the kernel command line:

   coreos.inst.install_dev=sda (1)
   coreos.inst.ignition_url=http://example.com/worker.ign (2)

   1	Specify the block device of the system to install to.
   2	Specify the URL of the compute Ignition config file. Only HTTP and HTTPS protocols are supported.

4. Press Enter to complete the installation. After FCOS installs, the system reboots. After the system reboots, it applies the Ignition config file that you specified.

5. Continue to create more compute machines for your cluster.

Creating more FCOS machines by PXE or iPXE booting

You can create more Fedora CoreOS (FCOS) compute machines for your bare metal cluster by using PXE or iPXE booting.

Prerequisites

• Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.

• Obtain the URLs of the FCOS ISO image, compressed metal BIOS, kernel, and initramfs files that you uploaded to your HTTP server during cluster installation.

• You have access to the PXE booting infrastructure that you used to create the machines for your OKD cluster during installation. The machines must boot from their local disks after FCOS is installed on them.

• If you use UEFI, you have access to the grub.conf file that you modified during OKD installation.

Procedure

1. Confirm that your PXE or iPXE installation for the FCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> (1)
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img (2)

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the FCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

• For iPXE:

  kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img (1)
  initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img (2)

  1

  Specify locations of the FCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

  2

  Specify the location of the initramfs file that you uploaded to your HTTP server.

This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

1. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Approving the certificate signing requests for your machines

When you add machines to a cluster, two pending certificate signing requests (CSRs) are generated for each machine that you added. You must confirm that these CSRs are approved or, if necessary, approve them yourself. The client requests must be approved first, followed by the server requests.

Prerequisites

• You added machines to your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.25.0
   master-1  Ready     master  63m  v1.25.0
   master-2  Ready     master  64m  v1.25.0

   The output lists all of the machines that you created.

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> (1)

     1	<csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> (1)

     1	<csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.25.0
   master-1  Ready     master  73m  v1.25.0
   master-2  Ready     master  74m  v1.25.0
   worker-0  Ready     worker  11m  v1.25.0
   worker-1  Ready     worker  11m  v1.25.0

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Additional information

• For more information on CSRs, see Certificate Signing Requests.

Adding a new FCOS worker node with a custom /var partition in AWS

OKD supports partitioning devices during installation by using machine configs that are processed during the bootstrap. However, if you use /var partitioning, the device name must be determined at installation and cannot be changed. You cannot add different instance types as nodes if they have a different device naming schema. For example, if you configured the /var partition with the default AWS device name for m4.large instances, dev/xvdb, you cannot directly add an AWS m5.large instance, as m5.large instances use a /dev/nvme1n1 device by default. The device might fail to partition due to the different naming schema.

The procedure in this section shows how to add a new Fedora CoreOS (FCOS) compute node with an instance that uses a different device name from what was configured at installation. You create a custom user data secret and configure a new compute machine set. These steps are specific to an AWS cluster. The principles apply to other cloud deployments also. However, the device naming schema is different for other deployments and should be determined on a per-case basis.

Procedure

1. On a command line, change to the openshift-machine-api namespace:

   $ oc project openshift-machine-api

2. Create a new secret from the worker-user-data secret:

   1. Export the userData section of the secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.userData | base64decode}}' | jq > userData.txt

   2. Edit the text file to add the storage, filesystems, and systemd stanzas for the partitions you want to use for the new node. You can specify any Ignition configuration parameters as needed.

      Do not change the values in the ignition stanza.

      {
        "ignition": {
          "config": {
            "merge": [
              {
                "source": "https:...."
              }
            ]
          },
          "security": {
            "tls": {
              "certificateAuthorities": [
                {
                  "source": "data:text/plain;charset=utf-8;base64,.....=="
                }
              ]
            }
          },
          "version": "3.2.0"
        },
        "storage": {
          "disks": [
            {
              "device": "/dev/nvme1n1", (1)
              "partitions": [
                {
                  "label": "var",
                  "sizeMiB": 50000, (2)
                  "startMiB": 0 (3)
                }
              ]
            }
          ],
          "filesystems": [
            {
              "device": "/dev/disk/by-partlabel/var", (4)
              "format": "xfs", (5)
              "path": "/var" (6)
            }
          ]
        },
        "systemd": {
          "units": [ (7)
            {
              "contents": "[Unit]\nBefore=local-fs.target\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nOptions=defaults,pquota\n[Install]\nWantedBy=local-fs.target\n",
              "enabled": true,
              "name": "var.mount"
            }
          ]
        }
      }

      1	Specifies an absolute path to the AWS block device.
      2	Specifies the size of the data partition in Mebibytes.
      3	Specifies the start of the partition in Mebibytes. When adding a data partition to the boot disk, a minimum value of 25000 MB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of FCOS might overwrite the beginning of the data partition.
      4	Specifies an absolute path to the /var partition.
      5	Specifies the filesystem format.
      6	Specifies the mount-point of the filesystem while Ignition is running relative to where the root filesystem will be mounted. This is not necessarily the same as where it should be mounted in the real root, but it is encouraged to make it the same.
      7	Defines a systemd mount unit that mounts the /dev/disk/by-partlabel/var device to the /var partition.

   3. Extract the disableTemplating section from the work-user-data secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.disableTemplating | base64decode}}' | jq > disableTemplating.txt

   4. Create the new user data secret file from the two text files. This user data secret passes the additional node partition information in the userData.txt file to the newly created node.

      $ oc create secret generic worker-user-data-x5 --from-file=userData=userData.txt --from-file=disableTemplating=disableTemplating.txt

3. Create a new compute machine set for the new node:

   1. Create a new compute machine set YAML file, similar to the following, which is configured for AWS. Add the required partitions and the newly-created user data secret:

      Use an existing compute machine set as a template and change the parameters as needed for the new node.

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: auto-52-92tf4
        name: worker-us-east-2-nvme1n1 (1)
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: auto-52-92tf4
            machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: auto-52-92tf4
              machine.openshift.io/cluster-api-machine-role: worker
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
          spec:
            metadata: {}
            providerSpec:
              value:
                ami:
                  id: ami-0c2dbd95931a
                apiVersion: awsproviderconfig.openshift.io/v1beta1
                blockDevices:
                - DeviceName: /dev/nvme1n1 (2)
                  ebs:
                    encrypted: true
                    iops: 0
                    volumeSize: 120
                    volumeType: gp2
                - DeviceName: /dev/nvme1n2 (3)
                  ebs:
                    encrypted: true
                    iops: 0
                    volumeSize: 50
                    volumeType: gp2
                credentialsSecret:
                  name: aws-cloud-credentials
                deviceIndex: 0
                iamInstanceProfile:
                  id: auto-52-92tf4-worker-profile
                instanceType: m6i.large
                kind: AWSMachineProviderConfig
                metadata:
                  creationTimestamp: null
                placement:
                  availabilityZone: us-east-2b
                  region: us-east-2
                securityGroups:
                - filters:
                  - name: tag:Name
                    values:
                    - auto-52-92tf4-worker-sg
                subnet:
                  id: subnet-07a90e5db1
                tags:
                - name: kubernetes.io/cluster/auto-52-92tf4
                  value: owned
                userDataSecret:
                  name: worker-user-data-x5 (4)

      1	Specifies a name for the new node.
      2	Specifies an absolute path to the AWS block device, here an encrypted EBS volume.
      3	Optional. Specifies an additional EBS volume.
      4	Specifies the user data secret file.

   2. Create the compute machine set:

      $ oc create -f <file-name>.yaml

      The machines might take a few moments to become available.

4. Verify that the new partition and nodes are created:

   1. Verify that the compute machine set is created:

      $ oc get machineset

      Example output

      NAME                                               DESIRED   CURRENT   READY   AVAILABLE   AGE
      ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1a        1         1         1       1           124m
      ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1b        2         2         2       2           124m
      worker-us-east-2-nvme1n1                           1         1         1       1           2m35s (1)

      1	This is the new compute machine set.

   2. Verify that the new node is created:

      $ oc get nodes

      Example output

      NAME                           STATUS   ROLES    AGE     VERSION
      ip-10-0-128-78.ec2.internal    Ready    worker   117m    v1.25.0
      ip-10-0-146-113.ec2.internal   Ready    master   127m    v1.25.0
      ip-10-0-153-35.ec2.internal    Ready    worker   118m    v1.25.0
      ip-10-0-176-58.ec2.internal    Ready    master   126m    v1.25.0
      ip-10-0-217-135.ec2.internal   Ready    worker   2m57s   v1.25.0 (1)
      ip-10-0-225-248.ec2.internal   Ready    master   127m    v1.25.0
      ip-10-0-245-59.ec2.internal    Ready    worker   116m    v1.25.0

      1	This is new new node.

   3. Verify that the custom /var partition is created on the new node:

      $ oc debug node/<node-name> -- chroot /host lsblk

      For example:

      $ oc debug node/ip-10-0-217-135.ec2.internal -- chroot /host lsblk

      Example output

      NAME        MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
      nvme0n1     202:0    0   120G  0 disk
      |-nvme0n1p1 202:1    0     1M  0 part
      |-nvme0n1p2 202:2    0   127M  0 part
      |-nvme0n1p3 202:3    0   384M  0 part /boot
      `-nvme0n1p4 202:4    0 119.5G  0 part /sysroot
      nvme1n1     202:16   0    50G  0 disk
      `-nvme1n1p1 202:17   0  48.8G  0 part /var (1)

      1	The nvme1n1 device is mounted to the /var partition.

Additional resources

• For more information on how OKD uses disk partitioning, see Disk partitioning.

Deploying machine health checks

Understand and deploy machine health checks.

About machine health checks

To monitor machine health, create a resource to define the configuration for a controller. Set a condition to check, such as staying in the NotReady status for five minutes or displaying a permanent condition in the node-problem-detector, and a label for the set of machines to monitor.

The controller that observes a MachineHealthCheck resource checks for the defined condition. If a machine fails the health check, the machine is automatically deleted and one is created to take its place. When a machine is deleted, you see a machine deleted event.

To limit disruptive impact of the machine deletion, the controller drains and deletes only one node at a time. If there are more unhealthy machines than the maxUnhealthy threshold allows for in the targeted pool of machines, remediation stops and therefore enables manual intervention.

To stop the check, remove the resource.

Limitations when deploying machine health checks

There are limitations to consider before deploying a machine health check:

• Only machines owned by a machine set are remediated by a machine health check.

• If the node for a machine is removed from the cluster, a machine health check considers the machine to be unhealthy and remediates it immediately.

• If the corresponding node for a machine does not join the cluster after the nodeStartupTimeout, the machine is remediated.

• A machine is remediated immediately if the Machine resource phase is Failed.

Additional resources

• About the Control Plane Machine Set Operator

Sample MachineHealthCheck resource

The MachineHealthCheck resource for all cloud-based installation types, and other than bare metal, resembles the following YAML file:

apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: example (1)
  namespace: openshift-machine-api
spec:
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-machine-role: <role> (2)
      machine.openshift.io/cluster-api-machine-type: <role> (2)
      machine.openshift.io/cluster-api-machineset: <cluster_name>-<label>-<zone> (3)
  unhealthyConditions:
  - type:    "Ready"
    timeout: "300s" (4)
    status: "False"
  - type:    "Ready"
    timeout: "300s" (4)
    status: "Unknown"
  maxUnhealthy: "40%" (5)
  nodeStartupTimeout: "10m" (6)

Short-circuiting machine health check remediation

Short-circuiting ensures that machine health checks remediate machines only when the cluster is healthy. Short-circuiting is configured through the maxUnhealthy field in the MachineHealthCheck resource.

If the user defines a value for the maxUnhealthy field, before remediating any machines, the MachineHealthCheck compares the value of maxUnhealthy with the number of machines within its target pool that it has determined to be unhealthy. Remediation is not performed if the number of unhealthy machines exceeds the maxUnhealthy limit.

The appropriate maxUnhealthy value depends on the scale of the cluster you deploy and how many machines the MachineHealthCheck covers. For example, you can use the maxUnhealthy value to cover multiple compute machine sets across multiple availability zones so that if you lose an entire zone, your maxUnhealthy setting prevents further remediation within the cluster. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

The maxUnhealthy field can be set as either an integer or percentage. There are different remediation implementations depending on the maxUnhealthy value.

Setting maxUnhealthy by using an absolute value

If maxUnhealthy is set to 2:

• Remediation will be performed if 2 or fewer nodes are unhealthy

• Remediation will not be performed if 3 or more nodes are unhealthy

These values are independent of how many machines are being checked by the machine health check.

Setting maxUnhealthy by using percentages

If maxUnhealthy is set to 40% and there are 25 machines being checked:

• Remediation will be performed if 10 or fewer nodes are unhealthy

• Remediation will not be performed if 11 or more nodes are unhealthy

If maxUnhealthy is set to 40% and there are 6 machines being checked:

• Remediation will be performed if 2 or fewer nodes are unhealthy

• Remediation will not be performed if 3 or more nodes are unhealthy

Creating a machine health check resource

You can create a MachineHealthCheck resource for machine sets in your cluster.

Prerequisites

• Install the oc command line interface.

Procedure

1. Create a healthcheck.yml file that contains the definition of your machine health check.

2. Apply the healthcheck.yml file to your cluster:

   $ oc apply -f healthcheck.yml

Scaling a compute machine set manually

To add or remove an instance of a machine in a compute machine set, you can manually scale the compute machine set.

This guidance is relevant to fully automated, installer-provisioned infrastructure installations. Customized, user-provisioned infrastructure installations do not have compute machine sets.

Prerequisites

• Install an OKD cluster and the oc command line.

• Log in to oc as a user with cluster-admin permission.

Procedure

1. View the compute machine sets that are in the cluster by running the following command:

   $ oc get machinesets -n openshift-machine-api

   The compute machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. View the compute machines that are in the cluster by running the following command:

   $ oc get machine -n openshift-machine-api

3. Set the annotation on the compute machine that you want to delete by running the following command:

   $ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/delete-machine="true"

4. Cordon and drain the node that you want to delete by running the following commands:

   $ oc adm cordon <node_name>

   $ oc adm drain <node_name>

5. Scale the compute machine set by running one of the following commands:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   You can alternatively apply the following YAML to scale the compute machine set: apiVersion: machine.openshift.io/v1beta1 kind: MachineSet metadata: name: <machineset> namespace: openshift-machine-api spec: replicas: 2

   You can scale the compute machine set up or down. It takes several minutes for the new machines to be available.

Verification

• Verify the deletion of the intended machine by running the following command:

  $ oc get machines

Understanding the difference between compute machine sets and the machine config pool

MachineSet objects describe OKD nodes with respect to the cloud or machine provider.

The MachineConfigPool object allows MachineConfigController components to define and provide the status of machines in the context of upgrades.

The MachineConfigPool object allows users to configure how upgrades are rolled out to the OKD nodes in the machine config pool.

The NodeSelector object can be replaced with a reference to the MachineSet object.

Recommended node host practices

The OKD node configuration file contains important options. For example, two parameters control the maximum number of pods that can be scheduled to a node: podsPerCore and maxPods.

When both options are in use, the lower of the two values limits the number of pods on a node. Exceeding these values can result in:

• Increased CPU utilization.

• Slow pod scheduling.

• Potential out-of-memory scenarios, depending on the amount of memory in the node.

• Exhausting the pool of IP addresses.

• Resource overcommitting, leading to poor user application performance.

podsPerCore sets the number of pods the node can run based on the number of processor cores on the node. For example, if podsPerCore is set to 10 on a node with 4 processor cores, the maximum number of pods allowed on the node will be 40.

kubeletConfig:
  podsPerCore: 10

Setting podsPerCore to 0 disables this limit. The default is 0. podsPerCore cannot exceed maxPods.

maxPods sets the number of pods the node can run to a fixed value, regardless of the properties of the node.

 kubeletConfig:
    maxPods: 250

Creating a KubeletConfig CRD to edit kubelet parameters

The kubelet configuration is currently serialized as an Ignition configuration, so it can be directly edited. However, there is also a new kubelet-config-controller added to the Machine Config Controller (MCC). This lets you use a KubeletConfig custom resource (CR) to edit the kubelet parameters.

Consider the following guidance:

• Create one KubeletConfig CR for each machine config pool with all the config changes you want for that pool. If you are applying the same content to all of the pools, you need only one KubeletConfig CR for all of the pools.

• Edit an existing KubeletConfig CR to modify existing settings or add new settings, instead of creating a CR for each change. It is recommended that you create a CR only to modify a different machine config pool, or for changes that are intended to be temporary, so that you can revert the changes.

• As needed, create multiple KubeletConfig CRs with a limit of 10 per cluster. For the first KubeletConfig CR, the Machine Config Operator (MCO) creates a machine config appended with kubelet. With each subsequent CR, the controller creates another kubelet machine config with a numeric suffix. For example, if you have a kubelet machine config with a -2 suffix, the next kubelet machine config is appended with -3.

If you want to delete the machine configs, delete them in reverse order to avoid exceeding the limit. For example, you delete the kubelet-3 machine config before deleting the kubelet-2 machine config.

Example KubeletConfig CR

$ oc get kubeletconfig

NAME                AGE
set-max-pods        15m

Example showing a KubeletConfig machine config

$ oc get mc | grep kubelet

...
99-worker-generated-kubelet-1                  b5c5119de007945b6fe6fb215db3b8e2ceb12511   3.2.0             26m
...

The following procedure is an example to show how to configure the maximum number of pods per node on the worker nodes.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CR for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: set-max-pods (1)

      1	If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

Procedure

1. View the available machine configuration objects that you can select:

   $ oc get machineconfig

   By default, the two kubelet-related configs are 01-master-kubelet and 01-worker-kubelet.

2. Check the current value for the maximum pods per node:

   $ oc describe node <node_name>

   For example:

   $ oc describe node ci-ln-5grqprb-f76d1-ncnqq-worker-a-mdv94

   Look for value: pods: <value> in the Allocatable stanza:

   Example output

   Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3500m
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15341844Ki
    pods:                        250

3. Set the maximum pods per node on the worker nodes by creating a custom resource file that contains the kubelet configuration:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods (1)
     kubeletConfig:
       maxPods: 500 (2)

   1	Enter the label from the machine config pool.
   2	Add the kubelet configuration. In this example, use maxPods to set the maximum pods per node.

   The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, 50 for kubeAPIQPS and 100 for kubeAPIBurst, are sufficient if there are limited pods running on each node. It is recommended to update the kubelet QPS and burst rates if there are enough CPU and memory resources on the node. apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: set-max-pods spec: machineConfigPoolSelector: matchLabels: custom-kubelet: set-max-pods kubeletConfig: maxPods: <pod_count> kubeAPIBurst: <burst_rate> kubeAPIQPS: <QPS>

   1. Update the machine config pool for workers with the label:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

   2. Create the KubeletConfig object:

      $ oc create -f change-maxPods-cr.yaml

   3. Verify that the KubeletConfig object is created:

      $ oc get kubeletconfig

      Example output

      NAME                AGE
      set-max-pods        15m

      Depending on the number of worker nodes in the cluster, wait for the worker nodes to be rebooted one by one. For a cluster with 3 worker nodes, this could take about 10 to 15 minutes.

4. Verify that the changes are applied to the node:

   1. Check on a worker node that the maxPods value changed:

      $ oc describe node <node_name>

   2. Locate the Allocatable stanza:

       ...
      Allocatable:
        attachable-volumes-gce-pd:  127
        cpu:                        3500m
        ephemeral-storage:          123201474766
        hugepages-1Gi:              0
        hugepages-2Mi:              0
        memory:                     14225400Ki
        pods:                       500 (1)
       ...

      1	In this example, the pods parameter should report the value you set in the KubeletConfig object.

5. Verify the change in the KubeletConfig object:

   $ oc get kubeletconfigs set-max-pods -o yaml

   This should show a status of True and type:Success, as shown in the following example:

   spec:
     kubeletConfig:
       maxPods: 500
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods
   status:
     conditions:
     - lastTransitionTime: "2021-06-30T17:04:07Z"
       message: Success
       status: "True"
       type: Success

Modifying the number of unavailable worker nodes

By default, only one machine is allowed to be unavailable when applying the kubelet-related configuration to the available worker nodes. For a large cluster, it can take a long time for the configuration change to be reflected. At any time, you can adjust the number of machines that are updating to speed up the process.

Procedure

1. Edit the worker machine config pool:

   $ oc edit machineconfigpool worker

2. Add the maxUnavailable field and set the value:

   spec:
     maxUnavailable: <node_count>

   When setting the value, consider the number of worker nodes that can be unavailable without affecting the applications running on the cluster.

Control plane node sizing

The control plane node resource requirements depend on the number and type of nodes and objects in the cluster. The following control plane node size recommendations are based on the results of a control plane density focused testing, or Cluster-density. This test creates the following objects across a given number of namespaces:

• 1 image stream

• 1 build

• 5 deployments, with 2 pod replicas in a sleep state, mounting 4 secrets, 4 config maps, and 1 downward API volume each

• 5 services, each one pointing to the TCP/8080 and TCP/8443 ports of one of the previous deployments

• 1 route pointing to the first of the previous services

• 10 secrets containing 2048 random string characters

• 10 config maps containing 2048 random string characters

The data from the table above is based on an OKD running on top of AWS, using r5.4xlarge instances as control-plane nodes and m5.2xlarge instances as worker nodes.

On a large and dense cluster with three control plane nodes, the CPU and memory usage will spike up when one of the nodes is stopped, rebooted, or fails. The failures can be due to unexpected issues with power, network, underlying infrastructure, or intentional cases where the cluster is restarted after shutting it down to save costs. The remaining two control plane nodes must handle the load in order to be highly available, which leads to increase in the resource usage. This is also expected during upgrades because the control plane nodes are cordoned, drained, and rebooted serially to apply the operating system updates, as well as the control plane Operators update. To avoid cascading failures, keep the overall CPU and memory resource usage on the control plane nodes to at most 60% of all available capacity to handle the resource usage spikes. Increase the CPU and memory on the control plane nodes accordingly to avoid potential downtime due to lack of resources.

Operator Lifecycle Manager (OLM ) runs on the control plane nodes and its memory footprint depends on the number of namespaces and user installed operators that OLM needs to manage on the cluster. Control plane nodes need to be sized accordingly to avoid OOM kills. Following data points are based on the results from cluster maximums testing.

Setting up CPU Manager

Procedure

1. Optional: Label a node:

   # oc label node perf-node.example.com cpumanager=true

2. Edit the MachineConfigPool of the nodes where CPU Manager should be enabled. In this example, all workers have CPU Manager enabled:

   # oc edit machineconfigpool worker

3. Add a label to the worker machine config pool:

   metadata:
     creationTimestamp: 2020-xx-xxx
     generation: 3
     labels:
       custom-kubelet: cpumanager-enabled

4. Create a KubeletConfig, cpumanager-kubeletconfig.yaml, custom resource (CR). Refer to the label created in the previous step to have the correct nodes updated with the new kubelet config. See the machineConfigPoolSelector section:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: cpumanager-enabled
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: cpumanager-enabled
     kubeletConfig:
        cpuManagerPolicy: static (1)
        cpuManagerReconcilePeriod: 5s (2)

   1	Specify a policy: none. This policy explicitly enables the existing default CPU affinity scheme, providing no affinity beyond what the scheduler does automatically. static. This policy allows pods with certain resource characteristics to be granted increased CPU affinity and exclusivity on the node. If static, you must use a lowercase s.
   2	Optional. Specify the CPU Manager reconcile frequency. The default is 5s.

5. Create the dynamic kubelet config:

   # oc create -f cpumanager-kubeletconfig.yaml

   This adds the CPU Manager feature to the kubelet config and, if needed, the Machine Config Operator (MCO) reboots the node. To enable CPU Manager, a reboot is not needed.

6. Check for the merged kubelet config:

   # oc get machineconfig 99-worker-XXXXXX-XXXXX-XXXX-XXXXX-kubelet -o json | grep ownerReference -A7

   Example output

          "ownerReferences": [
               {
                   "apiVersion": "machineconfiguration.openshift.io/v1",
                   "kind": "KubeletConfig",
                   "name": "cpumanager-enabled",
                   "uid": "7ed5616d-6b72-11e9-aae1-021e1ce18878"
               }
           ]

7. Check the worker for the updated kubelet.conf:

   # oc debug node/perf-node.example.com
   sh-4.2# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager

   Example output

   cpuManagerPolicy: static        (1)
   cpuManagerReconcilePeriod: 5s   (1)

   1	These settings were defined when you created the KubeletConfig CR.

8. Create a pod that requests a core or multiple cores. Both limits and requests must have their CPU value set to a whole integer. That is the number of cores that will be dedicated to this pod:

   # cat cpumanager-pod.yaml

   Example output

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: cpumanager-
   spec:
     containers:
     - name: cpumanager
       image: gcr.io/google_containers/pause-amd64:3.0
       resources:
         requests:
           cpu: 1
           memory: "1G"
         limits:
           cpu: 1
           memory: "1G"
     nodeSelector:
       cpumanager: "true"

9. Create the pod:

   # oc create -f cpumanager-pod.yaml

10. Verify that the pod is scheduled to the node that you labeled:

    # oc describe pod cpumanager

    Example output

    Name:               cpumanager-6cqz7
    Namespace:          default
    Priority:           0
    PriorityClassName:  <none>
    Node:  perf-node.example.com/xxx.xx.xx.xxx
    ...
    Limits:
         cpu:     1
         memory:  1G
       Requests:
         cpu:        1
         memory:     1G
    ...
    QoS Class:       Guaranteed
    Node-Selectors:  cpumanager=true

11. Verify that the cgroups are set up correctly. Get the process ID (PID) of the pause process:

    # ├─init.scope
    │ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
    └─kubepods.slice
     ├─kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice
     │ ├─crio-b5437308f1a574c542bdf08563b865c0345c8f8c0b0a655612c.scope
     │ └─32706 /pause

    Pods of quality of service (QoS) tier Guaranteed are placed within the kubepods.slice. Pods of other QoS tiers end up in child cgroups of kubepods:

    # cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice/crio-b5437308f1ad1a7db0574c542bdf08563b865c0345c86e9585f8c0b0a655612c.scope
    # for i in `ls cpuset.cpus tasks` ; do echo -n "$i "; cat $i ; done

    Example output

    cpuset.cpus 1
    tasks 32706

12. Check the allowed CPU list for the task:

    # grep ^Cpus_allowed_list /proc/32706/status

    Example output

    Cpus_allowed_list:    1

13. Verify that another pod (in this case, the pod in the burstable QoS tier) on the system cannot run on the core allocated for the Guaranteed pod:

    # cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc494a073_6b77_11e9_98c0_06bba5c387ea.slice/crio-c56982f57b75a2420947f0afc6cafe7534c5734efc34157525fa9abbf99e3849.scope/cpuset.cpus
    0
    # oc describe node perf-node.example.com

    Example output

    ...
    Capacity:
    attachable-volumes-aws-ebs:  39
    cpu:                         2
    ephemeral-storage:           124768236Ki
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      8162900Ki
    pods:                        250
    Allocatable:
    attachable-volumes-aws-ebs:  39
    cpu:                         1500m
    ephemeral-storage:           124768236Ki
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      7548500Ki
    pods:                        250
    -------                               ----                           ------------  ----------  ---------------  -------------  ---
     default                                 cpumanager-6cqz7               1 (66%)       1 (66%)     1G (12%)         1G (12%)       29m
    Allocated resources:
     (Total limits may be over 100 percent, i.e., overcommitted.)
     Resource                    Requests          Limits
     --------                    --------          ------
     cpu                         1440m (96%)       1 (66%)

    This VM has two CPU cores. The system-reserved setting reserves 500 millicores, meaning that half of one core is subtracted from the total capacity of the node to arrive at the Node Allocatable amount. You can see that Allocatable CPU is 1500 millicores. This means you can run one of the CPU Manager pods since each will take one whole core. A whole core is equivalent to 1000 millicores. If you try to schedule a second pod, the system will accept the pod, but it will never be scheduled:

    NAME                    READY   STATUS    RESTARTS   AGE
    cpumanager-6cqz7        1/1     Running   0          33m
    cpumanager-7qc2t        0/1     Pending   0          11s

Huge pages

Understand and configure huge pages.

What huge pages do

Memory is managed in blocks known as pages. On most systems, a page is 4Ki. 1Mi of memory is equal to 256 pages; 1Gi of memory is 256,000 pages, and so on. CPUs have a built-in memory management unit that manages a list of these pages in hardware. The Translation Lookaside Buffer (TLB) is a small hardware cache of virtual-to-physical page mappings. If the virtual address passed in a hardware instruction can be found in the TLB, the mapping can be determined quickly. If not, a TLB miss occurs, and the system falls back to slower, software-based address translation, resulting in performance issues. Since the size of the TLB is fixed, the only way to reduce the chance of a TLB miss is to increase the page size.

A huge page is a memory page that is larger than 4Ki. On x86_64 architectures, there are two common huge page sizes: 2Mi and 1Gi. Sizes vary on other architectures. To use huge pages, code must be written so that applications are aware of them. Transparent Huge Pages (THP) attempt to automate the management of huge pages without application knowledge, but they have limitations. In particular, they are limited to 2Mi page sizes. THP can lead to performance degradation on nodes with high memory utilization or fragmentation due to defragmenting efforts of THP, which can lock memory pages. For this reason, some applications may be designed to (or recommend) usage of pre-allocated huge pages instead of THP.

How huge pages are consumed by apps

Nodes must pre-allocate huge pages in order for the node to report its huge page capacity. A node can only pre-allocate huge pages for a single size.

Huge pages can be consumed through container-level resource requirements using the resource name hugepages-<size>, where size is the most compact binary notation using integer values supported on a particular node. For example, if a node supports 2048KiB page sizes, it exposes a schedulable resource hugepages-2Mi. Unlike CPU or memory, huge pages do not support over-commitment.

apiVersion: v1
kind: Pod
metadata:
  generateName: hugepages-volume-
spec:
  containers:
  - securityContext:
      privileged: true
    image: rhel7:latest
    command:
    - sleep
    - inf
    name: example
    volumeMounts:
    - mountPath: /dev/hugepages
      name: hugepage
    resources:
      limits:
        hugepages-2Mi: 100Mi (1)
        memory: "1Gi"
        cpu: "1"
  volumes:
  - name: hugepage
    emptyDir:
      medium: HugePages

Allocating huge pages of a specific size

Some platforms support multiple huge page sizes. To allocate huge pages of a specific size, precede the huge pages boot command parameters with a huge page size selection parameter hugepagesz=<size>. The <size> value must be specified in bytes with an optional scale suffix [kKmMgG]. The default huge page size can be defined with the default_hugepagesz=<size> boot parameter.

Huge page requirements

• Huge page requests must equal the limits. This is the default if limits are specified, but requests are not.

• Huge pages are isolated at a pod scope. Container isolation is planned in a future iteration.

• EmptyDir volumes backed by huge pages must not consume more huge page memory than the pod request.

• Applications that consume huge pages via shmget() with SHM_HUGETLB must run with a supplemental group that matches proc/sys/vm/hugetlb_shm_group.

Configuring huge pages

Nodes must pre-allocate huge pages used in an OKD cluster. There are two ways of reserving huge pages: at boot time and at run time. Reserving at boot time increases the possibility of success because the memory has not yet been significantly fragmented. The Node Tuning Operator currently supports boot time allocation of huge pages on specific nodes.

At boot time

Procedure

To minimize node reboots, the order of the steps below needs to be followed:

1. Label all nodes that need the same huge pages setting by a label.

   $ oc label node <node_using_hugepages> node-role.kubernetes.io/worker-hp=

2. Create a file with the following content and name it hugepages-tuned-boottime.yaml:

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: hugepages (1)
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile: (2)
     - data: |
         [main]
         summary=Boot time configuration for hugepages
         include=openshift-node
         [bootloader]
         cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=50 (3)
       name: openshift-node-hugepages
     recommend:
     - machineConfigLabels: (4)
         machineconfiguration.openshift.io/role: "worker-hp"
       priority: 30
       profile: openshift-node-hugepages

   1	Set the name of the Tuned resource to hugepages.
   2	Set the profile section to allocate huge pages.
   3	Note the order of parameters is important as some platforms support huge pages of various sizes.
   4	Enable machine config pool based matching.

3. Create the Tuned hugepages object

   $ oc create -f hugepages-tuned-boottime.yaml

4. Create a file with the following content and name it hugepages-mcp.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-hp
     labels:
       worker-hp: ""
   spec:
     machineConfigSelector:
       matchExpressions:
         - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]}
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-hp: ""

5. Create the machine config pool:

   $ oc create -f hugepages-mcp.yaml

Given enough non-fragmented memory, all the nodes in the worker-hp machine config pool should now have 50 2Mi huge pages allocated.

$ oc get node <node_using_hugepages> -o jsonpath="{.status.allocatable.hugepages-2Mi}"
100Mi

Understanding device plugins

The device plugin provides a consistent and portable solution to consume hardware devices across clusters. The device plugin provides support for these devices through an extension mechanism, which makes these devices available to Containers, provides health checks of these devices, and securely shares them.

A device plugin is a gRPC service running on the nodes (external to the kubelet) that is responsible for managing specific hardware resources. Any device plugin must support following remote procedure calls (RPCs):

service DevicePlugin {
      // GetDevicePluginOptions returns options to be communicated with Device
      // Manager
      rpc GetDevicePluginOptions(Empty) returns (DevicePluginOptions) {}
      // ListAndWatch returns a stream of List of Devices
      // Whenever a Device state change or a Device disappears, ListAndWatch
      // returns the new list
      rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
      // Allocate is called during container creation so that the Device
      // Plug-in can run device specific operations and instruct Kubelet
      // of the steps to make the Device available in the container
      rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
      // PreStartcontainer is called, if indicated by Device Plug-in during
      // registration phase, before each container start. Device plug-in
      // can run device specific operations such as reseting the device
      // before making devices available to the container
      rpc PreStartcontainer(PreStartcontainerRequest) returns (PreStartcontainerResponse) {}
}

Example device plugins

• Nvidia GPU device plugin for COS-based operating system

• Nvidia official GPU device plugin

• Solarflare device plugin

• KubeVirt device plugins: vfio and kvm

• Kubernetes device plugin for IBM Crypto Express (CEX) cards

Methods for deploying a device plugin

• Daemon sets are the recommended approach for device plugin deployments.

• Upon start, the device plugin will try to create a UNIX domain socket at /var/lib/kubelet/device-plugin/ on the node to serve RPCs from Device Manager.

• Since device plugins must manage hardware resources, access to the host file system, as well as socket creation, they must be run in a privileged security context.

• More specific details regarding deployment steps can be found with each device plugin implementation.

Understanding the Device Manager

Device Manager provides a mechanism for advertising specialized node hardware resources with the help of plugins known as device plugins.

You can advertise specialized hardware without requiring any upstream code changes.

Device Manager advertises devices as Extended Resources. User pods can consume devices, advertised by Device Manager, using the same Limit/Request mechanism, which is used for requesting any other Extended Resource.

Upon start, the device plugin registers itself with Device Manager invoking Register on the /var/lib/kubelet/device-plugins/kubelet.sock and starts a gRPC service at /var/lib/kubelet/device-plugins/<plugin>.sock for serving Device Manager requests.

Device Manager, while processing a new registration request, invokes ListAndWatch remote procedure call (RPC) at the device plugin service. In response, Device Manager gets a list of Device objects from the plugin over a gRPC stream. Device Manager will keep watching on the stream for new updates from the plugin. On the plugin side, the plugin will also keep the stream open and whenever there is a change in the state of any of the devices, a new device list is sent to the Device Manager over the same streaming connection.

While handling a new pod admission request, Kubelet passes requested Extended Resources to the Device Manager for device allocation. Device Manager checks in its database to verify if a corresponding plugin exists or not. If the plugin exists and there are free allocatable devices as well as per local cache, Allocate RPC is invoked at that particular device plugin.

Additionally, device plugins can also perform several other device-specific operations, such as driver installation, device initialization, and device resets. These functionalities vary from implementation to implementation.

Enabling Device Manager

Enable Device Manager to implement a device plugin to advertise specialized hardware without any upstream code changes.

Device Manager provides a mechanism for advertising specialized node hardware resources with the help of plugins known as device plugins.

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command. Perform one of the following steps:

   1. View the machine config:

      # oc describe machineconfig <name>

      For example:

      # oc describe machineconfig 00-worker

      Example output

      Name:         00-worker
      Namespace:
      Labels:       machineconfiguration.openshift.io/role=worker (1)

      1	Label required for the Device Manager.

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a Device Manager CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: devicemgr (1)
   spec:
     machineConfigPoolSelector:
       matchLabels:
          machineconfiguration.openshift.io: devicemgr (2)
     kubeletConfig:
       feature-gates:
         - DevicePlugins=true (3)

   1	Assign a name to CR.
   2	Enter the label from the Machine Config Pool.
   3	Set DevicePlugins to ‘true`.

2. Create the Device Manager:

   $ oc create -f devicemgr.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/devicemgr created

3. Ensure that Device Manager was actually enabled by confirming that /var/lib/kubelet/device-plugins/kubelet.sock is created on the node. This is the UNIX domain socket on which the Device Manager gRPC server listens for new plugin registrations. This sock file is created when the Kubelet is started only if Device Manager is enabled.

Taints and tolerations

Understand and work with taints and tolerations.

Understanding taints and tolerations

A taint allows a node to refuse a pod to be scheduled unless that pod has a matching toleration.

You apply taints to a node through the Node specification (NodeSpec) and apply tolerations to a pod through the Pod specification (PodSpec). When you apply a taint a node, the scheduler cannot place a pod on that node unless the pod can tolerate the taint.

Example taint in a node specification

spec:
  taints:
  - effect: NoExecute
    key: key1
    value: value1
....

Example toleration in a Pod spec

spec:
  tolerations:
  - key: "key1"
    operator: "Equal"
    value: "value1"
    effect: "NoExecute"
    tolerationSeconds: 3600
....

Taints and tolerations consist of a key, value, and effect.

1. If you add a NoSchedule taint to a control plane node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
   ...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   ...

A toleration matches a taint:

• If the operator parameter is set to Equal:

  • the key parameters are the same;

  • the value parameters are the same;

  • the effect parameters are the same.

• If the operator parameter is set to Exists:

  • the key parameters are the same;

  • the effect parameters are the same.

The following taints are built into OKD:

• node.kubernetes.io/not-ready: The node is not ready. This corresponds to the node condition Ready=False.

• node.kubernetes.io/unreachable: The node is unreachable from the node controller. This corresponds to the node condition Ready=Unknown.

• node.kubernetes.io/memory-pressure: The node has memory pressure issues. This corresponds to the node condition MemoryPressure=True.

• node.kubernetes.io/disk-pressure: The node has disk pressure issues. This corresponds to the node condition DiskPressure=True.

• node.kubernetes.io/network-unavailable: The node network is unavailable.

• node.kubernetes.io/unschedulable: The node is unschedulable.

• node.cloudprovider.kubernetes.io/uninitialized: When the node controller is started with an external cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint.

• node.kubernetes.io/pid-pressure: The node has pid pressure. This corresponds to the node condition PIDPressure=True.

  OKD does not set a default pid.available evictionHard.

Understanding how to use toleration seconds to delay pod evictions

You can specify how long a pod can remain bound to a node before being evicted by specifying the tolerationSeconds parameter in the Pod specification or MachineSet object. If a taint with the NoExecute effect is added to a node, a pod that does tolerate the taint, which has the tolerationSeconds parameter, the pod is not evicted until that time period expires.

Example output

spec:
  tolerations:
  - key: "key1"
    operator: "Equal"
    value: "value1"
    effect: "NoExecute"
    tolerationSeconds: 3600

Here, if this pod is running but does not have a matching toleration, the pod stays bound to the node for 3,600 seconds and then be evicted. If the taint is removed before that time, the pod is not evicted.

Understanding how to use multiple taints

You can put multiple taints on the same node and multiple tolerations on the same pod. OKD processes multiple taints and tolerations as follows:

1. Process the taints for which the pod has a matching toleration.

2. The remaining unmatched taints have the indicated effects on the pod:

   • If there is at least one unmatched taint with effect NoSchedule, OKD cannot schedule a pod onto that node.

   • If there is no unmatched taint with effect NoSchedule but there is at least one unmatched taint with effect PreferNoSchedule, OKD tries to not schedule the pod onto the node.

   • If there is at least one unmatched taint with effect NoExecute, OKD evicts the pod from the node if it is already running on the node, or the pod is not scheduled onto the node if it is not yet running on the node.

     • Pods that do not tolerate the taint are evicted immediately.

     • Pods that tolerate the taint without specifying tolerationSeconds in their Pod specification remain bound forever.

     • Pods that tolerate the taint with a specified tolerationSeconds remain bound for the specified amount of time.

For example:

• Add the following taints to the node:

  $ oc adm taint nodes node1 key1=value1:NoSchedule

  $ oc adm taint nodes node1 key1=value1:NoExecute

  $ oc adm taint nodes node1 key2=value2:NoSchedule

• The pod has the following tolerations:

  spec:
    tolerations:
    - key: "key1"
      operator: "Equal"
      value: "value1"
      effect: "NoSchedule"
    - key: "key1"
      operator: "Equal"
      value: "value1"
      effect: "NoExecute"

In this case, the pod cannot be scheduled onto the node, because there is no toleration matching the third taint. The pod continues running if it is already running on the node when the taint is added, because the third taint is the only one of the three that is not tolerated by the pod.

Understanding pod scheduling and node conditions (taint node by condition)

The Taint Nodes By Condition feature, which is enabled by default, automatically taints nodes that report conditions such as memory pressure and disk pressure. If a node reports a condition, a taint is added until the condition clears. The taints have the NoSchedule effect, which means no pod can be scheduled on the node unless the pod has a matching toleration.

The scheduler checks for these taints on nodes before scheduling pods. If the taint is present, the pod is scheduled on a different node. Because the scheduler checks for taints and not the actual node conditions, you configure the scheduler to ignore some of these node conditions by adding appropriate pod tolerations.

To ensure backward compatibility, the daemon set controller automatically adds the following tolerations to all daemons:

• node.kubernetes.io/memory-pressure

• node.kubernetes.io/disk-pressure

• node.kubernetes.io/unschedulable (1.10 or later)

• node.kubernetes.io/network-unavailable (host network only)

You can also add arbitrary tolerations to daemon sets.

Understanding evicting pods by condition (taint-based evictions)

The Taint-Based Evictions feature, which is enabled by default, evicts pods from a node that experiences specific conditions, such as not-ready and unreachable. When a node experiences one of these conditions, OKD automatically adds taints to the node, and starts evicting and rescheduling the pods on different nodes.

Taint Based Evictions have a NoExecute effect, where any pod that does not tolerate the taint is evicted immediately and any pod that does tolerate the taint will never be evicted, unless the pod uses the tolerationSeconds parameter.

The tolerationSeconds parameter allows you to specify how long a pod stays bound to a node that has a node condition. If the condition still exists after the tolerationSeconds period, the taint remains on the node and the pods with a matching toleration are evicted. If the condition clears before the tolerationSeconds period, pods with matching tolerations are not removed.

If you use the tolerationSeconds parameter with no value, pods are never evicted because of the not ready and unreachable node conditions.

OKD automatically adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300, unless the Pod configuration specifies either toleration.

spec:
  tolerations:
  - key: node.kubernetes.io/not-ready
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 300 (1)
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 300

You can configure these tolerations as needed. For example, if you have an application with a lot of local state, you might want to keep the pods bound to node for a longer time in the event of network partition, allowing for the partition to recover and avoiding pod eviction.

Pods spawned by a daemon set are created with NoExecute tolerations for the following taints with no tolerationSeconds:

• node.kubernetes.io/unreachable

• node.kubernetes.io/not-ready

As a result, daemon set pods are never evicted because of these node conditions.

Tolerating all taints

You can configure a pod to tolerate all taints by adding an operator: "Exists" toleration with no key and value parameters. Pods with this toleration are not removed from a node that has taints.

Pod spec for tolerating all taints

spec:
  tolerations:
  - operator: "Exists"

Adding taints and tolerations

You add tolerations to pods and taints to nodes to allow the node to control which pods should or should not be scheduled on them. For existing pods and nodes, you should add the toleration to the pod first, then add the taint to the node to avoid pods being removed from the node before you can add the toleration.

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with an Equal operator

   spec:
     tolerations:
     - key: "key1" (1)
       value: "value1"
       operator: "Equal"
       effect: "NoExecute"
       tolerationSeconds: 3600 (2)

   1	The toleration parameters, as described in the Taint and toleration components table.
   2	The tolerationSeconds parameter specifies how long a pod can remain bound to a node before being evicted.

   For example:

   Sample pod configuration file with an Exists operator

   spec:
      tolerations:
       - key: "key1"
         operator: "Exists" (1)
         effect: "NoExecute"
         tolerationSeconds: 3600

   1	The Exists operator does not take a value.

   This example places a taint on node1 that has key key1, value value1, and taint effect NoExecute.

2. Add a taint to a node by using the following command with the parameters described in the Taint and toleration components table:

   $ oc adm taint nodes <node_name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 key1=value1:NoExecute

   This command places a taint on node1 that has key key1, value value1, and effect NoExecute.

   If you add a NoSchedule taint to a control plane node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default. For example: apiVersion: v1 kind: Node metadata: annotations: machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0 machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c … spec: taints: - effect: NoSchedule key: node-role.kubernetes.io/master …

   The tolerations on the pod match the taint on the node. A pod with either toleration can be scheduled onto node1.

Adding taints and tolerations using a compute machine set

You can add taints to nodes using a compute machine set. All nodes associated with the MachineSet object are updated with the taint. Tolerations respond to taints added by a compute machine set in the same manner as taints added directly to the nodes.

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with Equal operator

   spec:
     tolerations:
     - key: "key1" (1)
       value: "value1"
       operator: "Equal"
       effect: "NoExecute"
       tolerationSeconds: 3600 (2)

   1	The toleration parameters, as described in the Taint and toleration components table.
   2	The tolerationSeconds parameter specifies how long a pod is bound to a node before being evicted.

   For example:

   Sample pod configuration file with Exists operator

   spec:
     tolerations:
     - key: "key1"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 3600

2. Add the taint to the MachineSet object:

   1. Edit the MachineSet YAML for the nodes you want to taint or you can create a new MachineSet object:

      $ oc edit machineset <machineset>

   2. Add the taint to the spec.template.spec section:

      Example taint in a compute machine set specification

      spec:
      ....
        template:
      ....
          spec:
            taints:
            - effect: NoExecute
              key: key1
              value: value1
      ....

      This example places a taint that has the key key1, value value1, and taint effect NoExecute on the nodes.

   3. Scale down the compute machine set to 0:

      $ oc scale --replicas=0 machineset <machineset> -n openshift-machine-api

      You can alternatively apply the following YAML to scale the compute machine set: apiVersion: machine.openshift.io/v1beta1 kind: MachineSet metadata: name: <machineset> namespace: openshift-machine-api spec: replicas: 0

      Wait for the machines to be removed.

   4. Scale up the compute machine set as needed:

      $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

      Or:

      $ oc edit machineset <machineset> -n openshift-machine-api

      Wait for the machines to start. The taint is added to the nodes associated with the MachineSet object.

Binding a user to a node using taints and tolerations

If you want to dedicate a set of nodes for exclusive use by a particular set of users, add a toleration to their pods. Then, add a corresponding taint to those nodes. The pods with the tolerations are allowed to use the tainted nodes or any other nodes in the cluster.

If you want ensure the pods are scheduled to only those tainted nodes, also add a label to the same set of nodes and add a node affinity to the pods so that the pods can only be scheduled onto nodes with that label.

Procedure

To configure a node so that users can use only that node:

1. Add a corresponding taint to those nodes:

   For example:

   $ oc adm taint nodes node1 dedicated=groupName:NoSchedule

   You can alternatively apply the following YAML to add the taint: kind: Node apiVersion: v1 metadata: name: <node_name> labels: … spec: taints: - key: dedicated value: groupName effect: NoSchedule

2. Add a toleration to the pods by writing a custom admission controller.

Controlling nodes with special hardware using taints and tolerations

In a cluster where a small subset of nodes have specialized hardware, you can use taints and tolerations to keep pods that do not need the specialized hardware off of those nodes, leaving the nodes for pods that do need the specialized hardware. You can also require pods that need specialized hardware to use specific nodes.

You can achieve this by adding a toleration to pods that need the special hardware and tainting the nodes that have the specialized hardware.

Procedure

To ensure nodes with specialized hardware are reserved for specific pods:

1. Add a toleration to pods that need the special hardware.

   For example:

   spec:
     tolerations:
       - key: "disktype"
         value: "ssd"
         operator: "Equal"
         effect: "NoSchedule"
         tolerationSeconds: 3600

2. Taint the nodes that have the specialized hardware using one of the following commands:

   $ oc adm taint nodes <node-name> disktype=ssd:NoSchedule

   Or:

   $ oc adm taint nodes <node-name> disktype=ssd:PreferNoSchedule

   You can alternatively apply the following YAML to add the taint: kind: Node apiVersion: v1 metadata: name: <node_name> labels: … spec: taints: - key: disktype value: ssd effect: PreferNoSchedule

Removing taints and tolerations

You can remove taints from nodes and tolerations from pods as needed. You should add the toleration to the pod first, then add the taint to the node to avoid pods being removed from the node before you can add the toleration.

Procedure

To remove taints and tolerations:

1. To remove a taint from a node:

   $ oc adm taint nodes <node-name> <key>-

   For example:

   $ oc adm taint nodes ip-10-0-132-248.ec2.internal key1-

   Example output

   node/ip-10-0-132-248.ec2.internal untainted

2. To remove a toleration from a pod, edit the Pod spec to remove the toleration:

   spec:
     tolerations:
     - key: "key2"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 3600

Topology Manager

Understand and work with Topology Manager.

Topology Manager policies

Topology Manager aligns Pod resources of all Quality of Service (QoS) classes by collecting topology hints from Hint Providers, such as CPU Manager and Device Manager, and using the collected hints to align the Pod resources.

Topology Manager supports four allocation policies, which you assign in the cpumanager-enabled custom resource (CR):

none policy

This is the default policy and does not perform any topology alignment.

best-effort policy

For each container in a pod with the best-effort topology management policy, kubelet calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager stores the preferred NUMA Node affinity for that container. If the affinity is not preferred, Topology Manager stores this and admits the pod to the node.

restricted policy

For each container in a pod with the restricted topology management policy, kubelet calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager stores the preferred NUMA Node affinity for that container. If the affinity is not preferred, Topology Manager rejects this pod from the node, resulting in a pod in a Terminated state with a pod admission failure.

single-numa-node policy

For each container in a pod with the single-numa-node topology management policy, kubelet calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager determines if a single NUMA Node affinity is possible. If it is, the pod is admitted to the node. If a single NUMA Node affinity is not possible, the Topology Manager rejects the pod from the node. This results in a pod in a Terminated state with a pod admission failure.

Setting up Topology Manager

To use Topology Manager, you must configure an allocation policy in the cpumanager-enabled custom resource (CR). This file might exist if you have set up CPU Manager. If the file does not exist, you can create the file.

Prequisites

• Configure the CPU Manager policy to be static. See the Using CPU Manager in the Scalability and Performance section.

Procedure

To activate Topololgy Manager:

1. Configure the Topology Manager allocation policy in the cpumanager-enabled custom resource (CR).

   $ oc edit KubeletConfig cpumanager-enabled

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: cpumanager-enabled
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: cpumanager-enabled
     kubeletConfig:
        cpuManagerPolicy: static (1)
        cpuManagerReconcilePeriod: 5s
        topologyManagerPolicy: single-numa-node (2)

   1	This parameter must be static with a lowercase s.
   2	Specify your selected Topology Manager allocation policy. Here, the policy is single-numa-node. Acceptable values are: default, best-effort, restricted, single-numa-node.

Pod interactions with Topology Manager policies

The example Pod specs below help illustrate pod interactions with Topology Manager.

The following pod runs in the BestEffort QoS class because no resource requests or limits are specified.

spec:
  containers:
  - name: nginx
    image: nginx

The next pod runs in the Burstable QoS class because requests are less than limits.

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
      requests:
        memory: "100Mi"

If the selected policy is anything other than none, Topology Manager would not consider either of these Pod specifications.

The last example pod below runs in the Guaranteed QoS class because requests are equal to limits.

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "2"
        example.com/device: "1"
      requests:
        memory: "200Mi"
        cpu: "2"
        example.com/device: "1"

Topology Manager would consider this pod. The Topology Manager consults the CPU Manager static policy, which returns the topology of available CPUs. Topology Manager also consults Device Manager to discover the topology of available devices for example.com/device.

Topology Manager will use this information to store the best Topology for this container. In the case of this pod, CPU Manager and Device Manager will use this stored information at the resource allocation stage.

Resource requests and overcommitment

For each compute resource, a container may specify a resource request and limit. Scheduling decisions are made based on the request to ensure that a node has enough capacity available to meet the requested value. If a container specifies limits, but omits requests, the requests are defaulted to the limits. A container is not able to exceed the specified limit on the node.

The enforcement of limits is dependent upon the compute resource type. If a container makes no request or limit, the container is scheduled to a node with no resource guarantees. In practice, the container is able to consume as much of the specified resource as is available with the lowest local priority. In low resource situations, containers that specify no resource requests are given the lowest quality of service.

Scheduling is based on resources requested, while quota and hard limits refer to resource limits, which can be set higher than requested resources. The difference between request and limit determines the level of overcommit; for instance, if a container is given a memory request of 1Gi and a memory limit of 2Gi, it is scheduled based on the 1Gi request being available on the node, but could use up to 2Gi; so it is 200% overcommitted.

Cluster-level overcommit using the Cluster Resource Override Operator

The Cluster Resource Override Operator is an admission webhook that allows you to control the level of overcommit and manage container density across all the nodes in your cluster. The Operator controls how nodes in specific projects can exceed defined memory and CPU limits.

You must install the Cluster Resource Override Operator using the OKD console or CLI as shown in the following sections. During the installation, you create a ClusterResourceOverride custom resource (CR), where you set the level of overcommit, as shown in the following example:

apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
    name: cluster (1)
spec:
  podResourceOverride:
    spec:
      memoryRequestToLimitPercent: 50 (2)
      cpuRequestToLimitPercent: 25 (3)
      limitCPUToMemoryPercent: 200 (4)

When configured, overrides can be enabled per-project by applying the following label to the Namespace object for each project:

apiVersion: v1
kind: Namespace
metadata:
....
  labels:
    clusterresourceoverrides.admission.autoscaling.openshift.io/enabled: "true"
....

The Operator watches for the ClusterResourceOverride CR and ensures that the ClusterResourceOverride admission webhook is installed into the same namespace as the operator.

Installing the Cluster Resource Override Operator using the web console

You can use the OKD web console to install the Cluster Resource Override Operator to help control overcommit in your cluster.

Prerequisites

• The Cluster Resource Override Operator has no effect if limits have not been set on containers. You must specify default limits for a project using a LimitRange object or configure limits in Pod specs for the overrides to apply.

Procedure

To install the Cluster Resource Override Operator using the OKD web console:

1. In the OKD web console, navigate to Home → Projects

   1. Click Create Project.

   2. Specify clusterresourceoverride-operator as the name of the project.

   3. Click Create.

2. Navigate to Operators → OperatorHub.

   1. Choose ClusterResourceOverride Operator from the list of available Operators and click Install.

   2. On the Install Operator page, make sure A specific Namespace on the cluster is selected for Installation Mode.

   3. Make sure clusterresourceoverride-operator is selected for Installed Namespace.

   4. Select an Update Channel and Approval Strategy.

   5. Click Install.

3. On the Installed Operators page, click ClusterResourceOverride.

   1. On the ClusterResourceOverride Operator details page, click Create Instance.

   2. On the Create ClusterResourceOverride page, edit the YAML template to set the overcommit values as needed:

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        name: cluster (1)
      spec:
        podResourceOverride:
          spec:
            memoryRequestToLimitPercent: 50 (2)
            cpuRequestToLimitPercent: 25 (3)
            limitCPUToMemoryPercent: 200 (4)

      1	The name must be cluster.
      2	Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.
      3	Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.
      4	Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

   3. Click Create.

4. Check the current state of the admission webhook by checking the status of the cluster custom resource:

   1. On the ClusterResourceOverride Operator page, click cluster.

   2. On the ClusterResourceOverride Details page, click YAML. The mutatingWebhookConfigurationRef section appears when the webhook is called.

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
        creationTimestamp: "2019-12-18T22:35:02Z"
        generation: 1
        name: cluster
        resourceVersion: "127622"
        selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
        uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
      spec:
        podResourceOverride:
          spec:
            cpuRequestToLimitPercent: 25
            limitCPUToMemoryPercent: 200
            memoryRequestToLimitPercent: 50
      status:
      ....
          mutatingWebhookConfigurationRef: (1)
            apiVersion: admissionregistration.k8s.io/v1beta1
            kind: MutatingWebhookConfiguration
            name: clusterresourceoverrides.admission.autoscaling.openshift.io
            resourceVersion: "127621"
            uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3
      ....

      1	Reference to the ClusterResourceOverride admission webhook.