Post-installation cluster tasks

After installing OKD, you can further expand and customize your cluster to your requirements.

Available cluster customizations

You complete most of the cluster configuration and customization after you deploy your OKD cluster. A number of configuration resources are available.

You modify the configuration resources to configure the major features of the cluster, such as the image registry, networking configuration, image build behavior, and the identity provider.

For current documentation of the settings that you control by using these resources, use the oc explain command, for example oc explain builds --api-version=config.openshift.io/v1

Cluster configuration resources

All cluster configuration resources are globally scoped (not namespaced) and named cluster.

Operator configuration resources

These configuration resources are cluster-scoped instances, named cluster, which control the behavior of a specific component as owned by a particular Operator.

Additional configuration resources

These configuration resources represent a single instance of a particular component. In some cases, you can request multiple instances by creating multiple instances of the resource. In other cases, the Operator can use only a specific resource instance name in a specific namespace. Reference the component-specific documentation for details on how and when you can create additional resource instances.

Informational Resources

You use these resources to retrieve information about the cluster. Some configurations might require you to edit these resources directly.

Updating the global cluster pull secret

You can update the global pull secret for your cluster by either replacing the current pull secret or appending a new pull secret.

The procedure is required when users use a separate registry to store images than the registry used during installation.

Prerequisites

• You have access to the cluster as a user with the cluster-admin role.

Procedure

1. Optional: To append a new pull secret to the existing pull secret, complete the following steps:

   1. Enter the following command to download the pull secret:

      $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> (1)

      1	Provide the path to the pull secret file.

   2. Enter the following command to add the new pull secret:

      $ oc registry login --registry="<registry>" \ (1)
      --auth-basic="<username>:<password>" \ (2)
      --to=<pull_secret_location> (3)

      1	Provide the new registry. You can include multiple repositories within the same registry, for example: —registry=”<registry/my-namespace/my-repository>”.
      2	Provide the credentials of the new registry.
      3	Provide the path to the pull secret file.

      Alternatively, you can perform a manual update to the pull secret file.

2. Enter the following command to update the global pull secret for your cluster:

   $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> (1)

   1	Provide the path to the new pull secret file.

   This update is rolled out to all nodes, which can take some time depending on the size of your cluster.

   As of OKD 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot.

Adding worker nodes

After you deploy your OKD cluster, you can add worker nodes to scale cluster resources. There are different ways you can add worker nodes depending on the installation method and the environment of your cluster.

Adding worker nodes to installer-provisioned infrastructure clusters

For installer-provisioned infrastructure clusters, you can manually or automatically scale the MachineSet object to match the number of available bare-metal hosts.

To add a bare-metal host, you must configure all network prerequisites, configure an associated baremetalhost object, then provision the worker node to the cluster. You can add a bare-metal host manually or by using the web console.

• Adding worker nodes using the web console

• Adding worker nodes using YAML in the web console

• Manually adding a worker node to an installer-provisioned infrastructure cluster

Adding worker nodes to user-provisioned infrastructure clusters

For user-provisioned infrastructure clusters, you can add worker nodes by using a Fedora or FCOS ISO image and connecting it to your cluster using cluster Ignition config files. For RHEL worker nodes, the following example uses Ansible playbooks to add worker nodes to the cluster. For RHCOS worker nodes, the following example uses an ISO image and network booting to add worker nodes to the cluster.

• Adding RHCOS worker nodes to a user-provisioned infrastructure cluster

• Adding RHEL worker nodes to a user-provisioned infrastructure cluster

Adding worker nodes to clusters managed by the Assisted Installer

For clusters managed by the Assisted Installer, you can add worker nodes by using the Red Hat OpenShift Cluster Manager console, the Assisted Installer REST API or you can manually add worker nodes using an ISO image and cluster Ignition config files.

• Adding worker nodes using the OpenShift Cluster Manager

• Adding worker nodes using the Assisted Installer REST API

• Manually adding worker nodes to a SNO cluster

Adding worker nodes to clusters managed by the multicluster engine for Kubernetes

For clusters managed by the multicluster engine for Kubernetes, you can add worker nodes by using the dedicated multicluster engine console.

• Adding worker nodes for clusters managed by the multicluster engine for Kubernetes

Adjust worker nodes

If you incorrectly sized the worker nodes during deployment, adjust them by creating one or more new compute machine sets, scale them up, then scale the original compute machine set down before removing them.

Understanding the difference between compute machine sets and the machine config pool

MachineSet objects describe OKD nodes with respect to the cloud or machine provider.

The MachineConfigPool object allows MachineConfigController components to define and provide the status of machines in the context of upgrades.

The MachineConfigPool object allows users to configure how upgrades are rolled out to the OKD nodes in the machine config pool.

The NodeSelector object can be replaced with a reference to the MachineSet object.

Scaling a compute machine set manually

To add or remove an instance of a machine in a compute machine set, you can manually scale the compute machine set.

This guidance is relevant to fully automated, installer-provisioned infrastructure installations. Customized, user-provisioned infrastructure installations do not have compute machine sets.

Prerequisites

• Install an OKD cluster and the oc command line.

• Log in to oc as a user with cluster-admin permission.

Procedure

1. View the compute machine sets that are in the cluster by running the following command:

   $ oc get machinesets -n openshift-machine-api

   The compute machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. View the compute machines that are in the cluster by running the following command:

   $ oc get machine -n openshift-machine-api

3. Set the annotation on the compute machine that you want to delete by running the following command:

   $ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/delete-machine="true"

4. Cordon and drain the node that you want to delete by running the following commands:

   $ oc adm cordon <node_name>

   $ oc adm drain <node_name>

5. Scale the compute machine set by running one of the following commands:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   You can alternatively apply the following YAML to scale the compute machine set: apiVersion: machine.openshift.io/v1beta1 kind: MachineSet metadata: name: <machineset> namespace: openshift-machine-api spec: replicas: 2

   You can scale the compute machine set up or down. It takes several minutes for the new machines to be available.

Verification

• Verify the deletion of the intended machine by running the following command:

  $ oc get machines

The compute machine set deletion policy

Random, Newest, and Oldest are the three supported deletion options. The default is Random, meaning that random machines are chosen and deleted when scaling compute machine sets down. The deletion policy can be set according to the use case by modifying the particular compute machine set:

spec:
  deletePolicy: <delete_policy>
  replicas: <desired_replica_count>

Specific machines can also be prioritized for deletion by adding the annotation machine.openshift.io/delete-machine=true to the machine of interest, regardless of the deletion policy.

Creating default cluster-wide node selectors

You can use default cluster-wide node selectors on pods together with labels on nodes to constrain all pods created in a cluster to specific nodes.

With cluster-wide node selectors, when you create a pod in that cluster, OKD adds the default node selectors to the pod and schedules the pod on nodes with matching labels.

You configure cluster-wide node selectors by editing the Scheduler Operator custom resource (CR). You add labels to a node, a compute machine set, or a machine config. Adding the label to the compute machine set ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or machine config do not persist if the node or machine goes down.

Procedure

To add a default cluster-wide node selector:

1. Edit the Scheduler Operator CR to add the default cluster-wide node selectors:

   $ oc edit scheduler cluster

   Example Scheduler Operator CR with a node selector

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     name: cluster
   ...
   spec:
     defaultNodeSelector: type=user-node,region=east (1)
     mastersSchedulable: false

   1	Add a node selector with the appropriate <key>:<value> pairs.

   After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.

2. Add labels to a node by using a compute machine set or editing the node directly:

   • Use a compute machine set to add labels to nodes managed by the compute machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api (1)

        1	Add a <key>/<value> pair for each label.

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

        You can alternatively apply the following YAML to add labels to a compute machine set: apiVersion: machine.openshift.io/v1beta1 kind: MachineSet metadata: name: <machineset> namespace: openshift-machine-api spec: template: spec: metadata: labels: region: “east” type: “user-node”

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
          ...
        spec:
          ...
          template:
            metadata:
          ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
          ...

     3. Redeploy the nodes associated with that compute machine set by scaling down to 0 and scaling up the nodes:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. When the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.25.0

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east

        You can alternatively apply the following YAML to add labels to a node: kind: Node apiVersion: v1 metadata: name: <node_name> labels: type: “user-node” region: “east”

     2. Verify that the labels are added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.25.0

Improving cluster stability in high latency environments using worker latency profiles

All nodes send heartbeats to the Kubernetes Controller Manager Operator (kube controller) in the OKD cluster every 10 seconds, by default. If the cluster does not receive heartbeats from a node, OKD responds using several default mechanisms.

For example, if the Kubernetes Controller Manager Operator loses contact with a node after a configured period:

1. The node controller on the control plane updates the node health to Unhealthy and marks the node Ready condition as Unknown.

2. In response, the scheduler stops scheduling pods to that node.

3. The on-premise node controller adds a node.kubernetes.io/unreachable taint with a NoExecute effect to the node and schedules any pods on the node for eviction after five minutes, by default.

This behavior can cause problems if your network is prone to latency issues, especially if you have nodes at the network edge. In some cases, the Kubernetes Controller Manager Operator might not receive an update from a healthy node due to network latency. The Kubernetes Controller Manager Operator would then evict pods from the node even though the node is healthy. To avoid this problem, you can use worker latency profiles to adjust the frequency that the kubelet and the Kubernetes Controller Manager Operator wait for status updates before taking action. These adjustments help to ensure that your cluster runs properly in the event that network latency between the control plane and the worker nodes is not optimal.

These worker latency profiles are three sets of parameters that are pre-defined with carefully tuned values that let you control the reaction of the cluster to latency issues without needing to determine the best values manually.

Understanding worker latency profiles

Worker latency profiles are multiple sets of carefully-tuned values for the node-status-update-frequency, node-monitor-grace-period, default-not-ready-toleration-seconds and default-unreachable-toleration-seconds parameters. These parameters let you control the reaction of the cluster to latency issues without needing to determine the best values manually.

All worker latency profiles configure the following parameters:

• node-status-update-frequency. Specifies the amount of time in seconds that a kubelet updates its status to the Kubernetes Controller Manager Operator.

• node-monitor-grace-period. Specifies the amount of time in seconds that the Kubernetes Controller Manager Operator waits for an update from a kubelet before marking the node unhealthy and adding the node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taint to the node.

• default-not-ready-toleration-seconds. Specifies the amount of time in seconds after marking a node unhealthy that the Kubernetes Controller Manager Operator waits before evicting pods from that node.

• default-unreachable-toleration-seconds. Specifies the amount of time in seconds after marking a node unreachable that the Kubernetes Controller Manager Operator waits before evicting pods from that node.

The following Operators monitor the changes to the worker latency profiles and respond accordingly:

• The Machine Config Operator (MCO) updates the node-status-frequency parameter on the worker nodes.

• The Kubernetes API Server Operator updates the node-monitor-grace-period parameter on the control plane nodes.

• The Kubernetes Controller Manager Operator updates the default-not-ready-toleration-seconds and default-unreachable-toleration-seconds parameters on the control plance nodes.

While the default configuration works in most cases, OKD offers two other worker latency profiles for situations where the network is experiencing higher latency than usual. The three worker latency profiles are described in the following sections:

Default worker latency profile

With the Default profile, each kubelet reports its node status to the Kubelet Controller Manager Operator (kube controller) every 10 seconds. The Kubelet Controller Manager Operator checks the kubelet for a status every 5 seconds.

The Kubernetes Controller Manager Operator waits 40 seconds for a status update before considering that node unhealthy. It marks the node with the node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taint and evicts the pods on that node. If a pod on that node has the NoExecute toleration, the pod gets evicted in 300 seconds. If the pod has the tolerationSeconds parameter, the eviction waits for the period specified by that parameter.

Medium worker latency profile

Use the MediumUpdateAverageReaction profile if the network latency is slightly higher than usual.

The MediumUpdateAverageReaction profile reduces the frequency of kubelet updates to 20 seconds and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 2 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the tolerationSeconds parameter, the eviction waits for the period specified by that parameter.

The Kubernetes Controller Manager Operator waits for 2 minutes to consider a node unhealthy. In another minute, the eviction process starts.

Low worker latency profile

Use the LowUpdateSlowReaction profile if the network latency is extremely high.

The LowUpdateSlowReaction profile reduces the frequency of kubelet updates to 1 minute and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 5 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the tolerationSeconds parameter, the eviction waits for the period specified by that parameter.

The Kubernetes Controller Manager Operator waits for 5 minutes to consider a node unhealthy. In another minute, the eviction process starts.

Using worker latency profiles

To implement a worker latency profile to deal with network latency, edit the node.config object to add the name of the profile. You can change the profile at any time as latency increases or decreases.

You must move one worker latency profile at a time. For example, you cannot move directly from the Default profile to the LowUpdateSlowReaction worker latency profile. You must move from the default worker latency profile to the MediumUpdateAverageReaction profile first, then to LowUpdateSlowReaction. Similarly, when returning to the default profile, you must move from the low profile to the medium profile first, then to the default.

Procedure

To move from the default worker latency profile:

1. Move to the medium worker latency profile:

   1. Edit the node.config object:

      $ oc edit nodes.config/cluster

   2. Add spec.workerLatencyProfile: MediumUpdateAverageReaction:

      Example node.config object

      apiVersion: config.openshift.io/v1
      kind: Node
      metadata:
        annotations:
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        creationTimestamp: "2022-07-08T16:02:51Z"
        generation: 1
        name: cluster
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: 36282574-bf9f-409e-a6cd-3032939293eb
        resourceVersion: "1865"
        uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
      spec:
        workerLatencyProfile: MediumUpdateAverageReaction (1)
       ...

      1	Specifies the medium worker latency policy.

      Scheduling on each worker node is disabled as the change is being applied.

      When all nodes return to the Ready condition, you can use the following command to look in the Kubernetes Controller Manager to ensure it was applied:

      $ oc get KubeControllerManager -o yaml | grep -i workerlatency -A 5 -B 5

      Example output

       ...
          - lastTransitionTime: "2022-07-11T19:47:10Z"
            reason: ProfileUpdated
            status: "False"
            type: WorkerLatencyProfileProgressing
          - lastTransitionTime: "2022-07-11T19:47:10Z" (1)
            message: all static pod revision(s) have updated latency profile
            reason: ProfileUpdated
            status: "True"
            type: WorkerLatencyProfileComplete
          - lastTransitionTime: "2022-07-11T19:20:11Z"
            reason: AsExpected
            status: "False"
            type: WorkerLatencyProfileDegraded
          - lastTransitionTime: "2022-07-11T19:20:36Z"
            status: "False"
       ...

      1	Specifies that the profile is applied and active.

2. Optional: Move to the low worker latency profile:

   1. Edit the node.config object:

      $ oc edit nodes.config/cluster

   2. Change the spec.workerLatencyProfile value to LowUpdateSlowReaction:

      Example node.config object

      apiVersion: config.openshift.io/v1
      kind: Node
      metadata:
        annotations:
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        creationTimestamp: "2022-07-08T16:02:51Z"
        generation: 1
        name: cluster
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: 36282574-bf9f-409e-a6cd-3032939293eb
        resourceVersion: "1865"
        uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
      spec:
        workerLatencyProfile: LowUpdateSlowReaction (1)
       ...

      1	Specifies to use the low worker latency policy.

      Scheduling on each worker node is disabled as the change is being applied.

To change the low profile to medium or change the medium to low, edit the node.config object and set the spec.workerLatencyProfile parameter to the appropriate value.

Managing control plane machines

Control plane machine sets provide management capabilities for control plane machines that are similar to what compute machine sets provide for compute machines. The availability and initial status of control plane machine sets on your cluster depend on your cloud provider and the version of OKD that you installed. For more information, see Getting started with the Control Plane Machine Set Operator.

Creating infrastructure machine sets for production environments

You can create a compute machine set to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.

In a production deployment, it is recommended that you deploy at least three compute machine sets to hold infrastructure components. Both OpenShift Logging and Red Hat OpenShift Service Mesh deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. A configuration like this requires three different compute machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

For information on infrastructure nodes and which components can run on infrastructure nodes, see Creating infrastructure machine sets.

To create an infrastructure node, you can use a machine set, assign a label to the nodes, or use a machine config pool.

For sample machine sets that you can use with these procedures, see Creating machine sets for different clouds.

Applying a specific node selector to all infrastructure components causes OKD to schedule those workloads on nodes with that label.

Creating a compute machine set

In addition to the ones created by the installation program, you can create your own compute machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

• Deploy an OKD cluster.

• Install the OpenShift CLI (oc).

• Log in to oc as a user with cluster-admin permission.

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

   1. If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. Check values of a specific compute machine set:

      $ oc get machineset <machineset_name> -n \
           openshift-machine-api -o yaml

      Example output

      ...
      template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: agl030519-vplxk (1)
              machine.openshift.io/cluster-api-machine-role: worker (2)
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

      1	The cluster ID.
      2	A default node label.

2. Create the new MachineSet CR:

   $ oc create -f <file_name>.yaml

3. View the list of compute machine sets:

   $ oc get machineset -n openshift-machine-api

   Example output

   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
   agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1d   0         0                             55m
   agl030519-vplxk-worker-us-east-1e   0         0                             55m
   agl030519-vplxk-worker-us-east-1f   0         0                             55m

   When the new compute machine set is available, the DESIRED and CURRENT values match. If the compute machine set is not available, wait a few minutes and run the command again.

Creating an infrastructure node

Requirements of the cluster dictate that infrastructure, also called infra nodes, be provisioned. The installer only provides provisions for control plane and worker nodes. Worker nodes can be designated as infrastructure nodes or application, also called app, nodes through labeling.

Procedure

1. Add a label to the worker node that you want to act as application node:

   $ oc label node <node-name> node-role.kubernetes.io/app=""

2. Add a label to the worker nodes that you want to act as infrastructure nodes:

   $ oc label node <node-name> node-role.kubernetes.io/infra=""

3. Check to see if applicable nodes now have the infra role and app roles:

   $ oc get nodes

4. Create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces. This creates an intersection with any existing node selectors on a pod, which additionally constrains the pod’s selector.

   If the default node selector key conflicts with the key of a pod’s label, then the default node selector is not applied. However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as node-role.kubernetes.io/infra=””, when a pod’s label is set to a different node role, such as node-role.kubernetes.io/master=””, can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles. You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts.

   1. Edit the Scheduler object:

      $ oc edit scheduler cluster

   2. Add the defaultNodeSelector field with the appropriate node selector:

      apiVersion: config.openshift.io/v1
      kind: Scheduler
      metadata:
        name: cluster
      ...
      spec:
        defaultNodeSelector: topology.kubernetes.io/region=us-east-1 (1)
      ...

      1	This example node selector deploys pods on nodes in the us-east-1 region by default.

   3. Save the file to apply the changes.

You can now move infrastructure resources to the newly labeled infra nodes.

Additional resources

• For information on how to configure project node selectors to avoid cluster-wide node selector key conflicts, see Project node selectors.

Creating a machine config pool for infrastructure machines

If you need infrastructure machines to have dedicated configurations, you must create an infra pool.

Procedure

1. Add a label to the node you want to assign as the infra node with a specific label:

   $ oc label node <node_name> <label>

   $ oc label node ci-ln-n8mqwr2-f76d1-xscn2-worker-c-6fmtx node-role.kubernetes.io/infra=

2. Create a machine config pool that contains both the worker role and your custom role as machine config selector:

   $ cat infra.mcp.yaml

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: infra
   spec:
     machineConfigSelector:
       matchExpressions:
         - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} (1)
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/infra: "" (2)

   1	Add the worker role and your custom role.
   2	Add the label you added to the node as a nodeSelector.

   Custom machine config pools inherit machine configs from the worker pool. Custom pools use any machine config targeted for the worker pool, but add the ability to also deploy changes that are targeted at only the custom pool. Because a custom pool inherits resources from the worker pool, any change to the worker pool also affects the custom pool.

3. After you have the YAML file, you can create the machine config pool:

   $ oc create -f infra.mcp.yaml

4. Check the machine configs to ensure that the infrastructure configuration rendered successfully:

   $ oc get machineconfig

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   00-worker                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-1ae2a1e0-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-ssh                                                                                          3.2.0             31d
   99-worker-1ae64748-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-worker-ssh                                                                                          3.2.0             31d
   rendered-infra-4e48906dca84ee702959c71a53ee80e7             365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             23m
   rendered-master-072d4b2da7f88162636902b074e9e28e            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-master-3e88ec72aed3886dec061df60d16d1af            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-master-419bee7de96134963a15fdf9dd473b25            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-master-53f5c91c7661708adce18739cc0f40fb            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d
   rendered-master-a6a357ec18e5bce7f5ac426fc7c5ffcd            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-master-dc7f874ec77fc4b969674204332da037            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-1a75960c52ad18ff5dfa6674eb7e533d            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-2640531be11ba43c61d72e82dc634ce6            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-4e48906dca84ee702959c71a53ee80e7            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-worker-4f110718fe88e5f349987854a1147755            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-worker-afc758e194d6188677eb837842d3b379            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-worker-daa08cc1e8f5fcdeba24de60cd955cc3            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d

   You should see a new machine config, with the rendered-infra-* prefix.

5. Optional: To deploy changes to a custom pool, create a machine config that uses the custom pool name as the label, such as infra. Note that this is not required and only shown for instructional purposes. In this manner, you can apply any custom configurations specific to only your infra nodes.

   After you create the new machine config pool, the MCO generates a new rendered config for that pool, and associated nodes of that pool reboot to apply the new configuration.

   1. Create a machine config:

      $ cat infra.mc.yaml

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        name: 51-infra
        labels:
          machineconfiguration.openshift.io/role: infra (1)
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - path: /etc/infratest
              mode: 0644
              contents:
                source: data:,infra

      1	Add the label you added to the node as a nodeSelector.

   2. Apply the machine config to the infra-labeled nodes:

      $ oc create -f infra.mc.yaml

6. Confirm that your new machine config pool is available:

   $ oc get mcp

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   infra    rendered-infra-60e35c2e99f42d976e084fa94da4d0fc    True      False      False      1              1                   1                     0                      4m20s
   master   rendered-master-9360fdb895d4c131c7c4bebbae099c90   True      False      False      3              3                   3                     0                      91m
   worker   rendered-worker-60e35c2e99f42d976e084fa94da4d0fc   True      False      False      2              2                   2                     0                      91m

   In this example, a worker node was changed to an infra node.

Additional resources

• See Node configuration management with machine config pools for more information on grouping infra machines in a custom pool.

Assigning machine set resources to infrastructure nodes

After creating an infrastructure machine set, the worker and infra roles are applied to new infra nodes. Nodes with the infra role are not counted toward the total number of subscriptions that are required to run the environment, even when the worker role is also applied.

However, when an infra node is assigned the worker role, there is a chance that user workloads can get assigned inadvertently to the infra node. To avoid this, you can apply a taint to the infra node and tolerations for the pods that you want to control.

Binding infrastructure node workloads using taints and tolerations

If you have an infra node that has the infra and worker roles assigned, you must configure the node so that user workloads are not assigned to it.

Prerequisites

• Configure additional MachineSet objects in your OKD cluster.

Procedure

1. Add a taint to the infra node to prevent scheduling user workloads on it:

   1. Determine if the node has the taint:

      $ oc describe nodes <node_name>

      Sample output

      oc describe node ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Name:               ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Roles:              worker
       ...
      Taints:             node-role.kubernetes.io/infra:NoSchedule
       ...

      This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.

   2. If you have not configured a taint to prevent scheduling user workloads on it:

      $ oc adm taint nodes <node_name> <key>:<effect>

      For example:

      $ oc adm taint nodes node1 node-role.kubernetes.io/infra:NoSchedule

      You can alternatively apply the following YAML to add the taint: kind: Node apiVersion: v1 metadata: name: <node_name> labels: … spec: taints: - key: node-role.kubernetes.io/infra effect: NoSchedule …

      This example places a taint on node1 that has key node-role.kubernetes.io/infra and taint effect NoSchedule. Nodes with the NoSchedule effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.

      If a descheduler is used, pods violating node taints could be evicted from the cluster.

2. Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the Pod object specification:

   tolerations:
     - effect: NoSchedule (1)
       key: node-role.kubernetes.io/infra (2)
       operator: Exists (3)

   1	Specify the effect that you added to the node.
   2	Specify the key that you added to the node.
   3	Specify the Exists Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node.

   This toleration matches the taint created by the oc adm taint command. A pod with this toleration can be scheduled onto the infra node.

   Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.

3. Schedule the pod to the infra node using a scheduler. See the documentation for Controlling pod placement onto nodes for details.

Additional resources

• See Controlling pod placement using the scheduler for general information on scheduling a pod to a node.

Moving resources to infrastructure machine sets

Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created.

Moving the router

You can deploy the router pod to a different compute machine set. By default, the pod is deployed to a worker node.

Prerequisites

• Configure additional compute machine sets in your OKD cluster.

Procedure

1. View the IngressController custom resource for the router Operator:

   $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

   The command output resembles the following text:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: 2019-04-18T12:35:39Z
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
     namespace: openshift-ingress-operator
     resourceVersion: "11341"
     selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a
   spec: {}
   status:
     availableReplicas: 2
     conditions:
     - lastTransitionTime: 2019-04-18T12:36:15Z
       status: "True"
       type: Available
     domain: apps.<cluster>.example.com
     endpointPublishingStrategy:
       type: LoadBalancerService
     selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

   $ oc edit ingresscontroller default -n openshift-ingress-operator

     spec:
       nodePlacement:
         nodeSelector: (1)
           matchLabels:
             node-role.kubernetes.io/infra: ""
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved

   1	Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.

3. Confirm that the router pod is running on the infra node.

   1. View the list of router pods and note the node name of the running pod:

      $ oc get pod -n openshift-ingress -o wide

      Example output

      NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

   2. View the node status of the running pod:

      $ oc get node <node_name> (1)

      1	Specify the <node_name> that you obtained from the pod list.

      Example output

      NAME                          STATUS  ROLES         AGE   VERSION
      ip-10-0-217-226.ec2.internal  Ready   infra,worker  17h   v1.25.0

      Because the role list includes infra, the pod is running on the correct node.

Moving the default registry

You configure the registry Operator to deploy its pods to different nodes.

Prerequisites

• Configure additional compute machine sets in your OKD cluster.

Procedure

1. View the config/instance object:

   $ oc get configs.imageregistry.operator.openshift.io/cluster -o yaml

   Example output

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     creationTimestamp: 2019-02-05T13:52:05Z
     finalizers:
     - imageregistry.operator.openshift.io/finalizer
     generation: 1
     name: cluster
     resourceVersion: "56174"
     selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
     uid: 36fd3724-294d-11e9-a524-12ffeee2931b
   spec:
     httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
     logging: 2
     managementState: Managed
     proxy: {}
     replicas: 1
     requests:
       read: {}
       write: {}
     storage:
       s3:
         bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
         region: us-east-1
   status:
   ...

2. Edit the config/instance object:

   $ oc edit configs.imageregistry.operator.openshift.io/cluster

   spec:
     affinity:
       podAntiAffinity:
         preferredDuringSchedulingIgnoredDuringExecution:
         - podAffinityTerm:
             namespaces:
             - openshift-image-registry
             topologyKey: kubernetes.io/hostname
           weight: 100
     logLevel: Normal
     managementState: Managed
     nodeSelector: (1)
       node-role.kubernetes.io/infra: ""
     tolerations:
     - effect: NoSchedule
       key: node-role.kubernetes.io/infra
       value: reserved
     - effect: NoExecute
       key: node-role.kubernetes.io/infra
       value: reserved

   1	Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

3. Verify the registry pod has been moved to the infrastructure node.

   1. Run the following command to identify the node where the registry pod is located:

      $ oc get pods -o wide -n openshift-image-registry

   2. Confirm the node has the label you specified:

      $ oc describe node <node_name>

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Moving the monitoring solution

The monitoring stack includes multiple components, including Prometheus, Thanos Querier, and Alertmanager. The Cluster Monitoring Operator manages this stack. To redeploy the monitoring stack to infrastructure nodes, you can create and apply a custom config map.

Procedure

1. Edit the cluster-monitoring-config config map and change the nodeSelector to use the infra label:

   $ oc edit configmap cluster-monitoring-config -n openshift-monitoring

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |+
       alertmanagerMain:
         nodeSelector: (1)
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       prometheusK8s:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       prometheusOperator:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       k8sPrometheusAdapter:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       kubeStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       telemeterClient:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       openshiftStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       thanosQuerier:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute

   1	Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

2. Watch the monitoring pods move to the new machines:

   $ watch 'oc get pod -n openshift-monitoring -o wide'

3. If a component has not moved to the infra node, delete the pod with this component:

   $ oc delete pod -n openshift-monitoring <pod>

   The component from the deleted pod is re-created on the infra node.

Moving OpenShift Logging resources

You can configure the Cluster Logging Operator to deploy the pods for logging subsystem components, such as Elasticsearch and Kibana, to different nodes. You cannot move the Cluster Logging Operator pod from its installed location.

For example, you can move the Elasticsearch pods to a separate node because of high CPU, memory, and disk requirements.

Prerequisites

• The Red Hat OpenShift Logging and Elasticsearch Operators must be installed. These features are not installed by default.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   ...
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: (1)
           node-role.kubernetes.io/infra: ''
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: (1)
           node-role.kubernetes.io/infra: ''
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   ...

   1	Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

Verification

To verify that a component has moved, you can use the oc get pod -o wide command.

For example:

• You want to move the Kibana pod from the ip-10-0-147-79.us-east-2.compute.internal node:

  $ oc get pod kibana-5b8bdf44f9-ccpq9 -o wide

  Example output

  NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
  kibana-5b8bdf44f9-ccpq9   2/2     Running   0          27s   10.129.2.18   ip-10-0-147-79.us-east-2.compute.internal   <none>           <none>

• You want to move the Kibana pod to the ip-10-0-139-48.us-east-2.compute.internal node, a dedicated infrastructure node:

  $ oc get nodes

  Example output

  NAME                                         STATUS   ROLES          AGE   VERSION
  ip-10-0-133-216.us-east-2.compute.internal   Ready    master         60m   v1.25.0
  ip-10-0-139-146.us-east-2.compute.internal   Ready    master         60m   v1.25.0
  ip-10-0-139-192.us-east-2.compute.internal   Ready    worker         51m   v1.25.0
  ip-10-0-139-241.us-east-2.compute.internal   Ready    worker         51m   v1.25.0
  ip-10-0-147-79.us-east-2.compute.internal    Ready    worker         51m   v1.25.0
  ip-10-0-152-241.us-east-2.compute.internal   Ready    master         60m   v1.25.0
  ip-10-0-139-48.us-east-2.compute.internal    Ready    infra          51m   v1.25.0

  Note that the node has a node-role.kubernetes.io/infra: '' label:

  $ oc get node ip-10-0-139-48.us-east-2.compute.internal -o yaml

  Example output

  kind: Node
  apiVersion: v1
  metadata:
    name: ip-10-0-139-48.us-east-2.compute.internal
    selfLink: /api/v1/nodes/ip-10-0-139-48.us-east-2.compute.internal
    uid: 62038aa9-661f-41d7-ba93-b5f1b6ef8751
    resourceVersion: '39083'
    creationTimestamp: '2020-04-13T19:07:55Z'
    labels:
      node-role.kubernetes.io/infra: ''
  ...

• To move the Kibana pod, edit the ClusterLogging CR to add a node selector:

  apiVersion: logging.openshift.io/v1
  kind: ClusterLogging
  ...
  spec:
  ...
    visualization:
      kibana:
        nodeSelector: (1)
          node-role.kubernetes.io/infra: ''
        proxy:
          resources: null
        replicas: 1
        resources: null
      type: kibana

  1	Add a node selector to match the label in the node specification.

• After you save the CR, the current Kibana pod is terminated and new pod is deployed:

  $ oc get pods

  Example output

  NAME                                            READY   STATUS        RESTARTS   AGE
  cluster-logging-operator-84d98649c4-zb9g7       1/1     Running       0          29m
  elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg   2/2     Running       0          28m
  elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj   2/2     Running       0          28m
  elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78    2/2     Running       0          28m
  fluentd-42dzz                                   1/1     Running       0          28m
  fluentd-d74rq                                   1/1     Running       0          28m
  fluentd-m5vr9                                   1/1     Running       0          28m
  fluentd-nkxl7                                   1/1     Running       0          28m
  fluentd-pdvqb                                   1/1     Running       0          28m
  fluentd-tflh6                                   1/1     Running       0          28m
  kibana-5b8bdf44f9-ccpq9                         2/2     Terminating   0          4m11s
  kibana-7d85dcffc8-bfpfp                         2/2     Running       0          33s

• The new pod is on the ip-10-0-139-48.us-east-2.compute.internal node:

  $ oc get pod kibana-7d85dcffc8-bfpfp -o wide

  Example output

  NAME                      READY   STATUS        RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
  kibana-7d85dcffc8-bfpfp   2/2     Running       0          43s   10.131.0.22   ip-10-0-139-48.us-east-2.compute.internal   <none>           <none>

• After a few moments, the original Kibana pod is removed.

  $ oc get pods

  Example output

  NAME                                            READY   STATUS    RESTARTS   AGE
  cluster-logging-operator-84d98649c4-zb9g7       1/1     Running   0          30m
  elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg   2/2     Running   0          29m
  elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj   2/2     Running   0          29m
  elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78    2/2     Running   0          29m
  fluentd-42dzz                                   1/1     Running   0          29m
  fluentd-d74rq                                   1/1     Running   0          29m
  fluentd-m5vr9                                   1/1     Running   0          29m
  fluentd-nkxl7                                   1/1     Running   0          29m
  fluentd-pdvqb                                   1/1     Running   0          29m
  fluentd-tflh6                                   1/1     Running   0          29m
  kibana-7d85dcffc8-bfpfp                         2/2     Running   0          62s

About the cluster autoscaler

The cluster autoscaler adjusts the size of an OKD cluster to meet its current deployment needs. It uses declarative, Kubernetes-style arguments to provide infrastructure management that does not rely on objects of a specific cloud provider. The cluster autoscaler has a cluster scope, and is not associated with a particular namespace.

The cluster autoscaler increases the size of the cluster when there are pods that fail to schedule on any of the current worker nodes due to insufficient resources or when another node is necessary to meet deployment needs. The cluster autoscaler does not increase the cluster resources beyond the limits that you specify.

The cluster autoscaler computes the total memory, CPU, and GPU on all nodes the cluster, even though it does not manage the control plane nodes. These values are not single-machine oriented. They are an aggregation of all the resources in the entire cluster. For example, if you set the maximum memory resource limit, the cluster autoscaler includes all the nodes in the cluster when calculating the current memory usage. That calculation is then used to determine if the cluster autoscaler has the capacity to add more worker resources.

Every 10 seconds, the cluster autoscaler checks which nodes are unnecessary in the cluster and removes them. The cluster autoscaler considers a node for removal if the following conditions apply:

• The node utilization is less than the node utilization level threshold for the cluster. The node utilization level is the sum of the requested resources divided by the allocated resources for the node. If you do not specify a value in the ClusterAutoscaler custom resource, the cluster autoscaler uses a default value of 0.5, which corresponds to 50% utilization.

• The cluster autoscaler can move all pods running on the node to the other nodes.

• The cluster autoscaler does not have scale down disabled annotation.

If the following types of pods are present on a node, the cluster autoscaler will not remove the node:

• Pods with restrictive pod disruption budgets (PDBs).

• Kube-system pods that do not run on the node by default.

• Kube-system pods that do not have a PDB or have a PDB that is too restrictive.

• Pods that are not backed by a controller object such as a deployment, replica set, or stateful set.

• Pods with local storage.

• Pods that cannot be moved elsewhere because of a lack of resources, incompatible node selectors or affinity, matching anti-affinity, and so on.

• Unless they also have a "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation, pods that have a "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation.

For example, you set the maximum CPU limit to 64 cores and configure the cluster autoscaler to only create machines that have 8 cores each. If your cluster starts with 30 cores, the cluster autoscaler can add up to 4 more nodes with 32 cores, for a total of 62.

If you configure the cluster autoscaler, additional usage restrictions apply:

• Do not modify the nodes that are in autoscaled node groups directly. All nodes within the same node group have the same capacity and labels and run the same system pods.

• Specify requests for your pods.

• If you have to prevent pods from being deleted too quickly, configure appropriate PDBs.

• Confirm that your cloud provider quota is large enough to support the maximum node pools that you configure.

• Do not run additional node group autoscalers, especially the ones offered by your cloud provider.

The horizontal pod autoscaler (HPA) and the cluster autoscaler modify cluster resources in different ways. The HPA changes the deployment’s or replica set’s number of replicas based on the current CPU load. If the load increases, the HPA creates new replicas, regardless of the amount of resources available to the cluster. If there are not enough resources, the cluster autoscaler adds resources so that the HPA-created pods can run. If the load decreases, the HPA stops some replicas. If this action causes some nodes to be underutilized or completely empty, the cluster autoscaler deletes the unnecessary nodes.

The cluster autoscaler takes pod priorities into account. The Pod Priority and Preemption feature enables scheduling pods based on priorities if the cluster does not have enough resources, but the cluster autoscaler ensures that the cluster has resources to run all pods. To honor the intention of both features, the cluster autoscaler includes a priority cutoff function. You can use this cutoff to schedule “best-effort” pods, which do not cause the cluster autoscaler to increase resources but instead run only when spare resources are available.

Pods with priority lower than the cutoff value do not cause the cluster to scale up or prevent the cluster from scaling down. No new nodes are added to run the pods, and nodes running these pods might be deleted to free resources.

ClusterAutoscaler resource definition

This ClusterAutoscaler resource definition shows the parameters and sample values for the cluster autoscaler.

apiVersion: "autoscaling.openshift.io/v1"
kind: "ClusterAutoscaler"
metadata:
  name: "default"
spec:
  podPriorityThreshold: -10 (1)
  resourceLimits:
    maxNodesTotal: 24 (2)
    cores:
      min: 8 (3)
      max: 128 (4)
    memory:
      min: 4 (5)
      max: 256 (6)
    gpus:
      - type: nvidia.com/gpu (7)
        min: 0 (8)
        max: 16 (9)
      - type: amd.com/gpu
        min: 0
        max: 4
  logVerbosity: 4 (10)
  scaleDown: (11)
    enabled: true (12)
    delayAfterAdd: 10m (13)
    delayAfterDelete: 5m (14)
    delayAfterFailure: 30s (15)
    unneededTime: 5m (16)
    utilizationThreshold: "0.4" (17)

Deploying the cluster autoscaler

To deploy the cluster autoscaler, you create an instance of the ClusterAutoscaler resource.

Procedure

1. Create a YAML file for the ClusterAutoscaler resource that contains the customized resource definition.

2. Create the resource in the cluster:

   $ oc create -f <filename>.yaml (1)

   1	<filename> is the name of the resource file that you customized.

About the machine autoscaler

The machine autoscaler adjusts the number of Machines in the compute machine sets that you deploy in an OKD cluster. You can scale both the default worker compute machine set and any other compute machine sets that you create. The machine autoscaler makes more Machines when the cluster runs out of resources to support more deployments. Any changes to the values in MachineAutoscaler resources, such as the minimum or maximum number of instances, are immediately applied to the compute machine set they target.

MachineAutoscaler resource definition

This MachineAutoscaler resource definition shows the parameters and sample values for the machine autoscaler.

apiVersion: "autoscaling.openshift.io/v1beta1"
kind: "MachineAutoscaler"
metadata:
  name: "worker-us-east-1a" (1)
  namespace: "openshift-machine-api"
spec:
  minReplicas: 1 (2)
  maxReplicas: 12 (3)
  scaleTargetRef: (4)
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet (5)
    name: worker-us-east-1a (6)

Deploying the machine autoscaler

To deploy the machine autoscaler, you create an instance of the MachineAutoscaler resource.

Procedure

1. Create a YAML file for the MachineAutoscaler resource that contains the customized resource definition.

2. Create the resource in the cluster:

   $ oc create -f <filename>.yaml (1)

   1	<filename> is the name of the resource file that you customized.

Configuring Linux cgroup

By default, OKD uses Linux control group version 2 (cgroup v2) in your cluster. You can switch to Linux control group version 1 (cgroup v1), if needed.

cgroup v2 is the next version of the kernel control group and offers multiple improvements. However, it can have some unwanted effects on your nodes.

You can configure whether your cluster uses cgroup v1 or cgroup v2 by editing the node.config object. Enabling the other version of cgroup in OKD disables the current cgroup controllers and hierarchies in your cluster.

Prerequisites

• Have administrative privilege to a working OKD cluster.

Procedure

1. Create a MachineConfig object file that identifies the kernel argument (for example, worker-cgroup-v1.yaml)

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker (1)
     name: worker-cgroup-v1 (2)
   spec:
     config:
       ignition:
         version: 3.2.0
     kernelArguments:
       - systemd.unified_cgroup_hierarchy=0 (3)

   1	Applies the new kernel argument only to worker nodes.
   2	Applies a name to the machine config.
   3	Configures cgroup v1 on the associated nodes.

2. Create the new machine config:

   $ oc create -f 05-worker-cgroup-v1.yaml

3. Check to see that the new machine config was added:

   $ oc get MachineConfig

   Example output

   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-worker-cgroup-v1                                                                           3.2.0             105s
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   rendered-master-c5e92d98103061c4818cfcefcf462770   60746a843e7ef8855ae00f2ffcb655c53e0e8296   3.2.0             115s
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m

4. Check the nodes:

   $ oc get nodes

   Example output

   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.25.0
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.25.0
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.25.0
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.25.0
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.25.0
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.25.0

   You can see that The command disables scheduling on each worker node.

5. Check that the sys/fs/cgroup/cgroup2fs file has been moved to the tmpfs file system:

   $ stat -c %T -f /sys/fs/cgroup

   Example output

   tmpfs