Post-installation cluster tasks

After installing OKD, you can further expand and customize your cluster to your requirements.

Adjust worker nodes

If you incorrectly sized the worker nodes during deployment, adjust them by creating one or more new machine sets, scale them up, then scale the original machine set down before removing them.

Understanding the difference between machine sets and the machine config pool

MachineSet objects describe OKD nodes with respect to the cloud or machine provider.

The MachineConfigPool object allows MachineConfigController components to define and provide the status of machines in the context of upgrades.

The MachineConfigPool object allows users to configure how upgrades are rolled out to the OKD nodes in the machine config pool.

The NodeSelector object can be replaced with a reference to the MachineSet object.

Scaling a machine set manually

To add or remove an instance of a machine in a machine set, you can manually scale the machine set.

This guidance is relevant to fully automated, installer-provisioned infrastructure installations. Customized, user-provisioned infrastructure installations do not have machine sets.

Prerequisites

  • Install an OKD cluster and the oc command line.

  • Log in to oc as a user with cluster-admin permission.

Procedure

  1. View the machine sets that are in the cluster:

    1. $ oc get machinesets -n openshift-machine-api

    The machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

  2. Scale the machine set:

    1. $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

    Or:

    1. $ oc edit machineset <machineset> -n openshift-machine-api

    You can scale the machine set up or down. It takes several minutes for the new machines to be available.

The machine set deletion policy

Random, Newest, and Oldest are the three supported deletion options. The default is Random, meaning that random machines are chosen and deleted when scaling machine sets down. The deletion policy can be set according to the use case by modifying the particular machine set:

  1. spec:
  2.   deletePolicy: <delete_policy>
  3.   replicas: <desired_replica_count>

Specific machines can also be prioritized for deletion by adding the annotation machine.openshift.io/cluster-api-delete-machine to the machine of interest, regardless of the deletion policy.

By default, the OKD router pods are deployed on workers. Because the router is required to access some cluster resources, including the web console, do not scale the worker machine set to 0 unless you first relocate the router pods.

Custom machine sets can be used for use cases requiring that services run on specific nodes and that those services are ignored by the controller when the worker machine sets are scaling down. This prevents service disruption.

Creating default cluster-wide node selectors

You can use default cluster-wide node selectors on pods together with labels on nodes to constrain all pods created in a cluster to specific nodes.

With cluster-wide node selectors, when you create a pod in that cluster, OKD adds the default node selectors to the pod and schedules the pod on nodes with matching labels.

You configure cluster-wide node selectors by editing the Scheduler Operator custom resource (CR). You add labels to a node, a machine set, or a machine config. Adding the label to the machine set ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or machine config do not persist if the node or machine goes down.

You can add additional key/value pairs to a pod. But you cannot add a different value for a default key.

Procedure

To add a default cluster-wide node selector:

  1. Edit the Scheduler Operator CR to add the default cluster-wide node selectors:

    1. $ oc edit scheduler cluster

    Example Scheduler Operator CR with a node selector

    1. apiVersion: config.openshift.io/v1
    2. kind: Scheduler
    3. metadata:
    4.   name: cluster
    5. ...
    7. spec:
    8.   defaultNodeSelector: type=user-node,region=east (1)
    9.   mastersSchedulable: false
    10.   policy:
    11.     name: ""
    1Add a node selector with the appropriate <key>:<value> pairs.

    After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.

  2. Add labels to a node by using a machine set or editing the node directly:

    • Use a machine set to add labels to nodes managed by the machine set when a node is created:

      1. Run the following command to add labels to a MachineSet object:

        1. $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api (1)
        1Add a <key>/<value> pair for each label.

        For example:

        1. $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

      2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        1. $ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        Example output

        1. apiVersion: machine.openshift.io/v1beta1
        2. kind: MachineSet
        3. metadata:
        4. ...
        5. spec:
        6. ...
        7.   template:
        8.     metadata:
        9. ...
        10.     spec:
        11.       metadata:
        12.         labels:
        13.           region: east
        14.           type: user-node

      3. Redeploy the nodes associated with that machine set by scaling down to 0 and scaling up the nodes:

        For example:

        1. $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
        1. $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

      4. When the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:

        1. $ oc get nodes -l <key>=<value>

        For example:

        1. $ oc get nodes -l type=user-node

        Example output

        1. NAME                                       STATUS   ROLES    AGE   VERSION
        2. ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f

    • Add labels directly to a node:

      1. Edit the Node object for the node:

        1. $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        1. $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east

      2. Verify that the labels are added to the node using the oc get command:

        1. $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        1. $ oc get nodes -l type=user-node,region=east

        Example output

        1. NAME                                       STATUS   ROLES    AGE   VERSION
        2. ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f

Creating infrastructure machine sets for production environments

In a production deployment, deploy at least three machine sets to hold infrastructure components. Both the logging aggregation solution and the service mesh deploy Elasticsearch, and Elasticsearch requires three instances that are installed on different nodes. For high availability, install deploy these nodes to different availability zones. Since you need different machine sets for each availability zone, create at least three machine sets.

For sample machine sets that you can use with these procedures, see Creating machine sets for different clouds.

Creating a machine set

In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

  • Deploy an OKD cluster.

  • Install the OpenShift CLI (oc).

  • Log in to oc as a user with cluster-admin permission.

Procedure

  1. Create a new YAML file that contains the machine set custom resource (CR) sample and is named <file_name>.yaml.

    Ensure that you set the <clusterID> and <role> parameter values.

    1. If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

      1. $ oc get machinesets -n openshift-machine-api

      Example output

      1. NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      2. agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      3. agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      4. agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      5. agl030519-vplxk-worker-us-east-1d   0         0                             55m
      6. agl030519-vplxk-worker-us-east-1e   0         0                             55m
      7. agl030519-vplxk-worker-us-east-1f   0         0                             55m

    2. Check values of a specific machine set:

      1. $ oc get machineset <machineset_name> -n \
      2.      openshift-machine-api -o yaml

      Example output

      1. ...
      2. template:
      3.     metadata:
      4.       labels:
      5.         machine.openshift.io/cluster-api-cluster: agl030519-vplxk (1)
      6.         machine.openshift.io/cluster-api-machine-role: worker (2)
      7.         machine.openshift.io/cluster-api-machine-type: worker
      8.         machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a
      1The cluster ID.
      2A default node label.

  2. Create the new MachineSet CR:

    1. $ oc create -f <file_name>.yaml

  3. View the list of machine sets:

    1. $ oc get machineset -n openshift-machine-api

    Example output

    1. NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
    2. agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
    3. agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
    4. agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
    5. agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
    6. agl030519-vplxk-worker-us-east-1d   0         0                             55m
    7. agl030519-vplxk-worker-us-east-1e   0         0                             55m
    8. agl030519-vplxk-worker-us-east-1f   0         0                             55m

    When the new machine set is available, the DESIRED and CURRENT values match. If the machine set is not available, wait a few minutes and run the command again.

Creating an infrastructure node

See Creating infrastructure machine sets for installer-provisioned infrastructure environments or for any cluster where the control plane nodes (also known as the master nodes) are managed by the machine API.

Requirements of the cluster dictate that infrastructure, also called infra nodes, be provisioned. The installer only provides provisions for control plane and worker nodes. Worker nodes can be designated as infrastructure nodes or application, also called app, nodes through labeling.

Procedure

  1. Add a label to the worker node that you want to act as application node:

    1. $ oc label node <node-name> node-role.kubernetes.io/app=""

  2. Add a label to the worker nodes that you want to act as infrastructure nodes:

    1. $ oc label node <node-name> node-role.kubernetes.io/infra=""

  3. Check to see if applicable nodes now have the infra role and app roles:

    1. $ oc get nodes

  4. Create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces. This creates an intersection with any existing node selectors on a pod, which additionally constrains the pod’s selector.

    If the default node selector key conflicts with the key of a pod’s label, then the default node selector is not applied.

    However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as node-role.kubernetes.io/infra=””, when a pod’s label is set to a different node role, such as node-role.kubernetes.io/master=””, can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles.

    You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts.

    1. Edit the Scheduler object:

      1. $ oc edit scheduler cluster

    2. Add the defaultNodeSelector field with the appropriate node selector:

      1. apiVersion: config.openshift.io/v1
      2. kind: Scheduler
      3. metadata:
      4.   name: cluster
      5. ...
      6. spec:
      7.   defaultNodeSelector: topology.kubernetes.io/region=us-east-1 (1)
      8. ...
      1This example node selector deploys pods on nodes in the us-east-1 region by default.

    3. Save the file to apply the changes.

  5. Move infrastructure resources to the newly labeled infra nodes.

Additional resources

  • For information on how to configure project node selectors to avoid cluster-wide node selector key conflicts, see Project node selectors.

Creating a machine config pool for infrastructure machines

If you need infrastructure machines to have dedicated configurations, you must create an infra pool.

Procedure

  1. Add a label to the node you want to assign as the infra node with a specific label:

    1. $ oc label node <node_name> <label>
    1. $ oc label node ci-ln-n8mqwr2-f76d1-xscn2-worker-c-6fmtx node-role.kubernetes.io/infra=

  2. Create a machine config pool that contains both the worker role and your custom role as machine config selector:

    1. $ cat infra.mcp.yaml

    Example output

    1. apiVersion: machineconfiguration.openshift.io/v1
    2. kind: MachineConfigPool
    3. metadata:
    4.   name: infra
    5. spec:
    6.   machineConfigSelector:
    7.     matchExpressions:
    8.       - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} (1)
    9.   nodeSelector:
    10.     matchLabels:
    11.       node-role.kubernetes.io/infra: "" (2)
    1Add the worker role and your custom role.
    2Add the label you added to the node as a nodeSelector.

    Custom machine config pools inherit machine configs from the worker pool. Custom pools use any machine config targeted for the worker pool, but add the ability to also deploy changes that are targeted at only the custom pool. Because a custom pool inherits resources from the worker pool, any change to the worker pool also affects the custom pool.

  3. After you have the YAML file, you can create the machine config pool:

    1. $ oc create -f infra.mcp.yaml

  4. Check the machine configs to ensure that the infrastructure configuration rendered successfully:

    1. $ oc get machineconfig

    Example output

    1. NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
    2. 00-master                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    3. 00-worker                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    4. 01-master-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    5. 01-master-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    6. 01-worker-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    7. 01-worker-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    8. 99-master-1ae2a1e0-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    9. 99-master-ssh                                                                                          2.2.0             31d
    10. 99-worker-1ae64748-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             31d
    11. 99-worker-ssh                                                                                          2.2.0             31d
    12. rendered-infra-4e48906dca84ee702959c71a53ee80e7             365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             19s
    13. rendered-master-072d4b2da7f88162636902b074e9e28e            5b6fb8349a29735e48446d435962dec4547d3090   2.2.0             31d
    14. rendered-master-3e88ec72aed3886dec061df60d16d1af            02c07496ba0417b3e12b78fb32baf6293d314f79   2.2.0             31d
    15. rendered-master-419bee7de96134963a15fdf9dd473b25            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             17d
    16. rendered-master-53f5c91c7661708adce18739cc0f40fb            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             13d
    17. rendered-master-a6a357ec18e5bce7f5ac426fc7c5ffcd            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             7d3h
    18. rendered-master-dc7f874ec77fc4b969674204332da037            5b6fb8349a29735e48446d435962dec4547d3090   2.2.0             31d
    19. rendered-worker-1a75960c52ad18ff5dfa6674eb7e533d            5b6fb8349a29735e48446d435962dec4547d3090   2.2.0             31d
    20. rendered-worker-2640531be11ba43c61d72e82dc634ce6            5b6fb8349a29735e48446d435962dec4547d3090   2.2.0             31d
    21. rendered-worker-4e48906dca84ee702959c71a53ee80e7            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             7d3h
    22. rendered-worker-4f110718fe88e5f349987854a1147755            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             17d
    23. rendered-worker-afc758e194d6188677eb837842d3b379            02c07496ba0417b3e12b78fb32baf6293d314f79   2.2.0             31d
    24. rendered-worker-daa08cc1e8f5fcdeba24de60cd955cc3            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   2.2.0             13d

    You should see a new machine config, with the rendered-infra-* prefix.

  5. Optional: To deploy changes to a custom pool, create a machine config that uses the custom pool name as the label, such as infra. Note that this is not required and only shown for instructional purposes. In this manner, you can apply any custom configurations specific to only your infra nodes.

    After you create the new machine config pool, the MCO generates a new rendered config for that pool, and associated nodes of that pool reboot to apply the new configuration.

    1. Create a machine config:

      1. $ cat infra.mc.yaml

      Example output

      1. apiVersion: machineconfiguration.openshift.io/v1
      2. kind: MachineConfig
      3. metadata:
      4.   name: 51-infra
      5.   labels:
      6.     machineconfiguration.openshift.io/role: infra (1)
      7. spec:
      8.   config:
      9.     ignition:
      10.       version: 3.1.0
      11.     storage:
      12.       files:
      13.       - path: /etc/infratest
      14.         mode: 0644
      15.         contents:
      16.           source: data:,infra
      1Add the label you added to the node as a nodeSelector.

    2. Apply the machine config to the infra-labeled nodes:

      1. $ oc create -f infra.mc.yaml

  6. Confirm that your new machine config pool is available:

    1. $ oc get mcp

    Example output

    1. NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    2. infra    rendered-infra-60e35c2e99f42d976e084fa94da4d0fc    True      False      False      1              1                   1                     0                      4m20s
    3. master   rendered-master-9360fdb895d4c131c7c4bebbae099c90   True      False      False      3              3                   3                     0                      91m
    4. worker   rendered-worker-60e35c2e99f42d976e084fa94da4d0fc   True      False      False      2              2                   2                     0                      91m

    In this example, a worker node was changed to an infra node.

Additional resources

Assigning machine set resources to infrastructure nodes

After creating an infrastructure machine set, the worker and infra roles are applied to new infra nodes. Nodes with the infra role are not counted toward the total number of subscriptions that are required to run the environment, even when the worker role is also applied.

However, when an infra node is assigned the worker role, there is a chance that user workloads can get assigned inadvertently to the infra node. To avoid this, you can apply a taint to the infra node and tolerations for the pods that you want to control.

Binding infrastructure node workloads using taints and tolerations

If you have an infra node that has the infra and worker roles assigned, you must configure the node so that user workloads are not assigned to it.

It is recommended that you preserve the dual infra,worker label that is created for infra nodes and use taints and tolerations to manage nodes that user workloads are scheduled on. If you remove the worker label from the node, you must create a custom pool to manage it. A node with a label other than master or worker is not recognized by the MCO without a custom pool. Maintaining the worker label allows the node to be managed by the default worker machine config pool, if no custom pools that select the custom label exists. The infra label communicates to the cluster that it does not count toward the total number of subscriptions.

Prerequisites

  • Configure additional MachineSet objects in your OKD cluster.

Procedure

  1. Add a taint to the infra node to prevent scheduling user workloads on it:

    1. Determine if the node has the taint:

      1. $ oc describe nodes <node_name>

      Sample output

      1. oc describe node ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      2. Name:               ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      3. Roles:              worker
      4.  ...
      5. Taints:             node-role.kubernetes.io/infra:NoSchedule
      6.  ...

      This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.

    2. If you have not configured a taint to prevent scheduling user workloads on it:

      1. $ oc adm taint nodes <node_name> <key>:<effect>

      For example:

      1. $ oc adm taint nodes node1 node-role.kubernetes.io/infra:NoSchedule

      This example places a taint on node1 that has key node-role.kubernetes.io/infra and taint effect NoSchedule. Nodes with the NoSchedule effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.

      If a descheduler is used, pods violating node taints could be evicted from the cluster.

  2. Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the Pod object specification:

    1. tolerations:
    2.   - effect: NoSchedule (1)
    3.     key: node-role.kubernetes.io/infra (2)
    4.     operator: Exists (3)
    1Specify the effect that you added to the node.
    2Specify the key that you added to the node.
    3Specify the Exists Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node.

    This toleration matches the taint created by the oc adm taint command. A pod with this toleration can be scheduled onto the infra node.

    Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.

  3. Schedule the pod to the infra node using a scheduler. See the documentation for Controlling pod placement onto nodes for details.

Additional resources

Moving resources to infrastructure machine sets

Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created.

Moving the router

You can deploy the router pod to a different machine set. By default, the pod is deployed to a worker node.

Prerequisites

  • Configure additional machine sets in your OKD cluster.

Procedure

  1. View the IngressController custom resource for the router Operator:

    1. $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

    The command output resembles the following text:

    1. apiVersion: operator.openshift.io/v1
    2. kind: IngressController
    3. metadata:
    4.   creationTimestamp: 2019-04-18T12:35:39Z
    5.   finalizers:
    6.   - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
    7.   generation: 1
    8.   name: default
    9.   namespace: openshift-ingress-operator
    10.   resourceVersion: "11341"
    11.   selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
    12.   uid: 79509e05-61d6-11e9-bc55-02ce4781844a
    13. spec: {}
    14. status:
    15.   availableReplicas: 2
    16.   conditions:
    17.   - lastTransitionTime: 2019-04-18T12:36:15Z
    18.     status: "True"
    19.     type: Available
    20.   domain: apps.<cluster>.example.com
    21.   endpointPublishingStrategy:
    22.     type: LoadBalancerService
    23.   selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

  2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

    1. $ oc edit ingresscontroller default -n openshift-ingress-operator

    Add the nodeSelector stanza that references the infra label to the spec section, as shown:

    1.   spec:
    2.     nodePlacement:
    3.       nodeSelector:
    4.         matchLabels:
    5.           node-role.kubernetes.io/infra: ""

  3. Confirm that the router pod is running on the infra node.

    1. View the list of router pods and note the node name of the running pod:

      1. $ oc get pod -n openshift-ingress -o wide

      Example output

      1. NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      2. router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      3. router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

    2. View the node status of the running pod:

      1. $ oc get node <node_name> (1)
      1Specify the <node_name> that you obtained from the pod list.

      Example output

      1. NAME                          STATUS  ROLES         AGE   VERSION
      2. ip-10-0-217-226.ec2.internal  Ready   infra,worker  17h   v1.19.0

      Because the role list includes infra, the pod is running on the correct node.

Moving the default registry

You configure the registry Operator to deploy its pods to different nodes.

Prerequisites

  • Configure additional machine sets in your OKD cluster.

Procedure

  1. View the config/instance object:

    1. $ oc get configs.imageregistry.operator.openshift.io/cluster -o yaml

    Example output

    1. apiVersion: imageregistry.operator.openshift.io/v1
    2. kind: Config
    3. metadata:
    4.   creationTimestamp: 2019-02-05T13:52:05Z
    5.   finalizers:
    6.   - imageregistry.operator.openshift.io/finalizer
    7.   generation: 1
    8.   name: cluster
    9.   resourceVersion: "56174"
    10.   selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
    11.   uid: 36fd3724-294d-11e9-a524-12ffeee2931b
    12. spec:
    13.   httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
    14.   logging: 2
    15.   managementState: Managed
    16.   proxy: {}
    17.   replicas: 1
    18.   requests:
    19.     read: {}
    20.     write: {}
    21.   storage:
    22.     s3:
    23.       bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
    24.       region: us-east-1
    25. status:
    26. ...

  2. Edit the config/instance object:

    1. $ oc edit configs.imageregistry.operator.openshift.io/cluster

  3. Modify the spec section of the object to resemble the following YAML:

    1. spec:
    2.   affinity:
    3.     podAntiAffinity:
    4.       preferredDuringSchedulingIgnoredDuringExecution:
    5.       - podAffinityTerm:
    6.           namespaces:
    7.           - openshift-image-registry
    8.           topologyKey: kubernetes.io/hostname
    9.         weight: 100
    10.   logLevel: Normal
    11.   managementState: Managed
    12.   nodeSelector:
    13.     node-role.kubernetes.io/infra: ""

  4. Verify the registry pod has been moved to the infrastructure node.

    1. Run the following command to identify the node where the registry pod is located:

      1. $ oc get pods -o wide -n openshift-image-registry

    2. Confirm the node has the label you specified:

      1. $ oc describe node <node_name>

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Moving the monitoring solution

By default, the Prometheus Cluster Monitoring stack, which contains Prometheus, Grafana, and AlertManager, is deployed to provide cluster monitoring. It is managed by the Cluster Monitoring Operator. To move its components to different machines, you create and apply a custom config map.

Procedure

  1. Save the following ConfigMap definition as the cluster-monitoring-configmap.yaml file:

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: cluster-monitoring-config
    5.   namespace: openshift-monitoring
    6. data:
    7.   config.yaml: |+
    8.     alertmanagerMain:
    9.       nodeSelector:
    10.         node-role.kubernetes.io/infra: ""
    11.     prometheusK8s:
    12.       nodeSelector:
    13.         node-role.kubernetes.io/infra: ""
    14.     prometheusOperator:
    15.       nodeSelector:
    16.         node-role.kubernetes.io/infra: ""
    17.     grafana:
    18.       nodeSelector:
    19.         node-role.kubernetes.io/infra: ""
    20.     k8sPrometheusAdapter:
    21.       nodeSelector:
    22.         node-role.kubernetes.io/infra: ""
    23.     kubeStateMetrics:
    24.       nodeSelector:
    25.         node-role.kubernetes.io/infra: ""
    26.     telemeterClient:
    27.       nodeSelector:
    28.         node-role.kubernetes.io/infra: ""
    29.     openshiftStateMetrics:
    30.       nodeSelector:
    31.         node-role.kubernetes.io/infra: ""
    32.     thanosQuerier:
    33.       nodeSelector:
    34.         node-role.kubernetes.io/infra: ""

    Running this config map forces the components of the monitoring stack to redeploy to infrastructure nodes.

  2. Apply the new config map:

    1. $ oc create -f cluster-monitoring-configmap.yaml

  3. Watch the monitoring pods move to the new machines:

    1. $ watch 'oc get pod -n openshift-monitoring -o wide'

  4. If a component has not moved to the infra node, delete the pod with this component:

    1. $ oc delete pod -n openshift-monitoring <pod>

    The component from the deleted pod is re-created on the infra node.

Moving the cluster logging resources

You can configure the Cluster Logging Operator to deploy the pods for any or all of the Cluster Logging components, Elasticsearch, Kibana, and Curator to different nodes. You cannot move the Cluster Logging Operator pod from its installed location.

For example, you can move the Elasticsearch pods to a separate node because of high CPU, memory, and disk requirements.

Prerequisites

  • Cluster logging and Elasticsearch must be installed. These features are not installed by default.

Procedure

  1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

    1. $ oc edit ClusterLogging instance
    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogging
    4. ...
    6. spec:
    7.   collection:
    8.     logs:
    9.       fluentd:
    10.         resources: null
    11.       type: fluentd
    12.   curation:
    13.     curator:
    14.       nodeSelector: (1)
    15.         node-role.kubernetes.io/infra: ''
    16.       resources: null
    17.       schedule: 30 3 * * *
    18.     type: curator
    19.   logStore:
    20.     elasticsearch:
    21.       nodeCount: 3
    22.       nodeSelector: (1)
    23.         node-role.kubernetes.io/infra: ''
    24.       redundancyPolicy: SingleRedundancy
    25.       resources:
    26.         limits:
    27.           cpu: 500m
    28.           memory: 16Gi
    29.         requests:
    30.           cpu: 500m
    31.           memory: 16Gi
    32.       storage: {}
    33.     type: elasticsearch
    34.   managementState: Managed
    35.   visualization:
    36.     kibana:
    37.       nodeSelector: (1)
    38.         node-role.kubernetes.io/infra: ''
    39.       proxy:
    40.         resources: null
    41.       replicas: 1
    42.       resources: null
    43.     type: kibana
    45. ...
1Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node.

Verification

To verify that a component has moved, you can use the oc get pod -o wide command.

For example:

  • You want to move the Kibana pod from the ip-10-0-147-79.us-east-2.compute.internal node:

    1. $ oc get pod kibana-5b8bdf44f9-ccpq9 -o wide

    Example output

    1. NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
    2. kibana-5b8bdf44f9-ccpq9   2/2     Running   0          27s   10.129.2.18   ip-10-0-147-79.us-east-2.compute.internal   <none>           <none>

  • You want to move the Kibana Pod to the ip-10-0-139-48.us-east-2.compute.internal node, a dedicated infrastructure node:

    1. $ oc get nodes

    Example output

    1. NAME                                         STATUS   ROLES          AGE   VERSION
    2. ip-10-0-133-216.us-east-2.compute.internal   Ready    master         60m   v1.19.0
    3. ip-10-0-139-146.us-east-2.compute.internal   Ready    master         60m   v1.19.0
    4. ip-10-0-139-192.us-east-2.compute.internal   Ready    worker         51m   v1.19.0
    5. ip-10-0-139-241.us-east-2.compute.internal   Ready    worker         51m   v1.19.0
    6. ip-10-0-147-79.us-east-2.compute.internal    Ready    worker         51m   v1.19.0
    7. ip-10-0-152-241.us-east-2.compute.internal   Ready    master         60m   v1.19.0
    8. ip-10-0-139-48.us-east-2.compute.internal    Ready    infra          51m   v1.19.0

    Note that the node has a node-role.kubernetes.io/infra: '' label:

    1. $ oc get node ip-10-0-139-48.us-east-2.compute.internal -o yaml

    Example output

    1. kind: Node
    2. apiVersion: v1
    3. metadata:
    4.   name: ip-10-0-139-48.us-east-2.compute.internal
    5.   selfLink: /api/v1/nodes/ip-10-0-139-48.us-east-2.compute.internal
    6.   uid: 62038aa9-661f-41d7-ba93-b5f1b6ef8751
    7.   resourceVersion: '39083'
    8.   creationTimestamp: '2020-04-13T19:07:55Z'
    9.   labels:
    10.     node-role.kubernetes.io/infra: ''
    11. ...

  • To move the Kibana pod, edit the ClusterLogging CR to add a node selector:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogging
    4. ...
    6. spec:
    8. ...
    10.   visualization:
    11.     kibana:
    12.       nodeSelector: (1)
    13.         node-role.kubernetes.io/infra: ''
    14.       proxy:
    15.         resources: null
    16.       replicas: 1
    17.       resources: null
    18.     type: kibana
    1Add a node selector to match the label in the node specification.

  • After you save the CR, the current Kibana pod is terminated and new pod is deployed:

    1. $ oc get pods

    Example output

    1. NAME                                            READY   STATUS        RESTARTS   AGE
    2. cluster-logging-operator-84d98649c4-zb9g7       1/1     Running       0          29m
    3. elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg   2/2     Running       0          28m
    4. elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj   2/2     Running       0          28m
    5. elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78    2/2     Running       0          28m
    6. fluentd-42dzz                                   1/1     Running       0          28m
    7. fluentd-d74rq                                   1/1     Running       0          28m
    8. fluentd-m5vr9                                   1/1     Running       0          28m
    9. fluentd-nkxl7                                   1/1     Running       0          28m
    10. fluentd-pdvqb                                   1/1     Running       0          28m
    11. fluentd-tflh6                                   1/1     Running       0          28m
    12. kibana-5b8bdf44f9-ccpq9                         2/2     Terminating   0          4m11s
    13. kibana-7d85dcffc8-bfpfp                         2/2     Running       0          33s

  • The new pod is on the ip-10-0-139-48.us-east-2.compute.internal node:

    1. $ oc get pod kibana-7d85dcffc8-bfpfp -o wide

    Example output

    1. NAME                      READY   STATUS        RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
    2. kibana-7d85dcffc8-bfpfp   2/2     Running       0          43s   10.131.0.22   ip-10-0-139-48.us-east-2.compute.internal   <none>           <none>

  • After a few moments, the original Kibana pod is removed.

    1. $ oc get pods

    Example output

    1. NAME                                            READY   STATUS    RESTARTS   AGE
    2. cluster-logging-operator-84d98649c4-zb9g7       1/1     Running   0          30m
    3. elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg   2/2     Running   0          29m
    4. elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj   2/2     Running   0          29m
    5. elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78    2/2     Running   0          29m
    6. fluentd-42dzz                                   1/1     Running   0          29m
    7. fluentd-d74rq                                   1/1     Running   0          29m
    8. fluentd-m5vr9                                   1/1     Running   0          29m
    9. fluentd-nkxl7                                   1/1     Running   0          29m
    10. fluentd-pdvqb                                   1/1     Running   0          29m
    11. fluentd-tflh6                                   1/1     Running   0          29m
    12. kibana-7d85dcffc8-bfpfp                         2/2     Running   0          62s

About the cluster autoscaler

The cluster autoscaler adjusts the size of an OKD cluster to meet its current deployment needs. It uses declarative, Kubernetes-style arguments to provide infrastructure management that does not rely on objects of a specific cloud provider. The cluster autoscaler has a cluster scope, and is not associated with a particular namespace.

The cluster autoscaler increases the size of the cluster when there are pods that failed to schedule on any of the current nodes due to insufficient resources or when another node is necessary to meet deployment needs. The cluster autoscaler does not increase the cluster resources beyond the limits that you specify.

Ensure that the maxNodesTotal value in the ClusterAutoscaler resource definition that you create is large enough to account for the total possible number of machines in your cluster. This value must encompass the number of control plane machines and the possible number of compute machines that you might scale to.

The cluster autoscaler decreases the size of the cluster when some nodes are consistently not needed for a significant period, such as when it has low resource use and all of its important pods can fit on other nodes.

If the following types of pods are present on a node, the cluster autoscaler will not remove the node:

  • Pods with restrictive pod disruption budgets (PDBs).

  • Kube-system pods that do not run on the node by default.

  • Kube-system pods that do not have a PDB or have a PDB that is too restrictive.

  • Pods that are not backed by a controller object such as a deployment, replica set, or stateful set.

  • Pods with local storage.

  • Pods that cannot be moved elsewhere because of a lack of resources, incompatible node selectors or affinity, matching anti-affinity, and so on.

  • Unless they also have a "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation, pods that have a "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation.

If you configure the cluster autoscaler, additional usage restrictions apply:

  • Do not modify the nodes that are in autoscaled node groups directly. All nodes within the same node group have the same capacity and labels and run the same system pods.

  • Specify requests for your pods.

  • If you have to prevent pods from being deleted too quickly, configure appropriate PDBs.

  • Confirm that your cloud provider quota is large enough to support the maximum node pools that you configure.

  • Do not run additional node group autoscalers, especially the ones offered by your cloud provider.

The horizontal pod autoscaler (HPA) and the cluster autoscaler modify cluster resources in different ways. The HPA changes the deployment’s or replica set’s number of replicas based on the current CPU load. If the load increases, the HPA creates new replicas, regardless of the amount of resources available to the cluster. If there are not enough resources, the cluster autoscaler adds resources so that the HPA-created pods can run. If the load decreases, the HPA stops some replicas. If this action causes some nodes to be underutilized or completely empty, the cluster autoscaler deletes the unnecessary nodes.

The cluster autoscaler takes pod priorities into account. The Pod Priority and Preemption feature enables scheduling pods based on priorities if the cluster does not have enough resources, but the cluster autoscaler ensures that the cluster has resources to run all pods. To honor the intention of both features, the cluster autoscaler includes a priority cutoff function. You can use this cutoff to schedule “best-effort” pods, which do not cause the cluster autoscaler to increase resources but instead run only when spare resources are available.

Pods with priority lower than the cutoff value do not cause the cluster to scale up or prevent the cluster from scaling down. No new nodes are added to run the pods, and nodes running these pods might be deleted to free resources.

ClusterAutoscaler resource definition

This ClusterAutoscaler resource definition shows the parameters and sample values for the cluster autoscaler.

  1. apiVersion: "autoscaling.openshift.io/v1"
  2. kind: "ClusterAutoscaler"
  3. metadata:
  4.   name: "default"
  5. spec:
  6.   podPriorityThreshold: -10 (1)
  7.   resourceLimits:
  8.     maxNodesTotal: 24 (2)
  9.     cores:
  10.       min: 8 (3)
  11.       max: 128 (4)
  12.     memory:
  13.       min: 4 (5)
  14.       max: 256 (6)
  15.     gpus:
  16.       - type: nvidia.com/gpu (7)
  17.         min: 0 (8)
  18.         max: 16 (9)
  19.       - type: amd.com/gpu (7)
  20.         min: 0 (8)
  21.         max: 4 (9)
  22.   scaleDown: (10)
  23.     enabled: true (11)
  24.     delayAfterAdd: 10m (12)
  25.     delayAfterDelete: 5m (13)
  26.     delayAfterFailure: 30s (14)
  27.     unneededTime: 5m (15)
1Specify the priority that a pod must exceed to cause the cluster autoscaler to deploy additional nodes. Enter a 32-bit integer value. The podPriorityThreshold value is compared to the value of the PriorityClass that you assign to each pod.
2Specify the maximum number of nodes to deploy. This value is the total number of machines that are deployed in your cluster, not just the ones that the autoscaler controls. Ensure that this value is large enough to account for all of your control plane and compute machines and the total number of replicas that you specify in your MachineAutoscaler resources.
3Specify the minimum number of cores to deploy in the cluster.
4Specify the maximum number of cores to deploy in the cluster.
5Specify the minimum amount of memory, in GiB, in the cluster.
6Specify the maximum amount of memory, in GiB, in the cluster.
7Optionally, specify the type of GPU node to deploy. Only nvidia.com/gpu and amd.com/gpu are valid types.
8Specify the minimum number of GPUs to deploy in the cluster.
9Specify the maximum number of GPUs to deploy in the cluster.
10In this section, you can specify the period to wait for each action by using any valid ParseDuration interval, including ns, us, ms, s, m, and h.
11Specify whether the cluster autoscaler can remove unnecessary nodes.
12Optionally, specify the period to wait before deleting a node after a node has recently been added. If you do not specify a value, the default value of 10m is used.
13Specify the period to wait before deleting a node after a node has recently been deleted. If you do not specify a value, the default value of 10s is used.
14Specify the period to wait before deleting a node after a scale down failure occurred. If you do not specify a value, the default value of 3m is used.
15Specify the period before an unnecessary node is eligible for deletion. If you do not specify a value, the default value of 10m is used.

When performing a scaling operation, the cluster autoscaler remains within the ranges set in the ClusterAutoscaler resource definition, such as the minimum and maximum number of cores to deploy or the amount of memory in the cluster. However, the cluster autoscaler does not correct the current values in your cluster to be within those ranges.

Deploying the cluster autoscaler

To deploy the cluster autoscaler, you create an instance of the ClusterAutoscaler resource.

Procedure

  1. Create a YAML file for the ClusterAutoscaler resource that contains the customized resource definition.

  2. Create the resource in the cluster:

    1. $ oc create -f <filename>.yaml (1)
    1<filename> is the name of the resource file that you customized.

About the machine autoscaler

The machine autoscaler adjusts the number of Machines in the machine sets that you deploy in an OKD cluster. You can scale both the default worker machine set and any other machine sets that you create. The machine autoscaler makes more Machines when the cluster runs out of resources to support more deployments. Any changes to the values in MachineAutoscaler resources, such as the minimum or maximum number of instances, are immediately applied to the machine set they target.

You must deploy a machine autoscaler for the cluster autoscaler to scale your machines. The cluster autoscaler uses the annotations on machine sets that the machine autoscaler sets to determine the resources that it can scale. If you define a cluster autoscaler without also defining machine autoscalers, the cluster autoscaler will never scale your cluster.

MachineAutoscaler resource definition

This MachineAutoscaler resource definition shows the parameters and sample values for the machine autoscaler.

  1. apiVersion: "autoscaling.openshift.io/v1beta1"
  2. kind: "MachineAutoscaler"
  3. metadata:
  4.   name: "worker-us-east-1a" (1)
  5.   namespace: "openshift-machine-api"
  6. spec:
  7.   minReplicas: 1 (2)
  8.   maxReplicas: 12 (3)
  9.   scaleTargetRef: (4)
  10.     apiVersion: machine.openshift.io/v1beta1
  11.     kind: MachineSet (5)
  12.     name: worker-us-east-1a (6)
1Specify the machine autoscaler name. To make it easier to identify which machine set this machine autoscaler scales, specify or include the name of the machine set to scale. The machine set name takes the following form: <clusterid>-<machineset>-<aws-region-az>
2Specify the minimum number machines of the specified type that must remain in the specified zone after the cluster autoscaler initiates cluster scaling. If running in AWS, GCP, or Azure, this value can be set to 0. For other providers, do not set this value to 0.
3Specify the maximum number machines of the specified type that the cluster autoscaler can deploy in the specified AWS zone after it initiates cluster scaling. Ensure that the maxNodesTotal value in the ClusterAutoscaler resource definition is large enough to allow the machine autoscaler to deploy this number of machines.
4In this section, provide values that describe the existing machine set to scale.
5The kind parameter value is always MachineSet.
6The name value must match the name of an existing machine set, as shown in the metadata.name parameter value.

Deploying the machine autoscaler

To deploy the machine autoscaler, you create an instance of the MachineAutoscaler resource.

Procedure

  1. Create a YAML file for the MachineAutoscaler resource that contains the customized resource definition.

  2. Create the resource in the cluster:

    1. $ oc create -f <filename>.yaml (1)
    1<filename> is the name of the resource file that you customized.

Enabling Technology Preview features using FeatureGates

You can turn on a subset of the current Technology Preview features on for all nodes in the cluster by editing the FeatureGate custom resource (CR).

Understanding feature gates

You can use the FeatureGate custom resource (CR) to enable specific feature sets in your cluster. A feature set is a collection of OKD features that are not enabled by default.

For example, the TechPreviewNoUpgrade feature set allows you to enable a subset of the current Technology Preview features on test clusters, where you can fully test them, while leaving the features disabled on production clusters.

You can activate any of the following feature sets by using the FeatureGate CR:

Feature SetDescription

IPv6DualStackNoUpgrade

Enables the dual-stack networking mode in your cluster. Dual-stack networking supports the use of IPv4 and IPv6 simultaneously. Enabling this feature set is not supported, cannot be undone, and prevents upgrades. This feature set is not recommended on production clusters.

TechPreviewNoUpgrade

Enables Technology Preview features that are not part of the default features. Enabling this feature set cannot be undone and prevents upgrades. This feature set is not recommended on production clusters.

The following Technology Preview features are enabled by this feature set:

Enabling feature sets using the CLI

You can use the OpenShift CLI (oc) to enable feature sets for all of the nodes in a cluster by editing the FeatureGate custom resource (CR).

Prerequisites

  • You have installed the OpenShift CLI (oc).

Procedure

To enable feature sets:

  1. Edit the FeatureGate CR named cluster:

    1. $ oc edit featuregate cluster

    Sample FeatureGate custom resource

    1. apiVersion: config.openshift.io/v1
    2. kind: FeatureGate
    3. metadata:
    4.   name: cluster (1)
    5. spec:
    6.   featureSet: TechPreviewNoUpgrade (2)
    1The name of the FeatureGate CR must be cluster.
    2Add the feature sets that you want to enable in a comma-separated list:

    • IPv6DualStackNoUpgrade enables the dual-stack networking mode.

    • TechPreviewNoUpgrade enables specific Technology Preview features.

    Enabling the TechPreviewNoUpgrade or IPv6DualStackNoUpgrade feature sets cannot be undone and prevents upgrades. These feature sets are not recommended on production clusters.

etcd tasks

Back up etcd, enable or disable etcd encryption, or defragment etcd data.

About etcd encryption

By default, etcd data is not encrypted in OKD. You can enable etcd encryption for your cluster to provide an additional layer of data security. For example, it can help protect the loss of sensitive data if an etcd backup is exposed to the incorrect parties.

When you enable etcd encryption, the following OpenShift API server and Kubernetes API server resources are encrypted:

  • Secrets

  • Config maps

  • Routes

  • OAuth access tokens

  • OAuth authorize tokens

When you enable etcd encryption, encryption keys are created. These keys are rotated on a weekly basis. You must have these keys in order to restore from an etcd backup.

Keep in mind that etcd encryption only encrypts values, not keys. This means that resource types, namespaces, and object names are unencrypted.

Enabling etcd encryption

You can enable etcd encryption to encrypt sensitive resources in your cluster.

It is not recommended to take a backup of etcd until the initial encryption process is complete. If the encryption process has not completed, the backup might be only partially encrypted.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

Procedure

  1. Modify the APIServer object:

    1. $ oc edit apiserver

  2. Set the encryption field type to aescbc:

    1. spec:
    2.   encryption:
    3.     type: aescbc (1)
    1The aescbc type means that AES-CBC with PKCS#7 padding and a 32 byte key is used to perform the encryption.

  3. Save the file to apply the changes.

    The encryption process starts. It can take 20 minutes or longer for this process to complete, depending on the size of your cluster.

  4. Verify that etcd encryption was successful.

    1. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully encrypted:

      1. $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      1. EncryptionCompleted
      2. All resources encrypted: routes.route.openshift.io, oauthaccesstokens.oauth.openshift.io, oauthauthorizetokens.oauth.openshift.io

      If the output shows EncryptionInProgress, this means that encryption is still in progress. Wait a few minutes and try again.

    2. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully encrypted:

      1. $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      1. EncryptionCompleted
      2. All resources encrypted: secrets, configmaps

      If the output shows EncryptionInProgress, this means that encryption is still in progress. Wait a few minutes and try again.

Disabling etcd encryption

You can disable encryption of etcd data in your cluster.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

Procedure

  1. Modify the APIServer object:

    1. $ oc edit apiserver

  2. Set the encryption field type to identity:

    1. spec:
    2.   encryption:
    3.     type: identity (1)
    1The identity type is the default value and means that no encryption is performed.

  3. Save the file to apply the changes.

    The decryption process starts. It can take 20 minutes or longer for this process to complete, depending on the size of your cluster.

  4. Verify that etcd decryption was successful.

    1. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully decrypted:

      1. $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      1. DecryptionCompleted
      2. Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, this means that decryption is still in progress. Wait a few minutes and try again.

    2. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully decrypted:

      1. $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      1. DecryptionCompleted
      2. Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, this means that decryption is still in progress. Wait a few minutes and try again.

Backing up etcd data

Follow these steps to back up etcd data by creating an etcd snapshot and backing up the resources for the static pods. This backup can be saved and used at a later time if you need to restore etcd.

Only save a backup from a single control plane host (also known as the master host). Do not take a backup from each control plane host in the cluster.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have checked whether the cluster-wide proxy is enabled.

    You can check whether the proxy is enabled by reviewing the output of oc get proxy cluster -o yaml. The proxy is enabled if the httpProxy, httpsProxy, and noProxy fields have values set.

Procedure

  1. Start a debug session for a control plane node:

    1. $ oc debug node/<node_name>

  2. Change your root directory to the host:

    1. sh-4.2# chroot /host

  3. If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables.

  4. Run the cluster-backup.sh script and pass in the location to save the backup to.

    The cluster-backup.sh script is maintained as a component of the etcd Cluster Operator and is a wrapper around the etcdctl snapshot save command.

    1. sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

    Example script output

    1. found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6
    2. found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-7
    3. found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-6
    4. found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3
    5. ede95fe6b88b87ba86a03c15e669fb4aa5bf0991c180d3c6895ce72eaade54a1
    6. etcdctl version: 3.4.14
    7. API version: 3.4
    8. {"level":"info","ts":1624647639.0188997,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db.part"}
    9. {"level":"info","ts":"2021-06-25T19:00:39.030Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
    10. {"level":"info","ts":1624647639.0301006,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://10.0.0.5:2379"}
    11. {"level":"info","ts":"2021-06-25T19:00:40.215Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
    12. {"level":"info","ts":1624647640.6032252,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://10.0.0.5:2379","size":"114 MB","took":1.584090459}
    13. {"level":"info","ts":1624647640.6047094,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db"}
    14. Snapshot saved at /home/core/assets/backup/snapshot_2021-06-25_190035.db
    15. {"hash":3866667823,"revision":31407,"totalKey":12828,"totalSize":114446336}
    16. snapshot db and kube resources are successfully saved to /home/core/assets/backup

    In this example, two files are created in the /home/core/assets/backup/ directory on the control plane host:

    • snapshot_<datetimestamp>.db: This file is the etcd snapshot. The cluster-backup.sh script confirms its validity.

    • static_kuberesources_<datetimestamp>.tar.gz: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot.

      If etcd encryption is enabled, it is recommended to store this second file separately from the etcd snapshot for security reasons. However, this file is required in order to restore from the etcd snapshot.

      Keep in mind that etcd encryption only encrypts values, not keys. This means that resource types, namespaces, and object names are unencrypted.

Defragmenting etcd data

Manual defragmentation must be performed periodically to reclaim disk space after etcd history compaction and other events cause disk fragmentation.

History compaction is performed automatically every five minutes and leaves gaps in the back-end database. This fragmented space is available for use by etcd, but is not available to the host file system. You must defragment etcd to make this space available to the host file system.

Because etcd writes data to disk, its performance strongly depends on disk performance. Consider defragmenting etcd every month, twice a month, or as needed for your cluster. You can also monitor the etcd_db_total_size_in_bytes metric to determine whether defragmentation is necessary.

Defragmenting etcd is a blocking action. The etcd member will not response until defragmentation is complete. For this reason, wait at least one minute between defragmentation actions on each of the pods to allow the cluster to recover.

Follow this procedure to defragment etcd data on each etcd member.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Determine which etcd member is the leader, because the leader should be defragmented last.

    1. Get the list of etcd pods:

      1. $ oc get pods -n openshift-etcd -o wide | grep -v quorum-guard | grep etcd

      Example output

      1. etcd-ip-10-0-159-225.example.redhat.com                3/3     Running     0          175m   10.0.159.225   ip-10-0-159-225.example.redhat.com   <none>           <none>
      2. etcd-ip-10-0-191-37.example.redhat.com                 3/3     Running     0          173m   10.0.191.37    ip-10-0-191-37.example.redhat.com    <none>           <none>
      3. etcd-ip-10-0-199-170.example.redhat.com                3/3     Running     0          176m   10.0.199.170   ip-10-0-199-170.example.redhat.com   <none>           <none>

    2. Choose a pod and run the following command to determine which etcd member is the leader:

      1. $ oc rsh -n openshift-etcd etcd-ip-10-0-159-225.us-west-1.compute.internal etcdctl endpoint status --cluster -w table

      Example output

      1. Defaulting container name to etcdctl.
      2. Use 'oc describe pod/etcd-ip-10-0-159-225.example.redhat.com -n openshift-etcd' to see all of the containers in this pod.
      3. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      4. |         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
      5. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      6. |  https://10.0.191.37:2379 | 251cd44483d811c3 |   3.4.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      7. | https://10.0.159.225:2379 | 264c7c58ecbdabee |   3.4.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      8. | https://10.0.199.170:2379 | 9ac311f93915cc79 |   3.4.9 |  104 MB |      true |      false |         7 |      91624 |              91624 |        |
      9. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

      Based on the IS LEADER column of this output, the https://10.0.199.170:2379 endpoint is the leader. Matching this endpoint with the output of the previous step, the pod name of the leader is etcd-ip-10-0-199-170.example.redhat.com.

  2. Defragment an etcd member.

    1. Connect to the running etcd container, passing in the name of a pod that is not the leader:

      1. $ oc rsh -n openshift-etcd etcd-ip-10-0-159-225.example.redhat.com

    2. Unset the ETCDCTL_ENDPOINTS environment variable:

      1. sh-4.4# unset ETCDCTL_ENDPOINTS

    3. Defragment the etcd member:

      1. sh-4.4# etcdctl --command-timeout=30s --endpoints=https://localhost:2379 defrag

      Example output

      1. Finished defragmenting etcd member[https://localhost:2379]

      If a timeout error occurs, increase the value for --command-timeout until the command succeeds.

    4. Verify that the database size was reduced:

      1. sh-4.4# etcdctl endpoint status -w table --cluster

      Example output

      1. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      2. |         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
      3. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      4. |  https://10.0.191.37:2379 | 251cd44483d811c3 |   3.4.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      5. | https://10.0.159.225:2379 | 264c7c58ecbdabee |   3.4.9 |   41 MB |     false |      false |         7 |      91624 |              91624 |        | (1)
      6. | https://10.0.199.170:2379 | 9ac311f93915cc79 |   3.4.9 |  104 MB |      true |      false |         7 |      91624 |              91624 |        |
      7. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

      This example shows that the database size for this etcd member is now 41 MB as opposed to the starting size of 104 MB.

    5. Repeat these steps to connect to each of the other etcd members and defragment them. Always defragment the leader last.

      Wait at least one minute between defragmentation actions to allow the etcd pod to recover. Until the etcd pod recovers, the etcd member will not respond.

  3. If any NOSPACE alarms were triggered due to the space quota being exceeded, clear them.

    1. Check if there are any NOSPACE alarms:

      1. sh-4.4# etcdctl alarm list

      Example output

      1. memberID:12345678912345678912 alarm:NOSPACE

    2. Clear the alarms:

      1. sh-4.4# etcdctl alarm disarm

Restoring to a previous cluster state

You can use a saved etcd backup to restore back to a previous cluster state. You use the etcd backup to restore a single control plane host. Then the etcd cluster Operator handles scaling to the remaining control plane hosts (also known as the master hosts).

When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an OKD 4.6.2 cluster must use an etcd backup that was taken from 4.6.2.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

  • A healthy control plane host to use as the recovery host.

  • SSH access to control plane hosts.

  • A backup directory containing both the etcd snapshot and the resources for the static pods, which were from the same backup. The file names in the directory must be in the following formats: snapshot_<datetimestamp>.db and static_kuberesources_<datetimestamp>.tar.gz.

Procedure

  1. Select a control plane host to use as the recovery host. This is the host that you will run the restore operation on.

  2. Establish SSH connectivity to each of the control plane nodes, including the recovery host.

    The Kubernetes API server becomes inaccessible after the restore process starts, so you cannot access the control plane nodes. For this reason, it is recommended to establish SSH connectivity to each control plane host in a separate terminal.

    If you do not complete this step, you will not be able to access the control plane hosts to complete the restore procedure, and you will be unable to recover your cluster from this state.

  3. Copy the etcd backup directory to the recovery control plane host.

    This procedure assumes that you copied the backup directory containing the etcd snapshot and the resources for the static pods to the /home/core/ directory of your recovery control plane host.

  4. Stop the static pods on all other control plane nodes.

    It is not required to manually stop the pods on the recovery host. The recovery script will stop the pods on the recovery host.

    1. Access a control plane host that is not the recovery host.

    2. Move the existing etcd pod file out of the kubelet manifest directory:

      1. [core@ip-10-0-154-194 ~]$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp

    3. Verify that the etcd pods are stopped.

      1. [core@ip-10-0-154-194 ~]$ sudo crictl ps | grep etcd | grep -v operator

      The output of this command should be empty. If it is not empty, wait a few minutes and check again.

    4. Move the existing Kubernetes API server pod file out of the kubelet manifest directory:

      1. [core@ip-10-0-154-194 ~]$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp

    5. Verify that the Kubernetes API server pods are stopped.

      1. [core@ip-10-0-154-194 ~]$ sudo crictl ps | grep kube-apiserver | grep -v operator

      The output of this command should be empty. If it is not empty, wait a few minutes and check again.

    6. Move the etcd data directory to a different location:

      1. [core@ip-10-0-154-194 ~]$ sudo mv /var/lib/etcd/ /tmp

    7. Repeat this step on each of the other control plane hosts that is not the recovery host.

  5. Access the recovery control plane host.

  6. If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables.

    You can check whether the proxy is enabled by reviewing the output of oc get proxy cluster -o yaml. The proxy is enabled if the httpProxy, httpsProxy, and noProxy fields have values set.

  7. Run the restore script on the recovery control plane host and pass in the path to the etcd backup directory:

    1. [core@ip-10-0-143-125 ~]$ sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup

    Example script output

    1. ...stopping kube-scheduler-pod.yaml
    2. ...stopping kube-controller-manager-pod.yaml
    3. ...stopping etcd-pod.yaml
    4. ...stopping kube-apiserver-pod.yaml
    5. Waiting for container etcd to stop
    6. .complete
    7. Waiting for container etcdctl to stop
    8. .............................complete
    9. Waiting for container etcd-metrics to stop
    10. complete
    11. Waiting for container kube-controller-manager to stop
    12. complete
    13. Waiting for container kube-apiserver to stop
    14. ..........................................................................................complete
    15. Waiting for container kube-scheduler to stop
    16. complete
    17. Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup
    18. starting restore-etcd static pod
    19. starting kube-apiserver-pod.yaml
    20. static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
    21. starting kube-controller-manager-pod.yaml
    22. static-pod-resources/kube-controller-manager-pod-7/kube-controller-manager-pod.yaml
    23. starting kube-scheduler-pod.yaml
    24. static-pod-resources/kube-scheduler-pod-8/kube-scheduler-pod.yaml

  8. Restart the kubelet service on all control plane hosts.

    1. From the recovery host, run the following command:

      1. [core@ip-10-0-143-125 ~]$ sudo systemctl restart kubelet.service

    2. Repeat this step on all other control plane hosts.

  9. Approve the pending CSRs:

    1. Get the list of current CSRs:

      1. $ oc get csr

      Example output

      1. NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
      2. csr-2s94x   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending (1)
      3. csr-4bd6t   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending (1)
      4. csr-4hl85   13m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending (2)
      5. csr-zhhhp   3m8s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending (2)
      6. ...
      1A pending kubelet service CSR (for user-provisioned installations).
      2A pending node-bootstrapper CSR.

    2. Review the details of a CSR to verify that it is valid:

      1. $ oc describe csr <csr_name> (1)
      1<csr_name> is the name of a CSR from the list of current CSRs.

    3. Approve each valid node-bootstrapper CSR:

      1. $ oc adm certificate approve <csr_name>

    4. For user-provisioned installations, approve each valid kubelet service CSR:

      1. $ oc adm certificate approve <csr_name>

  10. Verify that the single member control plane has started successfully.

    1. From the recovery host, verify that the etcd container is running.

      1. [core@ip-10-0-143-125 ~]$ sudo crictl ps | grep etcd | grep -v operator

      Example output

      1. 3ad41b7908e32       36f86e2eeaaffe662df0d21041eb22b8198e0e58abeeae8c743c3e6e977e8009                                                         About a minute ago   Running             etcd                                          0                   7c05f8af362f0

    2. From the recovery host, verify that the etcd pod is running.

      1. [core@ip-10-0-143-125 ~]$ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd

      If you attempt to run oc login prior to running this command and receive the following error, wait a few moments for the authentication controllers to start and try again.

      1. Unable to connect to the server: EOF

      Example output

      1. NAME                                             READY   STATUS      RESTARTS   AGE
      2. etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1          2m47s

      If the status is Pending, or the output lists more than one running etcd pod, wait a few minutes and check again.

  11. Force etcd redeployment.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command:

    1. $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge (1)
    1The forceRedeploymentReason value must be unique, which is why a timestamp is appended.

    When the etcd cluster Operator performs a redeployment, the existing nodes are started with new pods similar to the initial bootstrap scale up.

  12. Verify all nodes are updated to the latest revision.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command:

    1. $ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

    Review the NodeInstallerProgressing status condition for etcd to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

    1. AllNodesAtLatestRevision
    2. 3 nodes are at revision 7 (1)
    1In this example, the latest revision number is 7.

    If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

  13. After etcd is redeployed, force new rollouts for the control plane. The Kubernetes API server will reinstall itself on the other nodes because the kubelet is connected to API servers using an internal load balancer.

    In a terminal that has access to the cluster as a cluster-admin user, run the following commands.

    1. Update the kubeapiserver:

      1. $ oc patch kubeapiserver cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

      Verify all nodes are updated to the latest revision.

      1. $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

      Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

      1. AllNodesAtLatestRevision
      2. 3 nodes are at revision 7 (1)
      1In this example, the latest revision number is 7.

      If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

    2. Update the kubecontrollermanager:

      1. $ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

      Verify all nodes are updated to the latest revision.

      1. $ oc get kubecontrollermanager -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

      Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

      1. AllNodesAtLatestRevision
      2. 3 nodes are at revision 7 (1)
      1In this example, the latest revision number is 7.

      If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

    3. Update the kubescheduler:

      1. $ oc patch kubescheduler cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

      Verify all nodes are updated to the latest revision.

      1. $ oc get kubescheduler -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

      Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

      1. AllNodesAtLatestRevision
      2. 3 nodes are at revision 7 (1)
      1In this example, the latest revision number is 7.

      If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

  14. Verify that all control plane hosts have started and joined the cluster.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command:

    1. $ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd

    Example output

    1. etcd-ip-10-0-143-125.ec2.internal                2/2     Running     0          9h
    2. etcd-ip-10-0-154-194.ec2.internal                2/2     Running     0          9h
    3. etcd-ip-10-0-173-171.ec2.internal                2/2     Running     0          9h

Note that it might take several minutes after completing this procedure for all services to be restored. For example, authentication by using oc login might not immediately work until the OAuth server pods are restarted.

Pod disruption budgets

Understand and configure pod disruption budgets.

Understanding how to use pod disruption budgets to specify the number of pods that must be up

A pod disruption budget is part of the Kubernetes API, which can be managed with oc commands like other object types. They allow the specification of safety constraints on pods during operations, such as draining a node for maintenance.

PodDisruptionBudget is an API object that specifies the minimum number or percentage of replicas that must be up at a time. Setting these in projects can be helpful during node maintenance (such as scaling a cluster down or a cluster upgrade) and is only honored on voluntary evictions (not on node failures).

A PodDisruptionBudget object’s configuration consists of the following key parts:

  • A label selector, which is a label query over a set of pods.

  • An availability level, which specifies the minimum number of pods that must be available simultaneously, either:

    • minAvailable is the number of pods must always be available, even during a disruption.

    • maxUnavailable is the number of pods can be unavailable during a disruption.

A maxUnavailable of 0% or 0 or a minAvailable of 100% or equal to the number of replicas is permitted but can block nodes from being drained.

You can check for pod disruption budgets across all projects with the following:

  1. $ oc get poddisruptionbudget --all-namespaces

Example output

  1. NAMESPACE         NAME          MIN-AVAILABLE   SELECTOR
  2. another-project   another-pdb   4               bar=foo
  3. test-project      my-pdb        2               foo=bar

The PodDisruptionBudget is considered healthy when there are at least minAvailable pods running in the system. Every pod above that limit can be evicted.

Depending on your pod priority and preemption settings, lower-priority pods might be removed despite their pod disruption budget requirements.

Specifying the number of pods that must be up with pod disruption budgets

You can use a PodDisruptionBudget object to specify the minimum number or percentage of replicas that must be up at a time.

Procedure

To configure a pod disruption budget:

  1. Create a YAML file with the an object definition similar to the following:

    1. apiVersion: policy/v1beta1 (1)
    2. kind: PodDisruptionBudget
    3. metadata:
    4.   name: my-pdb
    5. spec:
    6.   minAvailable: 2  (2)
    7.   selector:  (3)
    8.     matchLabels:
    9.       foo: bar
    1PodDisruptionBudget is part of the policy/v1beta1 API group.
    2The minimum number of pods that must be available simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.
    3A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined.

    Or:

    1. apiVersion: policy/v1beta1 (1)
    2. kind: PodDisruptionBudget
    3. metadata:
    4.   name: my-pdb
    5. spec:
    6.   maxUnavailable: 25% (2)
    7.   selector: (3)
    8.     matchLabels:
    9.       foo: bar
    1PodDisruptionBudget is part of the policy/v1beta1 API group.
    2The maximum number of pods that can be unavailable simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.
    3A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined.

  2. Run the following command to add the object to project:

    1. $ oc create -f </path/to/file> -n <project_name>

Rotating or removing cloud provider credentials

After installing OKD, some organizations require the rotation or removal of the cloud provider credentials that were used during the initial installation.

To allow the cluster to use the new credentials, you must update the secrets that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials.

Rotating cloud provider credentials manually

If your cloud provider credentials are changed for any reason, you must manually update the secret that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials.

The process for rotating cloud credentials depends on the mode that the CCO is configured to use. After you rotate credentials for a cluster that is using mint mode, you must manually remove the component credentials that were created by the removed credential.

Prerequisites

  • Your cluster is installed on a platform that supports rotating cloud credentials manually with the CCO mode that you are using:

    • For mint mode, Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP) are supported.

    • For passthrough mode, AWS, Azure, GCP, Red Hat OpenStack Platform (RHOSP), oVirt, and VMware vSphere are supported.

  • You are using OKD version 4.6.18 or later.

  • You have changed the credentials that are used to interface with your cloud provider.

  • The new credentials have sufficient permissions for the mode CCO is configured to use in your cluster.

Procedure

  1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.

  2. In the table on the Secrets page, find the root secret for your cloud provider.

    PlatformSecret name

    AWS

    aws-creds

    Azure

    azure-credentials

    GCP

    gcp-credentials

    RHOSP

    openstack-credentials

    oVirt

    ovirt-credentials

    vSphere

    vsphere-creds

  3. Click the Options menu kebab in the same row as the secret and select Edit Secret.

  4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.

  5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

  6. If the CCO for your cluster is configured to use mint mode, delete each component secret that is referenced by the individual CredentialsRequest objects.

    1. Log in to the OKD CLI as a user with the cluster-admin role.

    2. Get the names and namespaces of all referenced component secrets:

      1. $ oc -n openshift-cloud-credential-operator get CredentialsRequest -o json | jq -r '.items[] | select (.spec[].kind=="<provider_spec>") | .spec.secretRef'

      Where <provider_spec> is the corresponding value for your cloud provider:

      Platform<provider_spec>

      AWS

      AWSProviderSpec

      Azure

      AzureProviderSpec

      GCP

      GCPProviderSpec

      Partial example output for AWS

      1. {
      2.   "name": "ebs-cloud-credentials",
      3.   "namespace": "openshift-cluster-csi-drivers"
      4. }
      5. {
      6.   "name": "cloud-credential-operator-iam-ro-creds",
      7.   "namespace": "openshift-cloud-credential-operator"
      8. }
      9. ...

    3. Delete each of the referenced component secrets:

      1. $ oc delete secret <secret_name> -n <secret_namespace>

      Where <secret_name> is the name of a secret and <secret_namespace> is the namespace that contains the secret.

      Example deletion of an AWS secret

      1. $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

  7. To verify that the credentials have changed:

    1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.

    2. Verify that the contents of the Value field or fields are different than the previously recorded information.

Removing cloud provider credentials

After installing an OKD cluster on Amazon Web Services (AWS) with the Cloud Credential Operator (CCO) in mint mode, you can remove the administrator-level credential secret from the kube-system namespace in the cluster. The administrator-level credential is required only during changes that require its elevated permissions, such as upgrades.

Prior to a non z-stream upgrade, you must reinstate the credential secret with the administrator-level credential. If the credential is not present, the upgrade might be blocked.

Prerequisites

  • Your cluster is installed on AWS with the CCO configured to use mint mode.

Procedure

  1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.

  2. In the table on the Secrets page, find the aws-creds root secret for AWS.

    PlatformSecret name

    AWS

    aws-creds

  3. Click the Options menu kebab in the same row as the secret and select Delete Secret.

Configuring image streams for a disconnected cluster

After installing OKD in a disconnected environment, configure the image streams for the Cluster Samples Operator and the must-gather image stream.

Using Cluster Samples Operator image streams with alternate or mirrored registries

Most image streams in the openshift namespace managed by the Cluster Samples Operator point to images located in the Red Hat registry at registry.redhat.io. Mirroring will not apply to these image streams.

The jenkins, jenkins-agent-maven, and jenkins-agent-nodejs image streams come from the install payload and are managed by the Samples Operator, so no further mirroring procedures are needed for those image streams.

Setting the samplesRegistry field in the Sample Operator configuration file to registry.redhat.io is redundant because it is already directed to registry.redhat.io for everything but Jenkins images and image streams.

The cli, installer, must-gather, and tests image streams, while part of the install payload, are not managed by the Cluster Samples Operator. These are not addressed in this procedure.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

  • Create a pull secret for your mirror registry.

Procedure

  1. Access the images of a specific image stream to mirror, for example:

    1. $ oc get is <imagestream> -n openshift -o json | jq .spec.tags[].from.name | grep registry.redhat.io

  2. Mirror images from registry.redhat.io associated with any image streams you need in the restricted network environment into one of the defined mirrors, for example:

    1. $ oc image mirror registry.redhat.io/rhscl/ruby-25-rhel7:latest ${MIRROR_ADDR}/rhscl/ruby-25-rhel7:latest

  3. Create the cluster’s image configuration object:

    1. $ oc create configmap registry-config --from-file=${MIRROR_ADDR_HOSTNAME}..5000=$path/ca.crt -n openshift-config

  4. Add the required trusted CAs for the mirror in the cluster’s image configuration object:

    1. $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge

  5. Update the samplesRegistry field in the Cluster Samples Operator configuration object to contain the hostname portion of the mirror location defined in the mirror configuration:

    1. $ oc edit configs.samples.operator.openshift.io -n openshift-cluster-samples-operator

    This is required because the image stream import process does not use the mirror or search mechanism at this time.

  6. Add any image streams that are not mirrored into the skippedImagestreams field of the Cluster Samples Operator configuration object. Or if you do not want to support any of the sample image streams, set the Cluster Samples Operator to Removed in the Cluster Samples Operator configuration object.

    The Cluster Samples Operator issues alerts if image stream imports are failing but the Cluster Samples Operator is either periodically retrying or does not appear to be retrying them.

    Many of the templates in the openshift namespace reference the image streams. So using Removed to purge both the image streams and templates will eliminate the possibility of attempts to use them if they are not functional because of any missing image streams.

Preparing your cluster to gather support data

Clusters using a restricted network must import the default must-gather image in order to gather debugging data for Red Hat support. The must-gather image is not imported by default, and clusters on a restricted network do not have access to the internet to pull the latest image from a remote repository.

Procedure

  1. If you have not added your mirror registry’s trusted CA to your cluster’s image configuration object as part of the Cluster Samples Operator configuration, perform the following steps:

    1. Create the cluster’s image configuration object:

      1. $ oc create configmap registry-config --from-file=${MIRROR_ADDR_HOSTNAME}..5000=$path/ca.crt -n openshift-config

    2. Add the required trusted CAs for the mirror in the cluster’s image configuration object:

      1. $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge

  2. Import the default must-gather image from your installation payload:

    1. $ oc import-image is/must-gather -n openshift

When running the oc adm must-gather command, use the --image flag and point to the payload image, as in the following example:

  1. $ oc adm must-gather --image=$(oc adm release info --image-for must-gather)

Additional resources