Configuring an OKD cluster for pods

As an administrator, you can create and maintain an efficient cluster for pods.

By keeping your cluster efficient, you can provide a better environment for your developers using such tools as what a pod does when it exits, ensuring that the required number of pods is always running, when to restart pods designed to run only once, limit the bandwidth available to pods, and how to keep pods running during disruptions.

Configuring how pods behave after restart

A pod restart policy determines how OKD responds when Containers in that pod exit. The policy applies to all Containers in that pod.

The possible values are:

• Always - Tries restarting a successfully exited Container on the pod continuously, with an exponential back-off delay (10s, 20s, 40s) until the pod is restarted. The default is Always.

• OnFailure - Tries restarting a failed Container on the pod with an exponential back-off delay (10s, 20s, 40s) capped at 5 minutes.

• Never - Does not try to restart exited or failed Containers on the pod. Pods immediately fail and exit.

After the pod is bound to a node, the pod will never be bound to another node. This means that a controller is necessary in order for a pod to survive node failure:

If a Container on a pod fails and the restart policy is set to OnFailure, the pod stays on the node and the Container is restarted. If you do not want the Container to restart, use a restart policy of Never.

If an entire pod fails, OKD starts a new pod. Developers must address the possibility that applications might be restarted in a new pod. In particular, applications must handle temporary files, locks, incomplete output, and so forth caused by previous runs.

For details on how OKD uses restart policy with failed Containers, see the Example States in the Kubernetes documentation.

Limiting the bandwidth available to pods

You can apply quality-of-service traffic shaping to a pod and effectively limit its available bandwidth. Egress traffic (from the pod) is handled by policing, which simply drops packets in excess of the configured rate. Ingress traffic (to the pod) is handled by shaping queued packets to effectively handle data. The limits you place on a pod do not affect the bandwidth of other pods.

Procedure

To limit the bandwidth on a pod:

1. Write an object definition JSON file, and specify the data traffic speed using kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations. For example, to limit both pod egress and ingress bandwidth to 10M/s:

   Limited Pod Object Definition

   {
       "kind": "Pod",
       "spec": {
           "containers": [
               {
                   "image": "openshift/hello-openshift",
                   "name": "hello-openshift"
               }
           ]
       },
       "apiVersion": "v1",
       "metadata": {
           "name": "iperf-slow",
           "annotations": {
               "kubernetes.io/ingress-bandwidth": "10M",
               "kubernetes.io/egress-bandwidth": "10M"
           }
       }
   }

2. Create the pod using the object definition:

   $ oc create -f <file_or_dir_path>

Understanding how to use pod disruption budgets to specify the number of pods that must be up

A pod disruption budget is part of the Kubernetes API, which can be managed with oc commands like other object types. They allow the specification of safety constraints on pods during operations, such as draining a node for maintenance.

PodDisruptionBudget is an API object that specifies the minimum number or percentage of replicas that must be up at a time. Setting these in projects can be helpful during node maintenance (such as scaling a cluster down or a cluster upgrade) and is only honored on voluntary evictions (not on node failures).

A PodDisruptionBudget object’s configuration consists of the following key parts:

• A label selector, which is a label query over a set of pods.

• An availability level, which specifies the minimum number of pods that must be available simultaneously, either:

  • minAvailable is the number of pods must always be available, even during a disruption.

  • maxUnavailable is the number of pods can be unavailable during a disruption.

You can check for pod disruption budgets across all projects with the following:

$ oc get poddisruptionbudget --all-namespaces

Example output

NAMESPACE         NAME          MIN-AVAILABLE   SELECTOR
another-project   another-pdb   4               bar=foo
test-project      my-pdb        2               foo=bar

The PodDisruptionBudget is considered healthy when there are at least minAvailable pods running in the system. Every pod above that limit can be evicted.

Specifying the number of pods that must be up with pod disruption budgets

You can use a PodDisruptionBudget object to specify the minimum number or percentage of replicas that must be up at a time.

Procedure

To configure a pod disruption budget:

1. Create a YAML file with the an object definition similar to the following:

   apiVersion: policy/v1beta1 (1)
   kind: PodDisruptionBudget
   metadata:
     name: my-pdb
   spec:
     minAvailable: 2  (2)
     selector:  (3)
       matchLabels:
         foo: bar

   1	PodDisruptionBudget is part of the policy/v1beta1 API group.
   2	The minimum number of pods that must be available simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.
   3	A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined.

   Or:

   apiVersion: policy/v1beta1 (1)
   kind: PodDisruptionBudget
   metadata:
     name: my-pdb
   spec:
     maxUnavailable: 25% (2)
     selector: (3)
       matchLabels:
         foo: bar

   1	PodDisruptionBudget is part of the policy/v1beta1 API group.
   2	The maximum number of pods that can be unavailable simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.
   3	A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined.

2. Run the following command to add the object to project:

   $ oc create -f </path/to/file> -n <project_name>

Preventing pod removal using critical pods

There are a number of core components that are critical to a fully functional cluster, but, run on a regular cluster node rather than the master. A cluster might stop working properly if a critical add-on is evicted.

Pods marked as critical are not allowed to be evicted.

Procedure

To make a pod critical:

1. Create a Pod spec or edit existing pods to include the system-cluster-critical priority class:

   spec:
     template:
       metadata:
         name: critical-pod
       priorityClassName: system-cluster-critical (1)

   1	Default priority class for pods that should never be evicted from a node.

   Alternatively, you can specify system-node-critical for pods that are important to the cluster but can be removed if necessary.

2. Create the pod:

   $ oc create -f <file-name>.yaml