Using tolerations to control OpenShift Logging pod placement

You can use taints and tolerations to ensure that OpenShift Logging pods run on specific nodes and that no other workload can run on those nodes.

Taints and tolerations are simple key:value pair. A taint on a node instructs the node to repel all pods that do not tolerate the taint.

The key is any string, up to 253 characters and the value is any string up to 63 characters. The string must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

Sample OpenShift Logging CR with tolerations

  1. apiVersion: "logging.openshift.io/v1"
  2. kind: "ClusterLogging"
  3. metadata:
  4.   name: "instance"
  5.   namespace: openshift-logging
  7. ...
  9. spec:
  10.   managementState: "Managed"
  11.   logStore:
  12.     type: "elasticsearch"
  13.     elasticsearch:
  14.       nodeCount: 3
  15.       tolerations: (1)
  16.       - key: "logging"
  17.         operator: "Exists"
  18.         effect: "NoExecute"
  19.         tolerationSeconds: 6000
  20.       resources:
  21.         limits:
  22.           memory: 16Gi
  23.         requests:
  24.           cpu: 200m
  25.           memory: 16Gi
  26.       storage: {}
  27.       redundancyPolicy: "ZeroRedundancy"
  28.   visualization:
  29.     type: "kibana"
  30.     kibana:
  31.       tolerations: (2)
  32.       - key: "logging"
  33.         operator: "Exists"
  34.         effect: "NoExecute"
  35.         tolerationSeconds: 6000
  36.       resources:
  37.         limits:
  38.           memory: 2Gi
  39.         requests:
  40.           cpu: 100m
  41.           memory: 1Gi
  42.       replicas: 1
  43.   collection:
  44.     logs:
  45.       type: "fluentd"
  46.       fluentd:
  47.         tolerations: (3)
  48.         - key: "logging"
  49.           operator: "Exists"
  50.           effect: "NoExecute"
  51.           tolerationSeconds: 6000
  52.         resources:
  53.           limits:
  54.             memory: 2Gi
  55.           requests:
  56.             cpu: 100m
  57.             memory: 1Gi
1This toleration is added to the Elasticsearch pods.
2This toleration is added to the Kibana pod.
3This toleration is added to the logging collector pods.

Using tolerations to control the log store pod placement

You can control which nodes the log store pods runs on and prevent other workloads from using those nodes by using tolerations on the pods.

You apply tolerations to the log store pods through the ClusterLogging custom resource (CR) and apply taints to a node through the node specification. A taint on a node is a key:value pair that instructs the node to repel all pods that do not tolerate the taint. Using a specific key:value pair that is not on other pods ensures only the log store pods can run on that node.

By default, the log store pods have the following toleration:

  1. tolerations:
  2. - effect: "NoExecute"
  3.   key: "node.kubernetes.io/disk-pressure"
  4.   operator: "Exists"

Prerequisites

  • OpenShift Logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want to schedule the OpenShift Logging pods:

    1. $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    1. $ oc adm taint nodes node1 elasticsearch=node:NoExecute

    This example places a taint on node1 that has key elasticsearch, value node, and taint effect NoExecute. Nodes with the NoExecute effect schedule only pods that match the taint and remove existing pods that do not match.

  2. Edit the logstore section of the ClusterLogging CR to configure a toleration for the Elasticsearch pods:

    1.   logStore:
    2.     type: "elasticsearch"
    3.     elasticsearch:
    4.       nodeCount: 1
    5.       tolerations:
    6.       - key: "elasticsearch"  (1)
    7.         operator: "Exists"  (2)
    8.         effect: "NoExecute"  (3)
    9.         tolerationSeconds: 6000  (4)
    1Specify the key that you added to the node.
    2Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.
    3Specify the NoExecute effect.
    4Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A pod with this toleration could be scheduled onto node1.

Using tolerations to control the log visualizer pod placement

You can control the node where the log visualizer pod runs and prevent other workloads from using those nodes by using tolerations on the pods.

You apply tolerations to the log visualizer pod through the ClusterLogging custom resource (CR) and apply taints to a node through the node specification. A taint on a node is a key:value pair that instructs the node to repel all pods that do not tolerate the taint. Using a specific key:value pair that is not on other pods ensures only the Kibana pod can run on that node.

Prerequisites

  • OpenShift Logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want to schedule the log visualizer pod:

    1. $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    1. $ oc adm taint nodes node1 kibana=node:NoExecute

    This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and remove existing pods that do not match.

  2. Edit the visualization section of the ClusterLogging CR to configure a toleration for the Kibana pod:

    1.   visualization:
    2.     type: "kibana"
    3.     kibana:
    4.       tolerations:
    5.       - key: "kibana"  (1)
    6.         operator: "Exists"  (2)
    7.         effect: "NoExecute"  (3)
    8.         tolerationSeconds: 6000 (4)
    1Specify the key that you added to the node.
    2Specify the Exists operator to require the key/value/effect parameters to match.
    3Specify the NoExecute effect.
    4Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A pod with this toleration would be able to schedule onto node1.

Using tolerations to control the log collector pod placement

You can ensure which nodes the logging collector pods run on and prevent other workloads from using those nodes by using tolerations on the pods.

You apply tolerations to logging collector pods through the ClusterLogging custom resource (CR) and apply taints to a node through the node specification. You can use taints and tolerations to ensure the pod does not get evicted for things like memory and CPU issues.

By default, the logging collector pods have the following toleration:

  1. tolerations:
  2. - key: "node-role.kubernetes.io/master"
  3.   operator: "Exists"
  4.   effect: "NoExecute"

Prerequisites

  • OpenShift Logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want logging collector pods to schedule logging collector pods:

    1. $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    1. $ oc adm taint nodes node1 collector=node:NoExecute

    This example places a taint on node1 that has key collector, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and removes existing pods that do not match.

  2. Edit the collection stanza of the ClusterLogging custom resource (CR) to configure a toleration for the logging collector pods:

    1.   collection:
    2.     logs:
    3.       type: "fluentd"
    4.       fluentd:
    5.         tolerations:
    6.         - key: "collector"  (1)
    7.           operator: "Exists"  (2)
    8.           effect: "NoExecute"  (3)
    9.           tolerationSeconds: 6000  (4)
    1Specify the key that you added to the node.
    2Specify the Exists operator to require the key/value/effect parameters to match.
    3Specify the NoExecute effect.
    4Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A pod with this toleration would be able to schedule onto node1.

Additional resources

For more information about taints and tolerations, see Controlling pod placement using node taints.