Using tolerations to control cluster logging pod placement

You can use taints and tolerations to ensure that cluster logging pods run on specific nodes and that no other workload can run on those nodes.

Taints and tolerations are simple key:value pair. A taint on a node instructs the node to repel all pods that do not tolerate the taint.

The key is any string, up to 253 characters and the value is any string up to 63 characters. The string must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

Sample cluster logging CR with tolerations

  1. apiVersion: "logging.openshift.io/v1"
  2. kind: "ClusterLogging"
  3. metadata:
  4.   name: "instance"
  5.   namespace: openshift-logging
  6. spec:
  7.   managementState: "Managed"
  8.   logStore:
  9.     type: "elasticsearch"
  10.     elasticsearch:
  11.       nodeCount: 1
  12.       tolerations: (1)
  13.       - key: "logging"
  14.         operator: "Exists"
  15.         effect: "NoExecute"
  16.         tolerationSeconds: 6000
  17.       resources:
  18.         limits:
  19.           memory: 8Gi
  20.         requests:
  21.           cpu: 100m
  22.           memory: 1Gi
  23.       storage: {}
  24.       redundancyPolicy: "ZeroRedundancy"
  25.   visualization:
  26.     type: "kibana"
  27.     kibana:
  28.       tolerations: (2)
  29.       - key: "logging"
  30.         operator: "Exists"
  31.         effect: "NoExecute"
  32.         tolerationSeconds: 6000
  33.       resources:
  34.         limits:
  35.           memory: 2Gi
  36.         requests:
  37.           cpu: 100m
  38.           memory: 1Gi
  39.       replicas: 1
  40.   collection:
  41.     logs:
  42.       type: "fluentd"
  43.       fluentd:
  44.         tolerations: (3)
  45.         - key: "logging"
  46.           operator: "Exists"
  47.           effect: "NoExecute"
  48.           tolerationSeconds: 6000
  49.         resources:
  50.           limits:
  51.             memory: 2Gi
  52.           requests:
  53.             cpu: 100m
  54.             memory: 1Gi
1This toleration is added to the Elasticsearch pods.
2This toleration is added to the Kibana pod.
3This toleration is added to the logging collector pods.

Using tolerations to control the log store pod placement

You can control which nodes the log store pods runs on and prevent other workloads from using those nodes by using tolerations on the pods.

You apply tolerations to the log store pods through the ClusterLogging custom resource (CR) and apply taints to a node through the node specification. A taint on a node is a key:value pair that instructs the node to repel all pods that do not tolerate the taint. Using a specific key:value pair that is not on other pods ensures only the log store pods can run on that node.

By default, the log store pods have the following toleration:

  1. tolerations:
  2. - effect: "NoExecute"
  3.   key: "node.kubernetes.io/disk-pressure"
  4.   operator: "Exists"

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want to schedule the cluster logging pods:

    1. $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    1. $ oc adm taint nodes node1 elasticsearch=node:NoExecute

    This example places a taint on node1 that has key elasticsearch, value node, and taint effect NoExecute. Nodes with the NoExecute effect schedule only pods that match the taint and remove existing pods that do not match.

  2. Edit the logstore section of the ClusterLogging CR to configure a toleration for the Elasticsearch pods:

    1.   logStore:
    2.     type: "elasticsearch"
    3.     elasticsearch:
    4.       nodeCount: 1
    5.       tolerations:
    6.       - key: "elasticsearch"  (1)
    7.         operator: "Exists"  (2)
    8.         effect: "NoExecute"  (3)
    9.         tolerationSeconds: 6000  (4)
    1Specify the key that you added to the node.
    2Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.
    3Specify the NoExecute effect.
    4Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A pod with this toleration could be scheduled onto node1.

Using tolerations to control the log visualizer pod placement

You can control the node where the log visualizer pod runs and prevent other workloads from using those nodes by using tolerations on the pods.

You apply tolerations to the log visualizer pod through the ClusterLogging custom resource (CR) and apply taints to a node through the node specification. A taint on a node is a key:value pair that instructs the node to repel all pods that do not tolerate the taint. Using a specific key:value pair that is not on other pods ensures only the Kibana pod can run on that node.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want to schedule the log visualizer pod:

    1. $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    1. $ oc adm taint nodes node1 kibana=node:NoExecute

    This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and remove existing pods that do not match.

  2. Edit the visualization section of the ClusterLogging CR to configure a toleration for the Kibana pod:

    1.   visualization:
    2.     type: "kibana"
    3.     kibana:
    4.       tolerations:
    5.       - key: "kibana"  (1)
    6.         operator: "Exists"  (2)
    7.         effect: "NoExecute"  (3)
    8.         tolerationSeconds: 6000 (4)
    1Specify the key that you added to the node.
    2Specify the Exists operator to require the key/value/effect parameters to match.
    3Specify the NoExecute effect.
    4Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A pod with this toleration would be able to schedule onto node1.

Using tolerations to control the log collector pod placement

You can ensure which nodes the logging collector pods run on and prevent other workloads from using those nodes by using tolerations on the pods.

You apply tolerations to logging collector pods through the ClusterLogging custom resource (CR) and apply taints to a node through the node specification. You can use taints and tolerations to ensure the pod does not get evicted for things like memory and CPU issues.

By default, the logging collector pods have the following toleration:

  1. tolerations:
  2. - key: "node-role.kubernetes.io/master"
  3.   operator: "Exists"
  4.   effect: "NoExecute"

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want logging collector pods to schedule logging collector pods:

    1. $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    1. $ oc adm taint nodes node1 collector=node:NoExecute

    This example places a taint on node1 that has key collector, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only pods that match the taint and removes existing pods that do not match.

  2. Edit the collection stanza of the ClusterLogging custom resource (CR) to configure a toleration for the logging collector pods:

    1.   collection:
    2.     logs:
    3.       type: "fluentd"
    4.       fluentd:
    5.         tolerations:
    6.         - key: "collector"  (1)
    7.           operator: "Exists"  (2)
    8.           effect: "NoExecute"  (3)
    9.           tolerationSeconds: 6000  (4)
    1Specify the key that you added to the node.
    2Specify the Exists operator to require the key/value/effect parameters to match.
    3Specify the NoExecute effect.
    4Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A pod with this toleration would be able to schedule onto node1.

Additional resources

For more information about taints and tolerations, see Controlling pod placement using node taints.