Taints and Tolerations

If users want to create nodes with large storage spaces and/or CPU resources for Longhorn only (to store replica data) and reject other general workloads, they can taint those nodes and add tolerations for Longhorn components. Then Longhorn can be deployed on those nodes.

Notice that the taint tolerations setting for one workload will not prevent it from being scheduled to the nodes that don’t contain the corresponding taints.

For more information about how taints and tolerations work, refer to the official Kubernetes documentation.

Setting up Taints and Tolerations

Longhorn consists of user-deployed components (for example, Longhorn Manager, Longhorn Driver, and Longhorn UI) and system-managed components (for example, Instance Manager, Backing Image Manager, Share Manager, CSI Driver, and Engine Image). You need to set tolerations for both types of components. See more details below.

Setting up Taints and Tolerations During installing Longhorn

  1. Set taint tolerations for user-deployed components (for example, Longhorn Manager, Longhorn Driver, and Longhorn UI).

    • If you install Longhorn through Rancher, you must copy and paste the following parameters into the YAML on the Rancher UI (click Edit as YAML during the installation) to apply the value to all user-deployed components.

      1.   global:
      2.     tolerations:
      3.     - key: "key"
      4.       operator: "Equal"
      5.       value: "value"
      6.       effect: "NoSchedule"
    1. * You can also specify the tolerations for each user-deployed component and it will override the global setting.
    2.  ```yaml
    3.    longhornManager:
    4.      tolerations:
    5.      - key: "key"
    6.        operator: "Equal"
    7.        value: "value"
    8.        effect: "NoSchedule"
    9.    longhornDriver:
    10.      tolerations:
    11.      - key: "key"
    12.        operator: "Equal"
    13.        value: "value"
    14.        effect: "NoSchedule"
    15.    longhornUI:
    16.      tolerations:
    17.      - key: "key"
    18.        operator: "Equal"
    19.        value: "value"
    20.        effect: "NoSchedule"
    • If you install Longhorn by using kubectl to apply the deployment YAML, you need to modify the taint tolerations section for Longhorn Manager, Longhorn UI, and Longhorn Driver Deployer. Then apply the YAMl files.
    • If you install Longhorn using Helm, you can change the Helm values for global.tolerations, longhornManager.tolerations, longhornUI.tolerations, longhornDriver.tolerations in the values.yaml file before installing the chart.

  2. Set taint tolerations for system-managed components (for example, Instance Manager, CSI Driver, and Engine images)

    Follow the Customize default settings to set taint tolerations by changing the value for the taint-toleratio default setting

    Note: Because of the limitation of Rancher 2.5.x, if you are using Rancher UI to install Longhorn, you need to click Edit As Yaml and add setting taintToleration to defaultSettings.

    For example:

    1. defaultSettings:
    2.   taintToleration: "key=value:NoSchedule"

Setting up Taints and Tolerations After Longhorn has been installed

Warning:

To ensure that your preferred toleration settings are immediately applied, stop all workloads and detach all Longhorn volumes before configuring the settings.

Since all Longhorn components will be restarted, the Longhorn system is unavailable temporarily.

When all Longhorn volumes are detached, the customized setting is immediately applied to the system-managed components. When one or more Longhorn volumes are still attached, the customized setting is applied to the Instance Manager only when no engines and replica instances are running. You are required to reconfigure the setting after detaching the remaining volumes. Alternatively, you can wait for the next setting synchronization, which will occur in an hour.

Don’t operate the Longhorn system while toleration settings are updated and Longhorn components are being restarted.

  1. Prepare

    To ensure that your preferred settings are immediately applied, stop all workloads and detach all Longhorn volumes before configuring the settings.

  2. Set taint tolerations for user-deployed components (for example, Longhorn Manager, Longhorn Driver, and Longhorn UI).

    • If you install Longhorn through Rancher, you must copy and paste the following parameters into the YAML on the Rancher UI (click Edit as YAML during the upgrade) to apply the value to all user-deployed components.

      1.   global:
      2.     tolerations:
      3.     - key: "key"
      4.       operator: "Equal"
      5.       value: "value"
      6.       effect: "NoSchedule"
  1. ```
  2. * You can also specify the tolerations for each user-deployed component and it will override the global setting.
  3.  ```yaml
  4.    longhornManager:
  5.      tolerations:
  6.      - key: "key"
  7.        operator: "Equal"
  8.        value: "value"
  9.        effect: "NoSchedule"
  10.    longhornDriver:
  11.      tolerations:
  12.      - key: "key"
  13.        operator: "Equal"
  14.        value: "value"
  15.        effect: "NoSchedule"
  16.    longhornUI:
  17.      tolerations:
  18.      - key: "key"
  19.        operator: "Equal"
  20.        value: "value"
  21.        effect: "NoSchedule"
  22. ```
  24. -   If you install Longhorn by using `kubectl` to apply [the deployment YAML](https://raw.githubusercontent.com/longhorn/longhorn/v1.1.1/deploy/longhorn.yaml), you need to modify the taint tolerations section for Longhorn Manager, Longhorn UI, and Longhorn Driver Deployer. Then reapply the YAMl files.
  25. -   If you install Longhorn using Helm, you can change the Helm values for `global.tolerations`, `longhornManager.tolerations`, `longhornUI.tolerations`, `longhornDriver.tolerations` in the `values.yaml` file, and then run `helm upgrade` to upgrade to the new version of the chart.

  1. Set taint tolerations for system-managed components (for example, Instance Manager, Backing Image Manager, Share Manager, CSI Driver, and Engine Image).

    The taint toleration setting can be found at Longhorn UI under Setting > General > Kubernetes Taint Toleration.

History

Available since v0.6.0