Configure a TiDB Cluster in Kubernetes

This document introduces how to configure a TiDB cluster for production deployment. It covers the following content:

Configure resources

Before deploying a TiDB cluster, it is necessary to configure the resources for each component of the cluster depending on your needs. PD, TiKV, and TiDB are the core service components of a TiDB cluster. In a production environment, you need to configure resources of these components according to their needs. For details, refer to Hardware Recommendations.

To ensure the proper scheduling and stable operation of the components of the TiDB cluster in Kubernetes, it is recommended to set Guaranteed-level quality of service (QoS) by making limits equal to requests when configuring resources. For details, refer to Configure Quality of Service for Pods.

If you are using a NUMA-based CPU, you need to enable Static‘s CPU management policy on the node for better performance. In order to allow the TiDB cluster component to monopolize the corresponding CPU resources, the CPU quota must be an integer greater than or equal to 1, apart from setting Guaranteed-level QoS as mentioned above. For details, refer to Control CPU Management Policies on the Node.

Configure TiDB deployment

To configure a TiDB deployment, you need to configure the TiDBCluster CR. Refer to the TidbCluster example for an example. For the complete configurations of TiDBCluster CR, refer to API documentation.

Configure a TiDB Cluster - 图1Note

It is recommended to organize configurations for a TiDB cluster under a directory of cluster_name and save it as ${cluster_name}/tidb-cluster.yaml. The modified configuration is not automatically applied to the TiDB cluster by default. The new configuration file is loaded only when the Pod restarts.

Cluster name

The cluster name can be configured by changing metadata.name in the TiDBCuster CR.

Version

Usually, components in a cluster are in the same version. It is recommended to configure spec.<pd/tidb/tikv/pump/tiflash/ticdc>.baseImage and spec.version, if you need to configure different versions for different components, you can configure spec.<pd/tidb/tikv/pump/tiflash/ticdc>.version.

Here are the formats of the parameters:

  • spec.version: the format is imageTag, such as v5.4.0

  • spec.<pd/tidb/tikv/pump/tiflash/ticdc>.baseImage: the format is imageName, such as pingcap/tidb

  • spec.<pd/tidb/tikv/pump/tiflash/ticdc>.version: the format is imageTag, such as v5.4.0

configUpdateStrategy

It is recommended that you configure spec.configUpdateStrategy: RollingUpdate to enable automatic update of configurations. This way, every time the configuration is updated, all components are rolling updated automatically, and the modified configuration is applied to the cluster.

enableDynamicConfiguration

It is recommended that you configure spec.enableDynamicConfiguration: true to enable the --advertise-status-addr startup parameter for TiKV.

Versions required:

  • TiDB 4.0.1 or later versions

pvReclaimPolicy

It is recommended that you configure spec.pvReclaimPolicy: Retain to ensure that the PV is retained even if the PVC is deleted. This is to ensure your data safety.

mountClusterClientSecret

PD and TiKV supports configuring mountClusterClientSecret. If TLS is enabled between cluster components, it is recommended to configure spec.pd.mountClusterClientSecret: true and spec.tikv.mountClusterClientSecret: true. Under such configuration, TiDB Operator automatically mounts the ${cluster_name}-cluster-client-secret certificate to the PD and TiKV container, so you can conveniently use pd-ctl and tikv-ctl.

Storage

Storage Class

You can set the storage class by modifying storageClassName of each component in ${cluster_name}/tidb-cluster.yaml and ${cluster_name}/tidb-monitor.yaml. For the storage classes supported by the Kubernetes cluster, check with your system administrator.

Different components of a TiDB cluster have different disk requirements. Before deploying a TiDB cluster, refer to the Storage Configuration document to select an appropriate storage class for each component according to the storage classes supported by the current Kubernetes cluster and usage scenario.

Configure a TiDB Cluster - 图2Note

When you create the TiDB cluster, if you set a storage class that does not exist in the Kubernetes cluster, then the TiDB cluster creation goes to the Pending state. In this situation, you must destroy the TiDB cluster in Kubernetes and retry the creation.

Multiple disks mounting

TiDB Operator supports mounting multiple PVs for PD, TiDB, TiKV, and TiCDC, which can be used for data writing for different purposes.

You can configure the storageVolumes field for each component to describe multiple user-customized PVs.

The meanings of the related fields are as follows:

  • storageVolume.name: The name of the PV.
  • storageVolume.storageClassName: The StorageClass that the PV uses. If not configured, spec.pd/tidb/tikv/ticdc.storageClassName will be used.
  • storageVolume.storageSize: The storage size of the requested PV.
  • storageVolume.mountPath: The path of the container to mount the PV to.

For example:

  1.   tikv:
  2.     ...
  3.     config: |
  4.       [rocksdb]
  5.         wal-dir = "/data_sbi/tikv/wal"
  6.       [titan]
  7.         dirname = "/data_sbj/titan/data"
  8.     storageVolumes:
  9.     - name: wal
  10.       storageSize: "2Gi"
  11.       mountPath: "/data_sbi/tikv/wal"
  12.     - name: titan
  13.       storageSize: "2Gi"
  14.       mountPath: "/data_sbj/titan/data"
  1.   tidb:
  2.     config: |
  3.       path = "/tidb/data"
  4.       [log.file]
  5.         filename = "/tidb/log/tidb.log"
  6.     storageVolumes:
  7.     - name: data
  8.       storageSize: "2Gi"
  9.       mountPath: "/tidb/data"
  10.     - name: log
  11.       storageSize: "2Gi"
  12.       mountPath: "/tidb/log"
  1.   pd:
  2.     config: |
  3.       data-dir=/pd/data
  4.       [log.file]
  5.         filename=/pd/log/pd.log
  6.     storageVolumes:
  7.     - name: data
  8.       storageSize: "10Gi"
  9.       mountPath: "/pd/data"
  10.     - name: log
  11.       storageSize: "10Gi"
  12.       mountPath: "/pd/log"
  1.   ticdc:
  2.     ...
  3.     config:
  4.       dataDir: /ticdc/data
  5.       logFile: /ticdc/log/cdc.log
  6.     storageVolumes:
  7.     - name: data
  8.       storageSize: "10Gi"
  9.       storageClassName: local-storage
  10.       mountPath: "/ticdc/data"
  11.     - name: log
  12.       storageSize: "10Gi"
  13.       storageClassName: local-storage
  14.       mountPath: "/ticdc/log"

Configure a TiDB Cluster - 图3Note

TiDB Operator uses some mount paths by default. For example, it mounts EmptyDir to the /var/log/tidb directory for the TiDB Pod. Therefore, avoid duplicate mountPath when you configure storageVolumes.

HostNetwork

For PD, TiKV, TiDB, TiFlash, TiCDC, and Pump, you can configure the Pods to use the host namespace HostNetwork.

To enable HostNetwork for all supported components, configure spec.hostNetwork: true.

To enable HostNetwork for specified components, configure hostNetwork: true for the components.

Discovery

TiDB Operator starts a Discovery service for each TiDB cluster. The Discovery service can return the corresponding startup parameters for each PD Pod to support the startup of the PD cluster. You can configure resources of the Discovery service using spec.discovery. For details, see Managing Resources for Containers.

A spec.discovery configuration example is as follows:

  1. spec:
  2.   discovery:
  3.     limits:
  4.       cpu: "0.2"
  5.     requests:
  6.       cpu: "0.2"
  7.   ...

Cluster topology

PD/TiKV/TiDB

The deployed cluster topology by default has three PD Pods, three TiKV Pods, and two TiDB Pods. In this deployment topology, the scheduler extender of TiDB Operator requires at least three nodes in the Kubernetes cluster to provide high availability. You can modify the replicas configuration to change the number of pods for each component.

Configure a TiDB Cluster - 图4Note

If the number of Kubernetes cluster nodes is less than three, one PD Pod goes to the Pending state, and neither TiKV Pods nor TiDB Pods are created. When the number of nodes in the Kubernetes cluster is less than three, to start the TiDB cluster, you can reduce the number of PD Pods in the default deployment to 1.

Enable TiFlash

If you want to enable TiFlash in the cluster, configure spec.pd.config.replication.enable-placement-rules: true and configure spec.tiflash in the ${cluster_name}/tidb-cluster.yaml file as follows:

  1.   pd:
  2.     config: |
  3.       ...
  4.       [replication]
  5.       enable-placement-rules = true
  6.   tiflash:
  7.     baseImage: pingcap/tiflash
  8.     maxFailoverCount: 0
  9.     replicas: 1
  10.     storageClaims:
  11.     - resources:
  12.         requests:
  13.           storage: 100Gi
  14.       storageClassName: local-storage

TiFlash supports mounting multiple Persistent Volumes (PVs). If you want to configure multiple PVs for TiFlash, configure multiple resources in tiflash.storageClaims, each resources with a separate storage request and storageClassName. For example:

  1.   tiflash:
  2.     baseImage: pingcap/tiflash
  3.     maxFailoverCount: 0
  4.     replicas: 1
  5.     storageClaims:
  6.     - resources:
  7.         requests:
  8.           storage: 100Gi
  9.       storageClassName: local-storage
  10.     - resources:
  11.         requests:
  12.           storage: 100Gi
  13.       storageClassName: local-storage

TiFlash mounts all PVs to directories such as /data0 and /data1 in the container in the order of configuration. TiFlash has four log files. The proxy log is printed in the standard output of the container. The other three logs are stored in the disk under the /data0 directory by default, which are /data0/logs/flash_cluster_manager.log, / data0/logs/error.log, /data0/logs/server.log. To modify the log storage path, refer to Configure TiFlash parameters.

Configure a TiDB Cluster - 图5Warning

Since TiDB Operator will mount PVs automatically in the order of the items in the storageClaims list, if you need to add more disks to TiFlash, make sure to append the new item only to the end of the original items, and DO NOT modify the order of the original items.

Enable TiCDC

If you want to enable TiCDC in the cluster, you can add TiCDC spec to the TiDBCluster CR. For example:

  1.   spec:
  2.     ticdc:
  3.       baseImage: pingcap/ticdc
  4.       replicas: 3

Deploy Enterprise Edition

To deploy Enterprise Edition of TiDB/PD/TiKV/TiFlash/TiCDC, edit the db.yaml file to set spec.<tidb/pd/tikv/tiflash/ticdc>.baseImage to the enterprise image (pingcap/<tidb/pd/tikv/tiflash/ticdc>-enterprise).

For example:

  1. spec:
  2.   ...
  3.   pd:
  4.     baseImage: pingcap/pd-enterprise
  5.   ...
  6.   tikv:
  7.     baseImage: pingcap/tikv-enterprise

Configure TiDB components

This section introduces how to configure the parameters of TiDB/TiKV/PD/TiFlash/TiCDC.

Configure TiDB parameters

TiDB parameters can be configured by spec.tidb.config in TidbCluster Custom Resource.

For example:

  1. spec:
  2.   tidb:
  3.     config: |
  4.       split-table = true
  5.       oom-action = "log"

For all the configurable parameters of TiDB, refer to TiDB Configuration File.

Configure a TiDB Cluster - 图6Note

If you deploy your TiDB cluster using CR, make sure that Config: {} is set, no matter you want to modify config or not. Otherwise, TiDB components might not be started successfully. This step is meant to be compatible with Helm deployment.

Configure TiKV parameters

TiKV parameters can be configured by spec.tikv.config in TidbCluster Custom Resource.

For example:

  1. spec:
  2.   tikv:
  3.     config: |
  4.       [storage]
  5.         [storage.block-cache]
  6.           capacity = "16GB"

For all the configurable parameters of TiKV, refer to TiKV Configuration File.

Configure a TiDB Cluster - 图7Note

If you deploy your TiDB cluster using CR, make sure that Config: {} is set, no matter you want to modify config or not. Otherwise, TiKV components might not be started successfully. This step is meant to be compatible with Helm deployment.

Configure PD parameters

PD parameters can be configured by spec.pd.config in TidbCluster Custom Resource.

For example:

  1. spec:
  2.   pd:
  3.     config: |
  4.       lease = 3
  5.       enable-prevote = true

For all the configurable parameters of PD, refer to PD Configuration File.

Configure a TiDB Cluster - 图8Note

  • If you deploy your TiDB cluster using CR, make sure that Config: {} is set, no matter you want to modify config or not. Otherwise, PD components might not be started successfully. This step is meant to be compatible with Helm deployment.
  • After the cluster is started for the first time, some PD configuration items are persisted in etcd. The persisted configuration in etcd takes precedence over that in PD. Therefore, after the first start, you cannot modify some PD configuration using parameters. You need to dynamically modify the configuration using SQL statements, pd-ctl, or PD server API. Currently, among all the configuration items listed in Modify PD configuration online, except log.level, all the other configuration items cannot be modified using parameters after the first start.

Configure TiFlash parameters

TiFlash parameters can be configured by spec.tiflash.config in TidbCluster Custom Resource.

For example:

  1. spec:
  2.   tiflash:
  3.     config:
  4.       config: |
  5.         [flash]
  6.           [flash.flash_cluster]
  7.             log = "/data0/logs/flash_cluster_manager.log"
  8.         [logger]
  9.           count = 10
  10.           level = "information"
  11.           errorlog = "/data0/logs/error.log"
  12.           log = "/data0/logs/server.log"

For all the configurable parameters of TiFlash, refer to TiFlash Configuration File.

Configure TiCDC start parameters

You can configure TiCDC start parameters through spec.ticdc.config in TidbCluster Custom Resource.

For example:

For TiDB Operator v1.2.0-rc.2 and later versions, configure the parameters in the TOML format as follows:

  1. spec:
  2.   ticdc:
  3.     config: |
  4.       gc-ttl = 86400
  5.       log-level = "info"

For TiDB Operator versions earlier than v1.2.0-rc.2, configure the parameters in the YAML format as follows:

  1. spec:
  2.   ticdc:
  3.     config:
  4.       timezone: UTC
  5.       gcTTL: 86400
  6.       logLevel: info

For all configurable start parameters of TiCDC, see TiCDC configuration.

Configure automatic failover thresholds of PD, TiDB, TiKV, and TiFlash

The automatic failover feature is enabled by default in TiDB Operator. When the Pods of PD, TiDB, TiKV, TiFlash fail or the corresponding nodes fail, TiDB Operator performs failover automatically and replenish the number of Pod replicas by scaling the corresponding components.

To avoid that the automatic failover feature creates too many Pods, you can configure the threshold of the maximum number of Pods that TiDB Operator can create during failover for each component. The default threshold is 3. If the threshold for a component is configured to 0, it means that the automatic failover feature is disabled for this component. An example configuration is as follows:

  1.   pd:
  2.     maxFailoverCount: 3
  3.   tidb:
  4.     maxFailoverCount: 3
  5.   tikv:
  6.     maxFailoverCount: 3
  7.   tiflash:
  8.     maxFailoverCount: 3

Configure a TiDB Cluster - 图9Note

For the following cases, configure maxFailoverCount: 0 explicitly:

  • The Kubernetes cluster does not have enough resources for TiDB Operator to scale out the new Pod. In such cases, the new Pod will be in the Pending state.
  • You do not want to enable the automatic failover function.

Configure graceful upgrade for TiDB cluster

When you perform a rolling update to the TiDB cluster, Kubernetes sends a TERM signal to the TiDB server before it stops the TiDB Pod. When the TiDB server receives the TERM signal, it tries to wait for all connections to close. After 15 seconds, the TiDB server forcibly closes all the connections and exits the process.

You can enable this feature by configuring the following items:

  • spec.tidb.terminationGracePeriodSeconds: The longest tolerable duration to delete the old TiDB Pod during the rolling upgrade. If this duration is exceeded, the TiDB Pod will be deleted forcibly.
  • spec.tidb.lifecycle: Sets the preStop hook for the TiDB Pod, which is the operation executed before the TiDB server stops.
  1. spec:
  2.   tidb:
  3.     terminationGracePeriodSeconds: 60
  4.     lifecycle:
  5.       preStop:
  6.         exec:
  7.           command:
  8.           - /bin/sh
  9.           - -c
  10.           - "sleep 10 && kill -QUIT 1"

The YAML file above:

  • Sets the longest tolerable duration to delete the TiDB Pod to 60 seconds. If the client does not close the connections after 60 seconds, these connections will be closed forcibly. You can adjust the value according to your needs.
  • Sets the value of preStop hook to sleep 10 && kill -QUIT 1. Here PID 1 refers to the PID of the TiDB server process in the TiDB Pod. When the TiDB server process receives the signal, it exits only after all the connections are closed by the client.

When Kubernetes deletes the TiDB Pod, it also removes the TiDB node from the service endpoints. This is to ensure that the new connection is not established to this TiDB node. However, because this process is asynchronous, you can make the system sleep for a few seconds before you send the kill signal, which makes sure that the TiDB node is removed from the endpoints.

Configure graceful upgrade for TiKV cluster

During TiKV upgrade, TiDB Operator evicts all Region leaders from TiKV Pod before restarting TiKV Pod. Only after the eviction is completed (which means the number of Region leaders on TiKV Pod drops to 0) or the eviction exceeds the specified timeout (1500 minutes by default), TiKV Pod is restarted. If TiKV has fewer than 2 replicas, TiDB Operator forces an upgrade without waiting for the timeout.

If the eviction of Region leaders exceeds the specified timeout, restarting TiKV Pod causes issues such as failures of some requests or more latency. To avoid the issues, you can configure the timeout spec.tikv.evictLeaderTimeout (1500 minutes by default) to a larger value. For example:

  1. spec:
  2.   tikv:
  3.     evictLeaderTimeout: 10000m

Configure a TiDB Cluster - 图10Warning

If the TiKV version is earlier than 4.0.14 or 5.0.3, due to a bug of TiKV, you need to configure the timeout spec.tikv.evictLeaderTimeout as large as possible to ensure that all Region leaders on the TiKV Pod can be evicted within the timeout. If you are not sure about the proper value, greater than ‘1500m’ is recommended.

Configure PV for TiDB slow logs

By default, TiDB Operator creates a slowlog volume (which is an EmptyDir) to store the slow logs, mounts the slowlog volume to /var/log/tidb, and prints slow logs in the stdout through a sidecar container.

Configure a TiDB Cluster - 图11Warning

By default, after a Pod is deleted (for example, rolling update), the slow query logs stored using the EmptyDir volume are lost. Make sure that a log collection solution has been deployed in the Kubernetes cluster to collect logs of all containers. If you do not deploy such a log collection solution, you must make the following configuration to use a persistent volume to store the slow query logs.

If you want to use a separate PV to store the slow logs, you can specify the name of the PV in spec.tidb.slowLogVolumeName, and then configure the PV in spec.tidb.storageVolumes or spec.tidb.additionalVolumes.

This section shows how to configure PV using spec.tidb.storageVolumes or spec.tidb.additionalVolumes.

Configure using spec.tidb.storageVolumes

Configure the TidbCluster CR as the following example. In the example, TiDB Operator uses the ${volumeName} PV to store slow logs. The log file path is ${mountPath}/${volumeName}.

For how to configure the spec.tidb.storageVolumes field, refer to Multiple disks mounting.

**Warning:

You need to configure storageVolumes before creating the cluster. After the cluster is created, adding or removing storageVolumes is no longer supported. For the storageVolumes already configured, except for increasing storageVolume.storageSize, other modifications are not supported. To increase storageVolume.storageSize, you need to make sure that the corresponding StorageClass supports dynamic expansion.

  1.   tidb:
  2.     ...
  3.     separateSlowLog: true  # can be ignored
  4.     slowLogVolumeName: ${volumeName}
  5.     storageVolumes:
  6.       # name must be consistent with slowLogVolumeName
  7.       - name: ${volumeName}
  8.         storageClassName: ${storageClass}
  9.         storageSize: "1Gi"
  10.         mountPath: ${mountPath}

Configure using spec.tidb.additionalVolumes

In the following example, NFS is used as the storage, and TiDB Operator uses the ${volumeName} PV to store slow logs. The log file path is ${mountPath}/${volumeName}.

For the supported PV types, refer to Persistent Volumes.

  1.   tidb:
  2.     ...
  3.     separateSlowLog: true  # can be ignored
  4.     slowLogVolumeName: ${volumeName}
  5.     additionalVolumes:
  6.     # name must be consistent with slowLogVolumeName
  7.     - name: ${volumeName}
  8.       nfs:
  9.         server: 192.168.0.2
  10.         path: /nfs
  11.     additionalVolumeMounts:
  12.     # name must be consistent with slowLogVolumeName
  13.     - name: ${volumeName}
  14.       mountPath: ${mountPath}

Configure TiDB service

You need to configure spec.tidb.service so that TiDB Operator creates a service for TiDB. You can configure Service with different types according to the scenarios, such as ClusterIP, NodePort, LoadBalancer, etc.

ClusterIP

ClusterIP exposes services through the internal IP of the cluster. When selecting this type of service, you can only access it within the cluster using ClusterIP or the Service domain name (${cluster_name}-tidb.${namespace}).

  1. spec:
  2.   ...
  3.   tidb:
  4.     service:
  5.       type: ClusterIP

NodePort

If there is no LoadBalancer, you can choose to expose the service through NodePort. NodePort exposes services through the node’s IP and static port. You can access a NodePort service from outside of the cluster by requesting NodeIP + NodePort.

  1. spec:
  2.   ...
  3.   tidb:
  4.     service:
  5.       type: NodePort
  6.       # externalTrafficPolicy: Local

NodePort has two modes:

  • externalTrafficPolicy=Cluster: All machines in the cluster allocate a NodePort port to TiDB, which is the default value.

    When using the Cluster mode, you can access the TiDB service through the IP and NodePort of any machine. If there is no TiDB Pod on the machine, the corresponding request will be forwarded to the machine with TiDB Pod.

    Configure a TiDB Cluster - 图12Note

    In this mode, the request source IP obtained by the TiDB service is the host IP, not the real client source IP, so access control based on the client source IP is not available in this mode.

  • externalTrafficPolicy=Local: Only the machine that TiDB is running on allocates a NodePort port to access the local TiDB instance.

LoadBalancer

If the TiDB cluster runs in an environment with LoadBalancer, such as on GCP or AWS, it is recommended to use the LoadBalancer feature of these cloud platforms by setting tidb.service.type=LoadBalancer.

  1. spec:
  2.   ...
  3.   tidb:
  4.     service:
  5.       annotations:
  6.         cloud.google.com/load-balancer-type: "Internal"
  7.       externalTrafficPolicy: Local
  8.       type: LoadBalancer

See Kubernetes Service Documentation to know more about the features of Service and what LoadBalancer in the cloud platform supports.

Configure high availability

Configure a TiDB Cluster - 图13Note

TiDB Operator provides a custom scheduler that guarantees TiDB service can tolerate host-level failures through the specified scheduling algorithm. Currently, the TiDB cluster uses this scheduler as the default scheduler, which is configured through the item spec.schedulerName. This section focuses on configuring a TiDB cluster to tolerate failures at other levels such as rack, zone, or region. This section is optional.

TiDB is a distributed database and its high availability must ensure that when any physical topology node fails, not only the service is unaffected, but also the data is complete and available. The two configurations of high availability are described separately as follows.

High availability of TiDB service

Use affinity to schedule pods

By configuring PodAntiAffinity, you can avoid the situation in which different instances of the same component are deployed on the same physical topology node. In this way, disaster recovery (high availability) is achieved. For the user guide of Affinity, see Affinity & AntiAffinity.

The following is an example of a typical service high availability setup:

  1. affinity:
  2.  podAntiAffinity:
  3.    preferredDuringSchedulingIgnoredDuringExecution:
  4.    # this term works when the nodes have the label named region
  5.    - weight: 10
  6.      podAffinityTerm:
  7.        labelSelector:
  8.          matchLabels:
  9.            app.kubernetes.io/instance: ${cluster_name}
  10.            app.kubernetes.io/component: "pd"
  11.        topologyKey: "region"
  12.        namespaces:
  13.        - ${namespace}
  14.    # this term works when the nodes have the label named zone
  15.    - weight: 20
  16.      podAffinityTerm:
  17.        labelSelector:
  18.          matchLabels:
  19.            app.kubernetes.io/instance: ${cluster_name}
  20.            app.kubernetes.io/component: "pd"
  21.        topologyKey: "zone"
  22.        namespaces:
  23.        - ${namespace}
  24.    # this term works when the nodes have the label named rack
  25.    - weight: 40
  26.      podAffinityTerm:
  27.        labelSelector:
  28.          matchLabels:
  29.            app.kubernetes.io/instance: ${cluster_name}
  30.            app.kubernetes.io/component: "pd"
  31.        topologyKey: "rack"
  32.        namespaces:
  33.        - ${namespace}
  34.    # this term works when the nodes have the label named kubernetes.io/hostname
  35.    - weight: 80
  36.      podAffinityTerm:
  37.        labelSelector:
  38.          matchLabels:
  39.            app.kubernetes.io/instance: ${cluster_name}
  40.            app.kubernetes.io/component: "pd"
  41.        topologyKey: "kubernetes.io/hostname"
  42.        namespaces:
  43.        - ${namespace}

Use topologySpreadConstraints to make pods evenly spread

By configuring topologySpreadConstraints, you can make pods evenly spread in different topologies. For instructions about configuring topologySpreadConstraints, see Pod Topology Spread Constraints.

To use topologySpreadConstraints, you must meet the following conditions:

  • Your Kubernetes cluster uses default-scheduler instead of tidb-scheduler. For details, refer to tidb-scheduler and default-scheduler.
  • Your Kubernetes cluster enables the EvenPodsSpread feature gate. If the Kubernetes version in use is earlier than v1.16 or if the EvenPodsSpread feature gate is disabled, the configuration of topologySpreadConstraints does not take effect.

You can either configure topologySpreadConstraints at a cluster level (spec.topologySpreadConstraints) for all components or at a component level (such as spec.tidb.topologySpreadConstraints) for specific components.

The following is an example configuration:

  1. topologySpreadConstraints:
  2. - topologyKey: kubernetes.io/hostname
  3. - topologyKey: topology.kubernetes.io/zone

The example configuration can make pods of the same component evenly spread on different zones and nodes.

Currently, topologySpreadConstraints only supports the configuration of the topologyKey field. In the pod spec, the above example configuration will be automatically expanded as follows:

  1. topologySpreadConstraints:
  2. - topologyKey: kubernetes.io/hostname
  3.   maxSkew: 1
  4.   whenUnsatisfiable: DoNotSchedule
  5.   labelSelector: <object>
  6. - topologyKey: topology.kubernetes.io/zone
  7.   maxSkew: 1
  8.   whenUnsatisfiable: DoNotSchedule
  9.   labelSelector: <object>

High availability of data

Before configuring the high availability of data, read Information Configuration of the Cluster Typology which describes how high availability of TiDB cluster is implemented.

To add the data high availability feature in Kubernetes:

  1. Set the label collection of topological location for PD

    Replace the location-labels information in the pd.config with the label collection that describes the topological location on the nodes in the Kubernetes cluster.

    Configure a TiDB Cluster - 图14Note

    • For PD versions < v3.0.9, the / in the label name is not supported.
    • If you configure host in the location-labels, TiDB Operator will get the value from the kubernetes.io/hostname in the node label.

  2. Set the topological information of the Node where the TiKV node is located.

    TiDB Operator automatically obtains the topological information of the Node for TiKV and calls the PD interface to set this information as the information of TiKV’s store labels. Based on this topological information, the TiDB cluster schedules the replicas of the data.

    If the Node of the current Kubernetes cluster does not have a label indicating the topological location, or if the existing label name of topology contains /, you can manually add a label to the Node by running the following command:

    1. kubectl label node ${node_name} region=${region_name} zone=${zone_name} rack=${rack_name} kubernetes.io/hostname=${host_name}

    In the command above, region, zone, rack, and kubernetes.io/hostname are just examples. The name and number of the label to be added can be arbitrarily defined, as long as it conforms to the specification and is consistent with the labels set by location-labels in pd.config.