Maintain Kubernetes Nodes that Hold the TiDB Cluster

Maintain Kubernetes Nodes that Hold the TiDB Cluster

TiDB is a highly available database that can run smoothly when some of the database nodes go offline. For this reason, you can safely shut down and maintain the Kubernetes nodes at the bottom layer without influencing TiDB’s service. Specifically, you need to adopt various maintenance strategies when handling PD, TiKV, and TiDB Pods because of their different characteristics.

This document introduces how to perform a temporary or long-term maintenance task for the Kubernetes nodes.

Prerequisites

Note

Before you maintain a node, you need to make sure that the remaining resources in the Kubernetes cluster are enough for running the TiDB cluster.

Maintain a node that can be recovered shortly

Mark the node to be maintained as non-schedulable to ensure that no new Pod is scheduled to it:
```
kubectl cordon ${node_name}
```
Check whether there is any TiKV Pod on the node to be maintained:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep tikv
```
If any TiKV Pod is found, for each TiKV Pod, perform the following operations:
1. Evict the TiKV Region Leader to another Pod.
2. Increase the maximum offline duration for TiKV Pods by configuring max-store-down-time of PD. After you maintain and recover the Kubernetes node within that duration, all TiKV Pods on that node will be automatically recovered.
  
  The following example shows how to set max-store-down-time to 60m. You can set it to any reasonable value.
```
pd-ctl config set max-store-down-time 60m
```
Check whether there is any PD Pod on the node to be maintained:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep pd
```
If any PD Pod is found, for each PD Pod, transfer the PD leader to other Pods.
Confirm that the node to be maintained no longer has any TiKV Pod or PD Pod:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name}
```
Migrate all Pods on the node to be maintained to other nodes:
```
kubectl drain ${node_name} --ignore-daemonsets
```
After running this command, all Pods on this node are automatically migrated to another available node.
Confirm that the node to be maintained no longer has any TiKV, TiDB, or PD Pod:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name}
```
When the maintenance is completed, after you recover the node, make sure that the node is in a healthy state:
```
watch kubectl get node ${node_name}
```
After the node goes into the Ready state, proceed with the following operations.
Lift the scheduling restriction on the node:
```
kubectl uncordon ${node_name}
```
Confirm that all Pods are running normally:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name}
```
When all Pods are running normally, you have successfully finished the maintenance task.

Maintain a node that cannot be recovered shortly

Check whether there is any TiKV Pod on the node to be maintained:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep tikv
```
If any TiKV Pod is found, for each TiKV Pod, reschedule the TiKV Pod to another node.
Check whether there is any PD Pod on the node to be maintained:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep pd
```
If any PD Pod is found, for each PD Pod, reschedule the PD Pod to another node.
Confirm that the node to be maintained no longer has any TiKV Pod or PD Pod:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name}
```
Migrate all Pods on the node to be maintained to other nodes:
```
kubectl drain ${node_name} --ignore-daemonsets
```
After running this command, all Pods on this node are automatically migrated to another available node.
Confirm that the node to be maintained no longer has any TiKV, TiDB, or PD Pod:
```
kubectl get pod --all-namespaces -o wide | grep ${node_name}
```
(Optional) If the node will be offline for a long time, it is recommended to delete the node from your Kubernetes cluster:
```
kubectl delete node ${node_name}
```

Reschedule PD Pods

If a node will be offline for a long time, to minimize the impact on your application, you can reschedule the PD Pods on this node to other nodes in advance.

If the node storage can be automatically migrated

If the node storage can be automatically migrated (such as EBS), to reschedule a PD Pod, you do not need to delete the PD member. You only need to transfer the PD Leader to another Pod and delete the old Pod.

Mark the node to be maintained as non-schedulable to ensure that no new Pod is scheduled to it:
```
kubectl cordon ${node_name}
```

Check the PD Pod on the node to be maintained:

kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep pd

Transfer the PD Leader to another Pod.

Delete the old PD Pod:

kubectl delete -n ${namespace} pod ${pod_name}

Confirm that the PD Pod is successfully scheduled to another node:
```
watch kubectl -n ${namespace} get pod -o wide
```

If the node storage cannot be automatically migrated

If the node storage cannot be automatically migrated (such as local storage), to reschedule a PD Pod, you need to delete the PD member.

Mark the node to be maintained as non-schedulable to ensure that no new Pod is scheduled to it:
```
kubectl cordon ${node_name}
```

Check the PD Pod on the node to be maintained:

kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep pd

Transfer the PD Leader to another Pod.
Take the PD Pod offline:
```
pd-ctl member delete name ${pod_name}
```
Confirm that the PD member is deleted:
```
pd-ctl member
```

Unbind the PD Pod with the local disk on the node.

Check the PersistentVolumeClaim used by the Pod:

kubectl -n ${namespace} get pvc -l tidb.pingcap.com/pod-name=${pod_name}

Delete the PersistentVolumeClaim:

kubectl delete -n ${namespace} pvc ${pvc_name} --wait=false

Delete the old PD Pod:

kubectl delete -n ${namespace} pod ${pod_name}

Confirm that the PD Pod is successfully scheduled to another node:
```
watch kubectl -n ${namespace} get pod -o wide
```

Reschedule TiKV Pods

If a node will be offline for a long time, to minimize the impact on your application, you can reschedule the TiKV Pods on this node to other nodes in advance.

If the node storage can be automatically migrated

If the node storage can be automatically migrated (such as EBS), to reschedule a TiKV Pod, you do not need to delete the whole TiKV store. You only need to evict the TiKV Region Leader to another Pod and delete the old Pod.

Mark the node to be maintained as non-schedulable to ensure that no new Pod is scheduled to it:
```
kubectl cordon ${node_name}
```

Check the TiKV Pod on the node to be maintained:

kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep tikv

Add annotation with a tidb.pingcap.com/evict-leader key to the TiKV Pod to trigger the graceful restart. After TiDB Operator evicts the TiKV Region Leader, TiDB Operator deletes the Pod.
```
kubectl -n ${namespace} annotate pod ${pod_name} tidb.pingcap.com/evict-leader="delete-pod"
```
Confirm that the TiKV Pod is successfully scheduled to another node:
```
watch kubectl -n ${namespace} get pod -o wide
```

Confirm that the Region Leader is transferred back:

kubectl -n ${namespace} get tc ${cluster_name} -ojson | jq ".status.tikv.stores | .[] | select ( .podName == \"${pod_name}\" ) | .leaderCount"

If the node storage cannot be automatically migrated

If the node storage cannot be automatically migrated (such as local storage), to reschedule a TiKV Pod, you need to delete the whole TiKV store.

Mark the node to be maintained as non-schedulable to ensure that no new Pod is scheduled to it:
```
kubectl cordon ${node_name}
```

Check the TiKV Pod on the node to be maintained:

kubectl get pod --all-namespaces -o wide | grep ${node_name} | grep tikv

Recreate a TiKV Pod.

Transfer PD Leader

Check the PD Leader:
```
pd-ctl member leader show
```
If the Leader Pod is on the node to be maintained, you need to transfer the PD Leader to a Pod on another node:
```
pd-ctl member leader transfer ${pod_name}
```
${pod_name} is the name of the PD Pod on another node.

Evict TiKV Region Leader

Add an annotation with a tidb.pingcap.com/evict-leader key to the TiKV Pod:

kubectl -n ${namespace} annotate pod ${pod_name} tidb.pingcap.com/evict-leader="none"

Confirm that all Region Leaders are migrated:

kubectl -n ${namespace} get tc ${cluster_name} -ojson | jq ".status.tikv.stores | .[] | select ( .podName == \"${pod_name}\" ) | .leaderCount"

If the output is 0, all Region Leaders are successfully migrated.

Recreate a TiKV Pod

Evict the TiKV Region Leader to another Pod.
Take the TiKV Pod offline.

Note

Before you take the TiKV Pod offline, make sure that the remaining TiKV Pods are not fewer than the TiKV replica number set in PD configuration (max-replicas, 3 by default). If the remaining TiKV Pods are not enough, scale out TiKV Pods before you take the TiKV Pod offline.
1. Check store-id of the TiKV Pod:
```
kubectl get -n ${namespace} tc ${cluster_name} -ojson | jq ".status.tikv.stores | .[] | select ( .podName == \"${pod_name}\" ) | .id"
```
2. In any of the PD Pods, use pd-ctl command to take the TiKV Pod offline:
```
kubectl exec -n ${namespace} ${cluster_name}-pd-0 -- /pd-ctl store delete ${store_id}
```
3. Wait for the store status (state_name) to become Tombstone:
```
kubectl exec -n ${namespace} ${cluster_name}-pd-0 -- watch /pd-ctl store ${store_id}
```
  Expected output
```
{
  "store": {
    "id": "${store_id}",
    // ...
    "state_name": "Tombstone"
  },
  // ...
}
```

Unbind the TiKV Pod with the currently used storage.

Check the PersistentVolumeClaim used by the Pod:

kubectl -n ${namespace} get pvc -l tidb.pingcap.com/pod-name=${pod_name}

Expected output

The NAME field is the name of PVC.

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES       STORAGECLASS   AGE
${pvc_name}   Bound    pvc-a8f16ca6-a675-448f-82c3-3cae624aa0e2   100Gi      RWO                standard       18m

Delete the PersistentVolumeClaim:

kubectl delete -n ${namespace} pvc ${pvc_name} --wait=false

Delete the old TiKV Pod and wait for the new TiKV Pod to join the cluster.

kubectl delete -n ${namespace} pod ${pod_name}

Wait for the state of the new TiKV Pod to become Up.

kubectl get -n ${namespace} tc ${cluster_name} -ojson | jq ".status.tikv.stores | .[] | select ( .podName == \"${pod_name}\" )"

Expected output

{
  "id": "${new_store_id}",
  "ip": "${pod_name}.${cluster_name}-tikv-peer.default.svc",
  "lastTransitionTime": "2022-03-08T06:39:58Z",
  "leaderCount": 3,
  "podName": "${pod_name}",
  "state": "Up"
}

As you can see from the output, the new TiKV Pod have a new store-id, and Region Leaders are migrated to this TiKV Pod automatically.

Remove the useless evict-leader-scheduler:

kubectl exec -n ${namespace} ${cluster_name}-pd-0 -- /pd-ctl scheduler remove evict-leader-scheduler-${store_id}