使用 Helm 部署 TDengine 集群

Helm 是 Kubernetes 的包管理器，上一节使用 Kubernetes 部署 TDengine 集群的操作已经足够简单，但 Helm 依然可以提供更强大的能力。

安装 Helm

  1. curl -fsSL -o get_helm.sh \
  2.   https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
  3. chmod +x get_helm.sh
  4. ./get_helm.sh

Helm 会使用 kubectl 和 kubeconfig 的配置来操作 Kubernetes，可以参考 Rancher 安装 Kubernetes 的配置来进行设置。

安装 TDengine Chart

TDengine Chart 尚未发布到 Helm 仓库，当前可以从 GitHub 直接下载：

  1. wget https://github.com/taosdata/TDengine-Operator/raw/3.0/helm/tdengine-3.0.2.tgz

获取当前 Kubernetes 的存储类：

  1. kubectl get storageclass

在 minikube 默认为 standard.

之后，使用 helm 命令安装：

  1. helm install tdengine tdengine-3.0.2.tgz \
  2.   --set storage.className=<your storage class name>

在 minikube 环境下，可以设置一个较小的容量避免超出磁盘可用空间：

  1. helm install tdengine tdengine-3.0.2.tgz \
  2.   --set storage.className=standard \
  3.   --set storage.dataSize=2Gi \
  4.   --set storage.logSize=10Mi

部署成功后，TDengine Chart 将会输出操作 TDengine 的说明：

  1. export POD_NAME=$(kubectl get pods --namespace default \
  2.   -l "app.kubernetes.io/name=tdengine,app.kubernetes.io/instance=tdengine" \
  3.   -o jsonpath="{.items[0].metadata.name}")
  4. kubectl --namespace default exec $POD_NAME -- taos -s "show dnodes; show mnodes"
  5. kubectl --namespace default exec -it $POD_NAME -- taos

可以创建一个表进行测试：

  1. kubectl --namespace default exec $POD_NAME -- \
  2.   taos -s "create database test;
  3.     use test;
  4.     create table t1 (ts timestamp, n int);
  5.     insert into t1 values(now, 1)(now + 1s, 2);
  6.     select * from t1;"

配置 Values

TDengine 支持 values.yaml 自定义。

通过 helm show values 可以获取 TDengine Chart 支持的全部 values 列表：

  1. helm show values tdengine-3.0.2.tgz

你可以将结果保存为 values.yaml，之后可以修改其中的各项参数，如 replica 数量，存储类名称，容量大小，TDengine 配置等，然后使用如下命令安装 TDengine 集群：

  1. helm install tdengine tdengine-3.0.2.tgz -f values.yaml

全部参数如下：

  1. # Default values for tdengine.
  2. # This is a YAML-formatted file.
  3. # Declare variables to be passed into helm templates.
  5. replicaCount: 1
  7. image:
  8.   prefix: tdengine/tdengine
  9.   #pullPolicy: Always
  10.   # Overrides the image tag whose default is the chart appVersion.
  11. #  tag: "3.0.2.0"
  13. service:
  14.   # ClusterIP is the default service type, use NodeIP only if you know what you are doing.
  15.   type: ClusterIP
  16.   ports:
  17.     # TCP range required
  18.     tcp: [6030, 6041, 6042, 6043, 6044, 6046, 6047, 6048, 6049, 6060]
  19.     # UDP range
  20.     udp: [6044, 6045]
  23. # Set timezone here, not in taoscfg
  24. timezone: "Asia/Shanghai"
  26. resources:
  27.   # We usually recommend not to specify default resources and to leave this as a conscious
  28.   # choice for the user. This also increases chances charts run on environments with little
  29.   # resources, such as Minikube. If you do want to specify resources, uncomment the following
  30.   # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  31.   # limits:
  32.   #   cpu: 100m
  33.   #   memory: 128Mi
  34.   # requests:
  35.   #   cpu: 100m
  36.   #   memory: 128Mi
  38. storage:
  39.   # Set storageClassName for pvc. K8s use default storage class if not set.
  40.   #
  41.   className: ""
  42.   dataSize: "100Gi"
  43.   logSize: "10Gi"
  45. nodeSelectors:
  46.   taosd:
  47.     # node selectors
  49. clusterDomainSuffix: ""
  50. # Config settings in taos.cfg file.
  51. #
  52. # The helm/k8s support will use environment variables for taos.cfg,
  53. # converting an upper-snake-cased variable like `TAOS_DEBUG_FLAG`,
  54. # to a camelCase taos config variable `debugFlag`.
  55. #
  56. # See the variable list at https://www.taosdata.com/cn/documentation/administrator .
  57. #
  58. # Note:
  59. # 1. firstEp/secondEp: should not be set here, it's auto generated at scale-up.
  60. # 2. serverPort: should not be set, we'll use the default 6030 in many places.
  61. # 3. fqdn: will be auto generated in kubernetes, user should not care about it.
  62. # 4. role: currently role is not supported - every node is able to be mnode and vnode.
  63. #
  64. # Btw, keep quotes "" around the value like below, even the value will be number or not.
  65. taoscfg:
  66.   # Starts as cluster or not, must be 0 or 1.
  67.   #   0: all pods will start as a separate TDengine server
  68.   #   1: pods will start as TDengine server cluster. [default]
  69.   CLUSTER: "1"
  71.   # number of replications, for cluster only
  72.   TAOS_REPLICA: "1"
  75.   # TAOS_NUM_OF_RPC_THREADS: number of threads for RPC
  76.   #TAOS_NUM_OF_RPC_THREADS: "2"
  78.   #
  79.   # TAOS_NUM_OF_COMMIT_THREADS: number of threads to commit cache data
  80.   #TAOS_NUM_OF_COMMIT_THREADS: "4"
  82.   # enable/disable installation / usage report
  83.   #TAOS_TELEMETRY_REPORTING: "1"
  85.   # time interval of system monitor, seconds
  86.   #TAOS_MONITOR_INTERVAL: "30"
  88.   # time interval of dnode status reporting to mnode, seconds, for cluster only
  89.   #TAOS_STATUS_INTERVAL: "1"
  91.   # time interval of heart beat from shell to dnode, seconds
  92.   #TAOS_SHELL_ACTIVITY_TIMER: "3"
  94.   # minimum sliding window time, milli-second
  95.   #TAOS_MIN_SLIDING_TIME: "10"
  97.   # minimum time window, milli-second
  98.   #TAOS_MIN_INTERVAL_TIME: "1"
  100.   # the compressed rpc message, option:
  101.   #  -1 (no compression)
  102.   #   0 (all message compressed),
  103.   # > 0 (rpc message body which larger than this value will be compressed)
  104.   #TAOS_COMPRESS_MSG_SIZE: "-1"
  106.   # max number of connections allowed in dnode
  107.   #TAOS_MAX_SHELL_CONNS: "50000"
  109.   # stop writing logs when the disk size of the log folder is less than this value
  110.   #TAOS_MINIMAL_LOG_DIR_G_B: "0.1"
  112.   # stop writing temporary files when the disk size of the tmp folder is less than this value
  113.   #TAOS_MINIMAL_TMP_DIR_G_B: "0.1"
  115.   # if disk free space is less than this value, taosd service exit directly within startup process
  116.   #TAOS_MINIMAL_DATA_DIR_G_B: "0.1"
  118.   # One mnode is equal to the number of vnode consumed
  119.   #TAOS_MNODE_EQUAL_VNODE_NUM: "4"
  121.   # enbale/disable http service
  122.   #TAOS_HTTP: "1"
  124.   # enable/disable system monitor
  125.   #TAOS_MONITOR: "1"
  127.   # enable/disable async log
  128.   #TAOS_ASYNC_LOG: "1"
  130.   #
  131.   # time of keeping log files, days
  132.   #TAOS_LOG_KEEP_DAYS: "0"
  134.   # The following parameters are used for debug purpose only.
  135.   # debugFlag 8 bits mask: FILE-SCREEN-UNUSED-HeartBeat-DUMP-TRACE_WARN-ERROR
  136.   # 131: output warning and error
  137.   # 135: output debug, warning and error
  138.   # 143: output trace, debug, warning and error to log
  139.   # 199: output debug, warning and error to both screen and file
  140.   # 207: output trace, debug, warning and error to both screen and file
  141.   #
  142.   # debug flag for all log type, take effect when non-zero value\
  143.   #TAOS_DEBUG_FLAG: "143"
  145.   # generate core file when service crash
  146.   #TAOS_ENABLE_CORE_FILE: "1"

扩容

关于扩容可参考上一节的说明，有一些额外的操作需要从 helm 的部署中获取。

首先，从部署中获取 StatefulSet 的名称。

  1. export STS_NAME=$(kubectl get statefulset \
  2.   -l "app.kubernetes.io/name=tdengine" \
  3.   -o jsonpath="{.items[0].metadata.name}")

扩容操作极其简单，增加 replica 即可。以下命令将 TDengine 扩充到三节点：

  1. kubectl scale --replicas 3 statefulset/$STS_NAME

使用命令 show dnodes 和 show mnodes 检查是否扩容成功。

缩容

Helm - 图1warning

缩容操作并没有完整测试，可能造成数据风险，请谨慎使用。

获取需要缩容的 dnode 列表，并手动 Drop。

  1. kubectl --namespace default exec $POD_NAME -- \
  2.   cat /var/lib/taos/dnode/dnodeEps.json \
  3.   | jq '.dnodeInfos[1:] |map(.dnodeFqdn + ":" + (.dnodePort|tostring)) | .[]' -r
  4. kubectl --namespace default exec $POD_NAME -- taos -s "show dnodes"
  5. kubectl --namespace default exec $POD_NAME -- taos -s 'drop dnode "<you dnode in list>"'

清理集群

Helm 管理下，清理操作也变得简单：

  1. helm uninstall tdengine

但 Helm 也不会自动移除 PVC，需要手动获取 PVC 然后删除掉。