使用StatefulSet部署有状态应用

StatefulSet 这个对象是专门用来部署用状态应用的，可以为Pod提供稳定的身份标识，包括hostname、启动顺序、DNS名称等。

下面以在kubernetes1.6版本中部署zookeeper和kafka为例讲解StatefulSet的使用，其中kafka依赖于zookeeper。

Dockerfile和配置文件见 zookeeper 和 kafka。

注：所有的镜像基于CentOS系统的JDK制作，为我的私人镜像，外部无法访问，yaml中没有配置持久化存储。

部署Zookeeper

Dockerfile中从远程获取zookeeper的安装文件，然后在定义了三个脚本：

  • zkGenConfig.sh：生成zookeeper配置文件
  • zkMetrics.sh：获取zookeeper的metrics
  • zkOk.sh：用来做ReadinessProb

我们在来看下这三个脚本的执行结果：

zkGenConfig.sh

zkMetrics.sh脚本实际上执行的是下面的命令：

  1. $ echo mntr | nc localhost $ZK_CLIENT_PORT >& 1
  2. zk_version    3.4.6-1569965, built on 02/20/2014 09:09 GMT
  3. zk_avg_latency    0
  4. zk_max_latency    5
  5. zk_min_latency    0
  6. zk_packets_received    427879
  7. zk_packets_sent    427890
  8. zk_num_alive_connections    3
  9. zk_outstanding_requests    0
  10. zk_server_state    leader
  11. zk_znode_count    18
  12. zk_watch_count    3
  13. zk_ephemerals_count    4
  14. zk_approximate_data_size    613
  15. zk_open_file_descriptor_count    29
  16. zk_max_file_descriptor_count    1048576
  17. zk_followers    1
  18. zk_synced_followers    1
  19. zk_pending_syncs    0

zkOk.sh脚本实际上执行的是下面的命令：

  1. $ echo ruok | nc 127.0.0.1 $ZK_CLIENT_PORT
  2. imok

zookeeper.yaml

下面是启动三个zookeeper实例的yaml配置文件：

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: zk-svc
  6.   labels:
  7.     app: zk-svc
  8. spec:
  9.   ports:
  10.   - port: 2888
  11.     name: server
  12.   - port: 3888
  13.     name: leader-election
  14.   clusterIP: None
  15.   selector:
  16.     app: zk
  17. ---
  18. apiVersion: v1
  19. kind: ConfigMap
  20. metadata:
  21.   name: zk-cm
  22. data:
  23.   jvm.heap: "1G"
  24.   tick: "2000"
  25.   init: "10"
  26.   sync: "5"
  27.   client.cnxns: "60"
  28.   snap.retain: "3"
  29.   purge.interval: "0"
  30. ---
  31. apiVersion: policy/v1beta1
  32. kind: PodDisruptionBudget
  33. metadata:
  34.   name: zk-pdb
  35. spec:
  36.   selector:
  37.     matchLabels:
  38.       app: zk
  39.   minAvailable: 2
  40. ---
  41. apiVersion: apps/v1beta1
  42. kind: StatefulSet
  43. metadata:
  44.   name: zk
  45. spec:
  46.   serviceName: zk-svc
  47.   replicas: 3
  48.   template:
  49.     metadata:
  50.       labels:
  51.         app: zk
  52.     spec:
  53.       affinity:
  54.         podAntiAffinity:
  55.           requiredDuringSchedulingIgnoredDuringExecution:
  56.             - labelSelector:
  57.                 matchExpressions:
  58.                   - key: "app"
  59.                     operator: In
  60.                     values:
  61.                     - zk
  62.               topologyKey: "kubernetes.io/hostname"
  63.       containers:
  64.       - name: k8szk
  65.         imagePullPolicy: Always
  66.         image: sz-pg-oam-docker-hub-001.tendcloud.com/library/zookeeper:3.4.6
  67.         resources:
  68.           requests:
  69.             memory: "2Gi"
  70.             cpu: "500m"
  71.         ports:
  72.         - containerPort: 2181
  73.           name: client
  74.         - containerPort: 2888
  75.           name: server
  76.         - containerPort: 3888
  77.           name: leader-election
  78.         env:
  79.         - name : ZK_REPLICAS
  80.           value: "3"
  81.         - name : ZK_HEAP_SIZE
  82.           valueFrom:
  83.             configMapKeyRef:
  84.                 name: zk-cm
  85.                 key: jvm.heap
  86.         - name : ZK_TICK_TIME
  87.           valueFrom:
  88.             configMapKeyRef:
  89.                 name: zk-cm
  90.                 key: tick
  91.         - name : ZK_INIT_LIMIT
  92.           valueFrom:
  93.             configMapKeyRef:
  94.                 name: zk-cm
  95.                 key: init
  96.         - name : ZK_SYNC_LIMIT
  97.           valueFrom:
  98.             configMapKeyRef:
  99.                 name: zk-cm
  100.                 key: tick
  101.         - name : ZK_MAX_CLIENT_CNXNS
  102.           valueFrom:
  103.             configMapKeyRef:
  104.                 name: zk-cm
  105.                 key: client.cnxns
  106.         - name: ZK_SNAP_RETAIN_COUNT
  107.           valueFrom:
  108.             configMapKeyRef:
  109.                 name: zk-cm
  110.                 key: snap.retain
  111.         - name: ZK_PURGE_INTERVAL
  112.           valueFrom:
  113.             configMapKeyRef:
  114.                 name: zk-cm
  115.                 key: purge.interval
  116.         - name: ZK_CLIENT_PORT
  117.           value: "2181"
  118.         - name: ZK_SERVER_PORT
  119.           value: "2888"
  120.         - name: ZK_ELECTION_PORT
  121.           value: "3888"
  122.         command:
  123.         - sh
  124.         - -c
  125.         - zkGenConfig.sh && zkServer.sh start-foreground
  126.         readinessProbe:
  127.           exec:
  128.             command:
  129.             - "zkOk.sh"
  130.           initialDelaySeconds: 10
  131.           timeoutSeconds: 5
  132.         livenessProbe:
  133.           exec:
  134.             command:
  135.             - "zkOk.sh"
  136.           initialDelaySeconds: 10
  137.           timeoutSeconds: 5
  138.       securityContext:
  139.         runAsUser: 1000
  140.         fsGroup: 1000

我们再主要下上面那三个脚本的用途。

部署kafka

Kafka的docker镜像制作跟zookeeper类似，都是从远程下载安装包后，解压安装。

与zookeeper不同的是，只要一个脚本，但是又依赖于我们上一步安装的zookeeper，kafkaGenConfig.sh用来生成kafka的配置文件。

我们来看下这个脚本。

  1. #!/bin/bash
  2. HOST=`hostname -s`
  3. if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
  4.         NAME=${BASH_REMATCH[1]}
  5.         ORD=${BASH_REMATCH[2]}
  6. else
  7.         echo "Failed to extract ordinal from hostname $HOST"
  8.         exit 1
  9. fi
  11. MY_ID=$((ORD+1))
  12. sed -i s"/broker.id=0/broker.id=$MY_ID/g" /opt/kafka/config/server.properties
  13. sed -i s'/zookeeper.connect=localhost:2181/zookeeper.connect=zk-0.zk-svc.brand.svc:2181,zk-1.zk-svc.brand.svc:2181,zk-2.zk-svc.brand.svc:2181/g' /opt/kafka/config/server.properties

该脚本根据statefulset生成的pod的hostname的后半截数字部分作为broker ID，同时再替换zookeeper的地址。

Kafka.yaml

下面是创建3个kafka实例的yaml配置。

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: kafka-svc
  6.   labels:
  7.     app: kafka
  8. spec:
  9.   ports:
  10.   - port: 9093
  11.     name: server
  12.   clusterIP: None
  13.   selector:
  14.     app: kafka
  15. ---
  16. apiVersion: policy/v1beta1
  17. kind: PodDisruptionBudget
  18. metadata:
  19.   name: kafka-pdb
  20. spec:
  21.   selector:
  22.     matchLabels:
  23.       app: kafka
  24.   minAvailable: 2
  25. ---
  26. apiVersion: apps/v1beta1
  27. kind: StatefulSet
  28. metadata:
  29.   name: kafka
  30. spec:
  31.   serviceName: kafka-svc
  32.   replicas: 3
  33.   template:
  34.     metadata:
  35.       labels:
  36.         app: kafka
  37.     spec:
  38.       affinity:
  39.         podAntiAffinity:
  40.           requiredDuringSchedulingIgnoredDuringExecution:
  41.             - labelSelector:
  42.                 matchExpressions:
  43.                   - key: "app"
  44.                     operator: In
  45.                     values:
  46.                     - kafka
  47.               topologyKey: "kubernetes.io/hostname"
  48.         podAffinity:
  49.           preferredDuringSchedulingIgnoredDuringExecution:
  50.              - weight: 1
  51.                podAffinityTerm:
  52.                  labelSelector:
  53.                     matchExpressions:
  54.                       - key: "app"
  55.                         operator: In
  56.                         values:
  57.                         - zk
  58.                  topologyKey: "kubernetes.io/hostname"
  59.       terminationGracePeriodSeconds: 300
  60.       containers:
  61.       - name: k8skafka
  62.         imagePullPolicy: Always
  63.         image: sz-pg-oam-docker-hub-001.tendcloud.com/library/kafka:2.10-0.8.2.1
  64.         resources:
  65.           requests:
  66.             memory: "1Gi"
  67.             cpu: 500m
  68.         env:
  69.         - name: KF_REPLICAS
  70.           value: "3"
  71.         ports:
  72.         - containerPort: 9093
  73.           name: server
  74.         command:
  75.         - /bin/bash
  76.         - -c
  77.         - "/opt/kafka/bin/kafkaGenConfig.sh && /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties"
  78.         env:
  79.         - name: KAFKA_HEAP_OPTS
  80.           value : "-Xmx512M -Xms512M"
  81.         - name: KAFKA_OPTS
  82.           value: "-Dlogging.level=DEBUG"
  83.         readinessProbe:
  84.            tcpSocket:
  85.              port: 9092
  86.            initialDelaySeconds: 15
  87.            timeoutSeconds: 1

参考

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

kubernetes contrib - statefulsets

http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html