07-5.集成 Ceph 持久化存储

Ceph 集群情况

机器列表：

  1. 172.27.128.100  mon.m7-common-dfs04
  2. 172.27.128.101  mon.m7-common-dfs03
  3. 172.27.128.102  mon.m7-common-dfs02
  4. 172.27.128.103  mon.m7-common-dfs01

组件部署情况：

  1. Deploy：m7-common-dfs04
  2. Mon：m7-common-dfs04 m7-common-dfs02 m7-common-dfs03
  3. Mgr：m7-common-dfs04 m7-common-dfs02 m7-common-dfs03
  4. Mds：m7-common-dfs04 m7-common-dfs02 m7-common-dfs03
  5. OSD：
  6.     0-2：mon.m7-common-dfs04
  7.     3-5：mon.m7-common-dfs03
  8.     6-8：mon.m7-common-dfs02
  9.     9-11：mon.m7-common-dfs01
  10. RGW：mon.m7-common-dfs04

安装 Ceph 客户端工具

需要在使用 Ceph 的每个 K8S 节点上安装 Ceph 客户端工具。

创建 yum 源配置文件：

  1. sudo yum install -y epel-release
  2. cat << "EOM" > /etc/yum.repos.d/ceph.repo
  3. [ceph]
  4. name=Ceph packages for $basearch
  5. baseurl=http://download.ceph.com/rpm-luminous/el7/$basearch
  6. enabled=1
  7. gpgcheck=1
  8. type=rpm-md
  9. gpgkey=https://download.ceph.com/keys/release.asc
  10. priority=1
  12. [ceph-noarch]
  13. name=Ceph noarch packages
  14. baseurl=https://download.ceph.com/rpm-luminous/el7/noarch
  15. enabled=1
  16. gpgcheck=1
  17. type=rpm-md
  18. gpgkey=https://download.ceph.com/keys/release.asc
  19. priority=1
  21. [ceph-source]
  22. name=Ceph source packages
  23. baseurl=http://download.ceph.com/rpm-luminous/el7/SRPMS
  24. enabled=1
  25. gpgcheck=1
  26. type=rpm-md
  27. gpgkey=https://download.ceph.com/keys/release.asc
  28. priority=1
  29. EOM
  • 注意：ceph repo 的版本需要与 ceph 集群版本一致，如上面配置的是 luminous 版本源。

安装 Ceph 客户端工具：

  1. yum clean all && yum update
  2. yum install -y ceph-common

安装的命令行工具列表：

  1. $ rpm -ql ceph-common|grep bin
  2. /usr/bin/ceph
  3. /usr/bin/ceph-authtool
  4. /usr/bin/ceph-brag
  5. /usr/bin/ceph-conf
  6. /usr/bin/ceph-dencoder
  7. /usr/bin/ceph-post-file
  8. /usr/bin/ceph-rbdnamer
  9. /usr/bin/ceph-syn
  10. /usr/bin/rados
  11. /usr/bin/rbd

挂载 CephFS

创建本地挂载目录，如 devops：

  1. sudo mkdir -p /etc/ceph /mnt/cephfs/k8s/devops

创建 secret 文件：

  1. sudo scp root@172.27.128.100:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
  2. sudo awk '/key = / {print $3}' /etc/ceph/ceph.client.admin.keyring >/etc/ceph/ceph-admin.secret

挂载 CephFS 根目录 /k8s/：

  1. sudo mount -t ceph 172.27.128.100:6789,172.27.128.101:6789,172.27.128.102:6789:/k8s/  /mnt/cephfs/k8s/devops -o name=admin,secretfile=/etc/ceph/ceph-admin.secret,_netdev,noatime
  • 需要指定多个 MDS 地址，以达到高可用和容错的目的。

在 CephFS 根目录 /k8s/ 中创建集群专用的数据目录，如 devops:

  1. mkdir /k8s/devops

将创建的集群专用 CephFS 目录（/k8s/devops）挂载到本地目录（/mnt/cephfs/k8s/devops）：

  1. sudo umount /mnt/cephfs/k8s/devops
  2. sudo mount -t ceph 172.27.128.100:6789,172.27.128.101:6789,172.27.128.102:6789:/k8s/devops  /mnt/cephfs/k8s/devops -o name=admin,secretfile=/etc/ceph/ceph-admin.secret,_netdev,noatime

在 /etc/fstab 中添加一行开启自动挂载记录：

  1. 172.27.128.100:6789:/k8s/devops/ /mnt/cephfs/k8s/devops ceph name=admin,secretfile=/etc/ceph/ceph-admin.secret,_netdev,noatime 0 0
  • 必须添加 _netdev 挂载参数，否则机器启动时会卡在挂载 cephfs 阶段。

创建 ceph admin secret

  1. [root@m7-demo-136001 k8s]# cd /opt/k8s
  2. [root@m7-demo-136001 k8s]# scp root@172.27.128.100:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
  4. [root@m7-demo-136001 k8s]# cat /etc/ceph/ceph.client.admin.keyring
  5. [client.admin]
  6.         key = AQCYLTdbCyxZBhAAbGfK3T2tczjbhhbR0UWq1w==
  8. [root@m7-demo-136001 k8s]# Secret=$(awk '/key = / {print $3}' /etc/ceph/ceph.client.admin.keyring | base64)
  9. [root@m7-demo-136001 k8s]# cat > ceph-secret-admin.yaml <<EOF
  10. apiVersion: v1
  11. kind: Secret
  12. type: kubernetes.io/rbd
  13. metadata:
  14.   name: ceph-secret-admin
  15. data:
  16.   key: $Secret
  17. EOF

注意，如果是通过命令行创建 secret-admin，则不需要对key进行base64编码：

$ kubectl create secret generic ceph-secret-admin —from-literal=key=’AQCYLTdbCyxZBhAAbGfK3T2tczjbhhbR0UWq1w==’ —namespace=kube-system —type=kubernetes.io/rbd

  1. [root@m7-demo-136001 k8s]#  kubectl create -f ceph-secret-admin.yaml
  2. secret "ceph-secret-admin" created

注意：PVC 只能使用所在命名空间的 rbd secret。所以上面的定义的 ceph-secret-admin 只能供 default 命名空间的 PVC 使用。其它命名空间的 PVC 如果需要用该 StorageClass，则需要在所在命名空间重新定义该 Secret；

创建 StorageClass

  1. [root@m7-demo-136001 k8s]# cat >ceph-rbd-storage-class.yaml <<EOF
  2. apiVersion: storage.k8s.io/v1
  3. kind: StorageClass
  4. metadata:
  5.    name: ceph
  6. provisioner: kubernetes.io/rbd
  7. parameters:
  8.     monitors: 172.27.128.100:6789,172.27.128.101:6789,172.27.128.102:6789
  9.     adminId: admin
  10.     adminSecretName: ceph-secret-admin
  11.     adminSecretNamespace: "default"
  12.     pool: rbd
  13.     userId: admin
  14.     userSecretName: ceph-secret-admin
  15. EOF
  • 额外的缺省参数： imageFormat、imageFeatures；
  • imageFormat 默认值为 1。如果指定为 2，则需要 v3.11 以上内核才行，支持 Clone 等高级特性；
  • imageFeatures 默认值为空，即不启用任何特性。目前支持 layering 特性；
  1. [root@m7-demo-136001 k8s]# kubectl create -f ceph-rbd-storage-class.yaml
  2. storageclass "ceph" created

查看创建的 StorageClass：

  1. $ kubectl get storageclass
  2. NAME      PROVISIONER
  3. ceph      kubernetes.io/rbd

通过将 StorageClass 对象的 storageclass.kubernetes.io/is-default-class annotations 设置为 true，可将该 StorageClass 设置为 Default StorageClass：

  1. $ kubectl patch storageclass ceph -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
  2. storageclass "ceph" patched
  3. $ kubectl get storageclass
  4. NAME             PROVISIONER
  5. ceph (default)   kubernetes.io/rbd

创建使用 StorageClass 的 PVC

  1. [root@m7-demo-136001 k8s]# cat >ceph-pvc-storageClass.json <<EOF
  2. {
  3.   "kind": "PersistentVolumeClaim",
  4.   "apiVersion": "v1",
  5.   "metadata": {
  6.     "name": "pvc-test-claim"
  7.   },
  8.   "spec": {
  9.     "accessModes": [
  10.       "ReadWriteOnce"
  11.     ],
  12.     "resources": {
  13.       "requests": {
  14.         "storage": "1Gi"
  15.       }
  16.     },
  17.     "storageClassName": "ceph"
  18.   }
  19. }
  20. EOF
  22. [root@m7-demo-136001 k8s]# kubectl create -f ceph-pvc-storageClass.json
  23. persistentvolumeclaim "pvc-test-claim" created
  25. [root@m7-demo-136001 k8s]# kubectl get pv
  26. NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                    STORAGECLASS   REASON    AGE
  27. pvc-0335828d-90b7-11e8-b43c-0cc47a2af650   1Gi        RWO            Delete           Bound     default/pvc-test-claim   ceph                     20m
  28. [root@m7-demo-136001 k8s]# kubectl describe pv pvc-0335828d-90b7-11e8-b43c-0cc47a2af650
  29. Name:            pvc-0335828d-90b7-11e8-b43c-0cc47a2af650
  30. Labels:          <none>
  31. Annotations:     kubernetes.io/createdby=rbd-dynamic-provisioner
  32.                  pv.kubernetes.io/bound-by-controller=yes
  33.                  pv.kubernetes.io/provisioned-by=kubernetes.io/rbd
  34. StorageClass:    ceph
  35. Status:          Bound
  36. Claim:           default/pvc-test-claim
  37. Reclaim Policy:  Delete
  38. Access Modes:    RWO
  39. Capacity:        1Gi
  40. Message:
  41. Source:
  42.     Type:          RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
  43.     CephMonitors:  [172.27.128.100:6789 172.27.128.101:6789 172.27.128.102:6789]
  44.     RBDImage:      kubernetes-dynamic-pvc-e1726fb7-90ba-11e8-9ba5-0cc47a2af650
  45.     FSType:
  46.     RBDPool:       rbd
  47.     RadosUser:     admin
  48.     Keyring:       /etc/ceph/keyring
  49.     SecretRef:     &{ceph-secret-admin}
  50.     ReadOnly:      false
  51. Events:            <none>

创建使用 PVC 的 Pod

  1. [root@m7-demo-136001 k8s]# cat > ceph-pv-test.yaml <<EOF
  2. apiVersion: v1
  3. kind: Pod
  4. metadata:
  5.   name: ceph-pv-test
  6. spec:
  7.   containers:
  8.   - name: busybox
  9.     image: busybox
  10.     command: ["sleep", "3600"]
  11.     volumeMounts:
  12.     - name: ceph-vol
  13.       mountPath: /mnt/rbd
  14.       readOnly: false
  15.   volumes:
  16.   - name: ceph-vol
  17.     persistentVolumeClaim:
  18.       claimName: pvc-test-claim
  19. EOF
  21. [root@m7-demo-136001 k8s]# kubectl create -f ceph-pv-test.yaml
  22. pod "ceph-pv-test" created
  24. [root@m7-demo-136001 k8s]# kubectl describe pods ceph-pv-test
  25. Name:         ceph-pv-test
  26. Namespace:    default
  27. Node:         m7-demo-136001/172.27.136.1
  28. Start Time:   Thu, 26 Jul 2018 18:25:48 +0800
  29. Labels:       <none>
  30. Annotations:  <none>
  31. Status:       Running
  32. IP:           172.30.24.43
  33. Containers:
  34.   busybox:
  35.     Container ID:  docker://c6c9f851cefe21a301caf10e4c18c68851c379d83477628560fa3c4006593c5b
  36.     Image:         busybox
  37.     Image ID:      docker-pullable://busybox@sha256:d21b79794850b4b15d8d332b451d95351d14c951542942a816eea69c9e04b240
  38.     Port:          <none>
  39.     Command:
  40.       sleep
  41.       3600
  42.     State:          Running
  43.       Started:      Thu, 26 Jul 2018 18:25:54 +0800
  44.     Ready:          True
  45.     Restart Count:  0
  46.     Environment:    <none>
  47.     Mounts:
  48.       /mnt/rbd from ceph-vol (rw)
  49.       /var/run/secrets/kubernetes.io/serviceaccount from default-token-4km88 (ro)
  50. Conditions:
  51.   Type           Status
  52.   Initialized    True
  53.   Ready          True
  54.   PodScheduled   True
  55. Volumes:
  56.   ceph-vol:
  57.     Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
  58.     ClaimName:  pvc-test-claim
  59.     ReadOnly:   false
  60.   default-token-4km88:
  61.     Type:        Secret (a volume populated by a Secret)
  62.     SecretName:  default-token-4km88
  63.     Optional:    false
  64. QoS Class:       BestEffort
  65. Node-Selectors:  <none>
  66. Tolerations:     <none>
  67. Events:
  68.   Type    Reason                 Age                From                       Message
  69.   ----    ------                 ----               ----                       -------
  70.   Normal  SuccessfulMountVolume  51s                kubelet, m7-demo-136001  MountVolume.SetUp succeeded for volume "default-token-4km88"
  71.   Normal  Scheduled              50s                default-scheduler          Successfully assigned ceph-pv-test to m7-demo-136001
  72.   Normal  SuccessfulMountVolume  50s (x2 over 50s)  kubelet, m7-demo-136001  MountVolume.SetUp succeeded for volume "pvc-0335828d-90b7-11e8-b43c-0cc47a2af650"
  73.   Normal  Pulling                49s                kubelet, m7-demo-136001  pulling image "busybox"
  74.   Normal  Pulled                 46s                kubelet, m7-demo-136001  Successfully pulled image "busybox"
  75.   Normal  Created                46s                kubelet, m7-demo-136001  Created container
  76.   Normal  Started                46s                kubelet, m7-demo-136001  Started container