Using an External Ceph Driver

These instructions are about using the external Ceph driver in an RKE2 cluster. If you are using RKE, additional steps are required. For details, refer to this section.

Requirements

Make sure ceph-common and xfsprogs packages are installed on SLE worker nodes.

Using the Ceph Driver with RKE

The resources below are fully compatible with RKE based clusters, but there is a need to do an additional kubelet configuration for RKE.

On RKE clusters, the kubelet component is running in a Docker container and doesn’t have access to the host’s kernel modules as rbd and libceph by default.

To solve this limitation, you can either run modprobe rbd on worker nodes, or configure the kubelet containers to automatically mount the /lib/modules directory from the host into the container.

For the kubelet configuration, put the following lines into the cluster.yml file prior to RKE cluster provisioning. You can also modify the cluster.yml later in the Rancher UI by clicking on Edit Cluster > Edit as YAML and restarting the worker nodes.

  1. services:
  2. kubelet:
  3. extra_binds:
  4. - '/lib/modules:/lib/modules:ro'

For more information about the extra_binds directive, refer to this section.

Installing the ceph-csi driver on an RKE2 cluster

Using an External Ceph Driver - 图1note

These steps are needed for dynamic RBD provisioning only.

For more information about the ceph-csi-rbd chart, refer to this page.

To get details about your SES cluster, run:

  1. ceph mon dump

Read its output:

  1. dumped monmap epoch 3
  2. epoch 3
  3. fsid 79179d9d-98d8-4976-ab2e-58635caa7235
  4. last_changed 2021-02-11T10:56:42.110184+0000
  5. created 2021-02-11T10:56:22.913321+0000
  6. min_mon_release 15 (octopus)
  7. 0: [v2:10.85.8.118:3300/0,v1:10.85.8.118:6789/0] mon.a
  8. 1: [v2:10.85.8.123:3300/0,v1:10.85.8.123:6789/0] mon.b
  9. 2: [v2:10.85.8.124:3300/0,v1:10.85.8.124:6789/0] mon.c

Later you’ll need the fsid and mon addresses values.

Install the ceph-csi Driver Using Helm

Run these commands:

  1. helm repo add ceph-csi https://ceph.github.io/csi-charts
  2. helm repo update
  3. helm search repo ceph-csi -l
  4. helm inspect values ceph-csi/ceph-csi-rbd > ceph-csi-rbd-values.yaml

Modify the ceph-csi-rbd-values.yaml file and keep there only the required changes:

  1. # ceph-csi-rbd-values.yaml
  2. csiConfig:
  3. - clusterID: "79179d9d-98d8-4976-ab2e-58635caa7235"
  4. monitors:
  5. - "10.85.8.118:6789"
  6. - "10.85.8.123:6789"
  7. - "10.85.8.124:6789"
  8. provisioner:
  9. name: provisioner
  10. replicaCount: 2

Make sure the ceph monitors are reachable from the RKE2 cluster, for example, by ping.

  1. kubectl create namespace ceph-csi-rbd
  2. helm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml
  3. kubectl rollout status deployment ceph-csi-rbd-provisioner -n ceph-csi-rbd
  4. helm status ceph-csi-rbd -n ceph-csi-rbd

in case you’d like to modify the configuration directly via Helm, you may adapt the ceph-csi-rbd-values.yaml file and call:

  1. helm upgrade \
  2. --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml

Creating RBD Ceph Resources

  1. # Create a ceph pool:
  2. ceph osd pool create myPool 64 64
  3. # Create a block device pool:
  4. rbd pool init myPool
  5. # Create a block device image:
  6. rbd create -s 2G myPool/image
  7. # Create a block device user and record the key:
  8. ceph auth get-or-create-key client.myPoolUser mon "allow r" osd "allow class-read object_prefix rbd_children, allow rwx pool=myPool" | tr -d '\n' | base64
  9. QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==
  10. # Encode the ceph user myPoolUser into a bash64 hash:
  11. echo "myPoolUser" | tr -d '\n' | base64
  12. bXlQb29sVXNlcg==
  13. # Create a block device admin user and record the key:
  14. ceph auth get-or-create-key client.myPoolAdmin mds 'allow *' mgr 'allow *' mon 'allow *' osd 'allow * pool=myPool' | tr -d '\n' | base64
  15. QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==
  16. # Encode the ceph user myPoolAdmin into a bash64 hash:
  17. echo "myPoolAdmin" | tr -d '\n' | base64
  18. bXlQb29sQWRtaW4=

Configure RBD Ceph Access Secrets

User Account

For static RBD provisioning (the image within the ceph pool must exist), run these commands:

  1. cat > ceph-user-secret.yaml << EOF
  2. apiVersion: v1
  3. kind: Secret
  4. metadata:
  5. name: ceph-user
  6. namespace: default
  7. type: kubernetes.io/rbd
  8. data:
  9. userID: bXlQb29sVXNlcg==
  10. userKey: QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==
  11. EOF
  12. kubectl apply -f ceph-user-secret.yaml

Admin Account

For dynamic RBD provisioning (used for automatic image creation within a given ceph pool), run these commands:

  1. cat > ceph-admin-secret.yaml << EOF
  2. apiVersion: v1
  3. kind: Secret
  4. metadata:
  5. name: ceph-admin
  6. namespace: default
  7. type: kubernetes.io/rbd
  8. data:
  9. userID: bXlQb29sQWRtaW4=
  10. userKey: QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==
  11. EOF
  12. kubectl apply -f ceph-admin-secret.yaml

Create RBD Testing Resources

Using RBD in Pods

  1. # pod
  2. cat > ceph-rbd-pod-inline.yaml << EOF
  3. apiVersion: v1
  4. kind: Pod
  5. metadata:
  6. name: ceph-rbd-pod-inline
  7. spec:
  8. containers:
  9. - name: ceph-rbd-pod-inline
  10. image: busybox
  11. command: ["sleep", "infinity"]
  12. volumeMounts:
  13. - mountPath: /mnt/ceph_rbd
  14. name: volume
  15. volumes:
  16. - name: volume
  17. rbd:
  18. monitors:
  19. - 10.85.8.118:6789
  20. - 10.85.8.123:6789
  21. - 10.85.8.124:6789
  22. pool: myPool
  23. image: image
  24. user: myPoolUser
  25. secretRef:
  26. name: ceph-user
  27. fsType: ext4
  28. readOnly: false
  29. EOF
  30. kubectl apply -f ceph-rbd-pod-inline.yaml
  31. kubectl get pod
  32. kubectl exec pod/ceph-rbd-pod-inline -- df -k | grep rbd

Using RBD in Persistent Volumes

  1. # pod-pvc-pv
  2. cat > ceph-rbd-pod-pvc-pv-allinone.yaml << EOF
  3. apiVersion: v1
  4. kind: PersistentVolume
  5. metadata:
  6. name: ceph-rbd-pv
  7. spec:
  8. capacity:
  9. storage: 2Gi
  10. accessModes:
  11. - ReadWriteOnce
  12. rbd:
  13. monitors:
  14. - 10.85.8.118:6789
  15. - 10.85.8.123:6789
  16. - 10.85.8.124:6789
  17. pool: myPool
  18. image: image
  19. user: myPoolUser
  20. secretRef:
  21. name: ceph-user
  22. fsType: ext4
  23. readOnly: false
  24. ---
  25. kind: PersistentVolumeClaim
  26. apiVersion: v1
  27. metadata:
  28. name: ceph-rbd-pvc
  29. spec:
  30. accessModes:
  31. - ReadWriteOnce
  32. resources:
  33. requests:
  34. storage: 2Gi
  35. ---
  36. apiVersion: v1
  37. kind: Pod
  38. metadata:
  39. name: ceph-rbd-pod-pvc-pv
  40. spec:
  41. containers:
  42. - name: ceph-rbd-pod-pvc-pv
  43. image: busybox
  44. command: ["sleep", "infinity"]
  45. volumeMounts:
  46. - mountPath: /mnt/ceph_rbd
  47. name: volume
  48. volumes:
  49. - name: volume
  50. persistentVolumeClaim:
  51. claimName: ceph-rbd-pvc
  52. EOF
  53. kubectl apply -f ceph-rbd-pod-pvc-pv-allinone.yaml
  54. kubectl get pv,pvc,pod
  55. kubectl exec pod/ceph-rbd-pod-pvc-pv -- df -k | grep rbd

Using RBD in Storage Classes

This example is for dynamic provisioning. The ceph-csi driver is needed.

  1. # pod-pvc-sc
  2. cat > ceph-rbd-pod-pvc-sc-allinone.yaml <<EOF
  3. apiVersion: storage.k8s.io/v1
  4. kind: StorageClass
  5. metadata:
  6. name: ceph-rbd-sc
  7. annotations:
  8. storageclass.kubernetes.io/is-default-class: "true"
  9. provisioner: rbd.csi.ceph.com
  10. parameters:
  11. clusterID: 79179d9d-98d8-4976-ab2e-58635caa7235
  12. pool: myPool
  13. imageFeatures: layering
  14. csi.storage.k8s.io/provisioner-secret-name: ceph-admin
  15. csi.storage.k8s.io/provisioner-secret-namespace: default
  16. csi.storage.k8s.io/controller-expand-secret-name: ceph-admin
  17. csi.storage.k8s.io/controller-expand-secret-namespace: default
  18. csi.storage.k8s.io/node-stage-secret-name: ceph-admin
  19. csi.storage.k8s.io/node-stage-secret-namespace: default
  20. reclaimPolicy: Delete
  21. allowVolumeExpansion: true
  22. mountOptions:
  23. - discard
  24. ---
  25. kind: PersistentVolumeClaim
  26. apiVersion: v1
  27. metadata:
  28. name: ceph-rbd-sc-pvc
  29. spec:
  30. accessModes:
  31. - ReadWriteOnce
  32. resources:
  33. requests:
  34. storage: 2Gi
  35. storageClassName: ceph-rbd-sc
  36. ---
  37. apiVersion: v1
  38. kind: Pod
  39. metadata:
  40. name: ceph-rbd-pod-pvc-sc
  41. spec:
  42. containers:
  43. - name: ceph-rbd-pod-pvc-sc
  44. image: busybox
  45. command: ["sleep", "infinity"]
  46. volumeMounts:
  47. - mountPath: /mnt/ceph_rbd
  48. name: volume
  49. volumes:
  50. - name: volume
  51. persistentVolumeClaim:
  52. claimName: ceph-rbd-sc-pvc
  53. EOF
  54. kubectl apply -f ceph-rbd-pod-pvc-sc-allinone.yaml
  55. kubectl get pv,pvc,sc,pod
  56. kubectl exec pod/ceph-rbd-pod-pvc-sc -- df -k | grep rbd

RKE2 Server/Master Provisioning

  1. sudo su
  2. curl -sfL https://get.rke2.io | sh -
  3. systemctl enable --now rke2-server
  4. cat > /root/.bashrc << EOF
  5. export PATH=$PATH:/var/lib/rancher/rke2/bin/
  6. export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
  7. EOF
  8. cat /var/lib/rancher/rke2/server/node-token
  9. token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED

RKE2 Agent/Worker provisioning

  1. mkdir -p /etc/rancher/rke2/
  2. cat > /etc/rancher/rke2/config.yaml << EOF
  3. server: https://10.100.103.23:9345
  4. token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED
  5. EOF
  6. curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -
  7. systemctl enable --now rke2-agent.service

To import the cluster into Rancher, click ☰ > Cluster Management. Then on the Clusters page, click Import Existing. Then run the provided kubectl command on the server/master node.

Tested Versions

OS for running RKE2 nodes: JeOS SLE15-SP2 with installed kernel-default-5.3.18-24.49

  1. kubectl version
  2. Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-22T12:00:00Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
  3. Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7+rke2r1", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-20T01:50:52Z", GoVersion:"go1.15.5b5", Compiler:"gc", Platform:"linux/amd64"}
  4. helm version
  5. version.BuildInfo{Version:"3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.12"}

Kubernetes version on RKE2 cluster: v1.19.7+rke2r1

Troubleshooting

In case you are using SUSE’s ceph-rook based on SES7, it might be useful to expose the monitors on hostNetwork by editing rook-1.4.5/ceph/cluster.yaml and setting spec.network.hostNetwork=true.

Also for operating the ceph-rook cluster, it is useful to deploy a toolbox on the Kubernetes cluster where ceph-rook is provisioned by kubectl apply -f rook-1.4.5/ceph/toolbox.yaml Then all the ceph related commands can be executed in the toolbox pod, for example, by running kubectl exec -it -n rook-ceph rook-ceph-tools-686d8b8bfb-2nvqp -- bash

Operating with the ceph - basic commands:

  1. ceph osd pool stats
  2. ceph osd pool delete myPool myPool --yes-i-really-really-mean-it
  3. rbd list -p myPool
  4. > csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38
  5. > image

Delete the image: rbd rm csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38 -p myPool

CephFS commands in rook toolbox:

  1. ceph -s
  2. ceph fs ls
  3. ceph fs fail cephfs
  4. ceph fs rm cephfs --yes-i-really-mean-it
  5. ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
  6. ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it

To prepare a cephfs filesystem, you can run this command on a rook cluster:

  1. kubectl apply -f rook-1.4.5/ceph/filesystem.yaml