Complete Example Using Ceph RBD

You are viewing documentation for a release that is no longer supported. The latest supported version of version 3 is [3.11]. For the most recent version 4, see [4]

You are viewing documentation for a release that is no longer supported. The latest supported version of version 3 is [3.11]. For the most recent version 4, see [4]

Overview

This topic provides an end-to-end example of using an existing Ceph cluster as an OKD persistent store. It is assumed that a working Ceph cluster is already set up. If not, consult the Overview of Red Hat Ceph Storage.

Persistent Storage Using Ceph Rados Block Device provides an explanation of persistent volumes (PVs), persistent volume claims (PVCs), and using Ceph RBD as persistent storage.

All oc …​ commands are executed on the OKD master host.

Installing the ceph-common Package

The ceph-common library must be installed on all schedulable OKD nodes:

The OKD all-in-one host is not often used to run pod workloads and, thus, is not included as a schedulable node.

  1. # yum install -y ceph-common

Creating the Ceph Secret

The ceph auth get-key command is run on a Ceph MON node to display the key value for the client.admin user:

Example 1. Ceph Secret Definition

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4. name: ceph-secret
  5. data:
  6. key: QVFBOFF2SlZheUJQRVJBQWgvS2cwT1laQUhPQno3akZwekxxdGc9PQ== (1)
1This base64 key is generated on one of the Ceph MON nodes using the ceph auth get-key client.admin | base64 command, then copying the output and pasting it as the secret key’s value.

Save the secret definition to a file, for example ceph-secret.yaml, then create the secret:

  1. $ oc create -f ceph-secret.yaml
  2. secret "ceph-secret" created

Verify that the secret was created:

  1. # oc get secret ceph-secret
  2. NAME TYPE DATA AGE
  3. ceph-secret Opaque 1 23d

Creating the Persistent Volume

Next, before creating the PV object in OKD, define the persistent volume file:

Example 2. Persistent Volume Object Definition Using Ceph RBD

  1. apiVersion: v1
  2. kind: PersistentVolume
  3. metadata:
  4. name: ceph-pv (1)
  5. spec:
  6. capacity:
  7. storage: 2Gi (2)
  8. accessModes:
  9. - ReadWriteOnce (3)
  10. rbd: (4)
  11. monitors: (5)
  12. - 192.168.122.133:6789
  13. pool: rbd
  14. image: ceph-image
  15. user: admin
  16. secretRef:
  17. name: ceph-secret (6)
  18. fsType: ext4 (7)
  19. readOnly: false
  20. persistentVolumeReclaimPolicy: Recycle
1The name of the PV, which is referenced in pod definitions or displayed in various oc volume commands.
2The amount of storage allocated to this volume.
3accessModes are used as labels to match a PV and a PVC. They currently do not define any form of access control. All block storage is defined to be single user (non-shared storage).
4This defines the volume type being used. In this case, the rbd plug-in is defined.
5This is an array of Ceph monitor IP addresses and ports.
6This is the Ceph secret, defined above. It is used to create a secure connection from OKD to the Ceph server.
7This is the file system type mounted on the Ceph RBD block device.

Save the PV definition to a file, for example ceph-pv.yaml, and create the persistent volume:

  1. # oc create -f ceph-pv.yaml
  2. persistentvolume "ceph-pv" created

Verify that the persistent volume was created:

  1. # oc get pv
  2. NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
  3. ceph-pv <none> 2147483648 RWO Available 2s

Creating the Persistent Volume Claim

A persistent volume claim (PVC) specifies the desired access mode and storage capacity. Currently, based on only these two attributes, a PVC is bound to a single PV. Once a PV is bound to a PVC, that PV is essentially tied to the PVC’s project and cannot be bound to by another PVC. There is a one-to-one mapping of PVs and PVCs. However, multiple pods in the same project can use the same PVC.

Example 3. PVC Object Definition

  1. kind: PersistentVolumeClaim
  2. apiVersion: v1
  3. metadata:
  4. name: ceph-claim
  5. spec:
  6. accessModes: (1)
  7. - ReadWriteOnce
  8. resources:
  9. requests:
  10. storage: 2Gi (2)
1As mentioned above for PVs, the accessModes do not enforce access right, but rather act as labels to match a PV to a PVC.
2This claim will look for PVs offering 2Gi or greater capacity.

Save the PVC definition to a file, for example ceph-claim.yaml, and create the PVC:

  1. # oc create -f ceph-claim.yaml
  2. persistentvolumeclaim "ceph-claim" created
  3. #and verify the PVC was created and bound to the expected PV:
  4. # oc get pvc
  5. NAME LABELS STATUS VOLUME CAPACITY ACCESSMODES AGE
  6. ceph-claim <none> Bound ceph-pv 1Gi RWX 21s
  7. (1)
1the claim was bound to the ceph-pv PV.

Creating the Pod

A pod definition file or a template file can be used to define a pod. Below is a pod specification that creates a single container and mounts the Ceph RBD volume for read-write access:

Example 4. Pod Object Definition

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4. name: ceph-pod1 (1)
  5. spec:
  6. containers:
  7. - name: ceph-busybox
  8. image: busybox (2)
  9. command: ["sleep", "60000"]
  10. volumeMounts:
  11. - name: ceph-vol1 (3)
  12. mountPath: /usr/share/busybox (4)
  13. readOnly: false
  14. volumes:
  15. - name: ceph-vol1 (3)
  16. persistentVolumeClaim:
  17. claimName: ceph-claim (5)
1The name of this pod as displayed by oc get pod.
2The image run by this pod. In this case, we are telling busybox to sleep.
3The name of the volume. This name must be the same in both the containers and volumes sections.
4The mount path as seen in the container.
5The PVC that is bound to the Ceph RBD cluster.

Save the pod definition to a file, for example ceph-pod1.yaml, and create the pod:

  1. # oc create -f ceph-pod1.yaml
  2. pod "ceph-pod1" created
  3. #verify pod was created
  4. # oc get pod
  5. NAME READY STATUS RESTARTS AGE
  6. ceph-pod1 1/1 Running 0 2m
  7. (1)
1After a minute or so, the pod will be in the Running state.

Defining Group and Owner IDs (Optional)

When using block storage, such as Ceph RBD, the physical block storage is managed by the pod. The group ID defined in the pod becomes the group ID of both the Ceph RBD mount inside the container, and the group ID of the actual storage itself. Thus, it is usually unnecessary to define a group ID in the pod specifiation. However, if a group ID is desired, it can be defined using **fsGroup**, as shown in the following pod definition fragment:

Example 5. Group ID Pod Definition

  1. ...
  2. spec:
  3. containers:
  4. - name:
  5. ...
  6. securityContext: (1)
  7. fsGroup: 7777 (2)
  8. ...
1securityContext must be defined at the pod level, not under a specific container.
2All containers in the pod will have the same fsGroup ID.

Setting ceph-user-secret as Default for Projects

If you would like to make the persistent storage available to every project you have to modify the default project template. You can read more on modifying the default project template. Read more on modifying the default project template. Adding this to your default project template allows every user who has access to create a project access to the Ceph cluster.

Example 6. Default Project Example

  1. ...
  2. apiVersion: v1
  3. kind: Template
  4. metadata:
  5. creationTimestamp: null
  6. name: project-request
  7. objects:
  8. - apiVersion: v1
  9. kind: Project
  10. metadata:
  11. annotations:
  12. openshift.io/description: ${PROJECT_DESCRIPTION}
  13. openshift.io/display-name: ${PROJECT_DISPLAYNAME}
  14. openshift.io/requester: ${PROJECT_REQUESTING_USER}
  15. creationTimestamp: null
  16. name: ${PROJECT_NAME}
  17. spec: {}
  18. status: {}
  19. - apiVersion: v1
  20. kind: Secret
  21. metadata:
  22. name: ceph-user-secret
  23. data:
  24. key: yoursupersecretbase64keygoeshere (1)
  25. type:
  26. kubernetes.io/rbd
  27. ...
1Place your super secret Ceph user key here in base64 format. See Creating the Ceph Secret.