Using OADP 1.2 Data Mover with Ceph storage

You can use OADP 1.2 Data Mover to backup and restore application data for clusters that use CephFS, CephRBD, or both.

OADP 1.2 Data Mover leverages Ceph features that support large-scale environments. One of these is the shallow copy method, which is available for OKD 4.12 and later. This feature supports backing up and restoring StorageClass and AccessMode resources other than what is found on the source persistent volume claim (PVC).

The CephFS shallow copy feature is a back up feature. It is not part of restore operations.

Prerequisites for using OADP 1.2 Data Mover with Ceph storage

The following prerequisites apply to all back up and restore operations of data using OpenShift API for Data Protection (OADP) 1.2 Data Mover in a cluster that uses Ceph storage:

  • You have installed OKD 4.12 or later.

  • You have installed the OADP Operator.

  • You have created a secret cloud-credentials in the namespace openshift-adp.

  • You have installed Red Hat OpenShift Data Foundation.

  • You have installed the latest VolSync Operator by using Operator Lifecycle Manager.

Defining custom resources for use with OADP 1.2 Data Mover

When you install Red Hat OpenShift Data Foundation, it automatically creates default CephFS and a CephRBD StorageClass and VolumeSnapshotClass custom resources (CRs). You must define these CRs for use with OpenShift API for Data Protection (OADP) 1.2 Data Mover.

After you define the CRs, you must make several other changes to your environment before you can perform your back up and restore operations.

Defining CephFS custom resources for use with OADP 1.2 Data Mover

When you install Red Hat OpenShift Data Foundation, it automatically creates a default CephFS StorageClass custom resource (CR) and a default CephFS VolumeSnapshotClass CR. You can define these CRs for use with OpenShift API for Data Protection (OADP) 1.2 Data Mover.

Procedure

  1. Define the VolumeSnapshotClass CR as in the following example:

    Example VolumeSnapshotClass CR

    1. apiVersion: snapshot.storage.k8s.io/v1
    2. deletionPolicy: Retain (1)
    3. driver: openshift-storage.cephfs.csi.ceph.com
    4. kind: VolumeSnapshotClass
    5. metadata:
    6.   annotations:
    7.     snapshot.storage.kubernetes.io/is-default-class: true (2)
    8.   labels:
    9.     velero.io/csi-volumesnapshot-class: true (3)
    10.   name: ocs-storagecluster-cephfsplugin-snapclass
    11. parameters:
    12.   clusterID: openshift-storage
    13.   csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
    14.   csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
    1Must be set to Retain.
    2Must be set to true.
    3Must be set to true.

  2. Define the StorageClass CR as in the following example:

    Example StorageClass CR

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4.   name: ocs-storagecluster-cephfs
    5.   annotations:
    6.     description: Provides RWO and RWX Filesystem volumes
    7.     storageclass.kubernetes.io/is-default-class: true (1)
    8. provisioner: openshift-storage.cephfs.csi.ceph.com
    9. parameters:
    10.   clusterID: openshift-storage
    11.   csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
    12.   csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
    13.   csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
    14.   csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
    15.   csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
    16.   csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
    17.   fsName: ocs-storagecluster-cephfilesystem
    18. reclaimPolicy: Delete
    19. allowVolumeExpansion: true
    20. volumeBindingMode: Immediate
    1Must be set to true.

Defining CephRBD custom resources for use with OADP 1.2 Data Mover

When you install Red Hat OpenShift Data Foundation, it automatically creates a default CephRBD StorageClass custom resource (CR) and a default CephRBD VolumeSnapshotClass CR. You can define these CRs for use with OpenShift API for Data Protection (OADP) 1.2 Data Mover.

Procedure

  1. Define the VolumeSnapshotClass CR as in the following example:

    Example VolumeSnapshotClass CR

    1. apiVersion: snapshot.storage.k8s.io/v1
    2. deletionPolicy: Retain (1)
    3. driver: openshift-storage.rbd.csi.ceph.com
    4. kind: VolumeSnapshotClass
    5. metadata:
    6.   labels:
    7.     velero.io/csi-volumesnapshot-class: true (2)
    8.   name: ocs-storagecluster-rbdplugin-snapclass
    9. parameters:
    10.   clusterID: openshift-storage
    11.   csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
    12.   csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
    1Must be set to Retain.
    2Must be set to true.

  2. Define the StorageClass CR as in the following example:

    Example StorageClass CR

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4.   name: ocs-storagecluster-ceph-rbd
    5.   annotations:
    6.     description: 'Provides RWO Filesystem volumes, and RWO and RWX Block volumes'
    7. provisioner: openshift-storage.rbd.csi.ceph.com
    8. parameters:
    9.   csi.storage.k8s.io/fstype: ext4
    10.   csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
    11.   csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    12.   csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    13.   csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    14.   imageFormat: '2'
    15.   clusterID: openshift-storage
    16.   imageFeatures: layering
    17.   csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
    18.   pool: ocs-storagecluster-cephblockpool
    19.   csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
    20. reclaimPolicy: Delete
    21. allowVolumeExpansion: true
    22. volumeBindingMode: Immediate

Defining additional custom resources for use with OADP 1.2 Data Mover

After you redefine the default StorageClass and CephRBD VolumeSnapshotClass custom resources (CRs), you must create the following CRs:

  • A CephFS StorageClass CR defined to use the shallow copy feature

  • A Restic Secret CR

Procedure

  1. Create a CephFS StorageClass CR and set the backingSnapshot parameter set to true as in the following example:

    Example CephFS StorageClass CR with backingSnapshot set to true

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4.   name: ocs-storagecluster-cephfs-shallow
    5.   annotations:
    6.     description: Provides RWO and RWX Filesystem volumes
    7.     storageclass.kubernetes.io/is-default-class: false
    8. provisioner: openshift-storage.cephfs.csi.ceph.com
    9. parameters:
    10.   csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
    11.   csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
    12.   csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
    13.   csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
    14.   clusterID: openshift-storage
    15.   fsName: ocs-storagecluster-cephfilesystem
    16.   csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
    17.   backingSnapshot: true (1)
    18.   csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
    19. reclaimPolicy: Delete
    20. allowVolumeExpansion: true
    21. volumeBindingMode: Immediate
    1Must be set to true.

    Ensure that the CephFS VolumeSnapshotClass and StorageClass CRs have the same value for provisioner.

  2. Configure a Restic Secret CR as in the following example:

    Example Restic Secret CR

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: <secret_name>
    5.   namespace: <namespace>
    6. type: Opaque
    7. stringData:
    8.   RESTIC_PASSWORD: <restic_password>

Backing up and restoring data using OADP 1.2 Data Mover and CephFS storage

You can use OpenShift API for Data Protection (OADP) 1.2 Data Mover to back up and restore data using CephFS storage by enabling the shallow copy feature of CephFS.

Prerequisites

  • A stateful application is running in a separate namespace with persistent volume claims (PVCs) using CephFS as the provisioner.

  • The StorageClass and VolumeSnapshotClass custom resources (CRs) are defined for CephFS and OADP 1.2 Data Mover.

  • There is a secret cloud-credentials in the openshift-adp namespace.

Creating a DPA for use with CephFS storage

You must create a Data Protection Application (DPA) CR before you use the OpenShift API for Data Protection (OADP) 1.2 Data Mover to back up and restore data using CephFS storage.

Procedure

  1. Verify that the deletionPolicy field of the VolumeSnapshotClass CR is set to Retain by running the following command:

    1. $ oc get volumesnapshotclass -A  -o jsonpath='{range .items[*]}{"Name: "}{.metadata.name}{"  "}{"Retention Policy: "}{.deletionPolicy}{"\n"}{end}'

  2. Verify that the labels of the VolumeSnapshotClass CR are set to true by running the following command:

    1. $ oc get volumesnapshotclass -A  -o jsonpath='{range .items[*]}{"Name: "}{.metadata.name}{"  "}{"labels: "}{.metadata.labels}{"\n"}{end}'

  3. Verify that the storageclass.kubernetes.io/is-default-class annotation of the StorageClass CR is set to true by running the following command:

    1. $ oc get storageClass -A  -o jsonpath='{range .items[*]}{"Name: "}{.metadata.name}{"  "}{"annotations: "}{.metadata.annotations}{"\n"}{end}'

  4. Create a Data Protection Application (DPA) CR similar to the following example:

    Example DPA CR

    1. apiVersion: oadp.openshift.io/v1alpha1
    2. kind: DataProtectionApplication
    3. metadata:
    4.   name: velero-sample
    5.   namespace: openshift-adp
    6. spec:
    7.   backupLocations:
    8.     - velero:
    9.         config:
    10.           profile: default
    11.           region: us-east-1
    12.         credential:
    13.           key: cloud
    14.           name: cloud-credentials
    15.         default: true
    16.         objectStorage:
    17.           bucket: <my_bucket>
    18.           prefix: velero
    19.        provider: aws
    20.     configuration:
    21.       restic:
    22.         enable: false  (1)
    23.       velero:
    24.         defaultPlugins:
    25.           - openshift
    26.           - aws
    27.           - csi
    28.           - vsm
    29.     features:
    30.       dataMover:
    31.         credentialName: <restic_secret_name> (2)
    32.         enable: true (3)
    33.         volumeOptionsForStorageClasses:
    34.           ocs-storagecluster-cephfs:
    35.             sourceVolumeOptions:
    36.               accessMode: ReadOnlyMany
    37.               cacheAccessMode: ReadWriteMany
    38.               cacheStorageClassName: ocs-storagecluster-cephfs
    39.               storageClassName: ocs-storagecluster-cephfs-shallow
    1There is no default value for the enable field. Valid values are true or false.
    2Use the Restic Secret that you created when you prepared your environment for working with OADP 1.2 Data Mover and Ceph. If you do not use your Restic Secret, the CR uses the default value dm-credential for this parameter.
    3There is no default value for the enable field. Valid values are true or false.

Backing up data using OADP 1.2 Data Mover and CephFS storage

You can use OpenShift API for Data Protection (OADP) 1.2 Data Mover to back up data using CephFS storage by enabling the shallow copy feature of CephFS storage.

Procedure

  1. Create a Backup CR as in the following example:

    Example Backup CR

    1. apiVersion: velero.io/v1
    2. kind: Backup
    3. metadata:
    4.   name: <backup_name>
    5.   namespace: <protected_ns>
    6. spec:
    7.   includedNamespaces:
    8.   - <app_ns>
    9.   storageLocation: velero-sample-1

  2. Monitor the progress of the VolumeSnapshotBackup CRs by completing the following steps:

    1. To check the progress of all the VolumeSnapshotBackup CRs, run the following command:

      1. $ oc get vsb -n <app_ns>

    2. To check the progress of a specific VolumeSnapshotBackup CR, run the following command:

      1. $ oc get vsb <vsb_name> -n <app_ns> -ojsonpath="{.status.phase}`

  3. Wait several minutes until the VolumeSnapshotBackup CR has the status Completed.

  4. Verify that there is at least one snapshot in the object store that is given in the Restic Secret. You can check for this snapshot in your targeted BackupStorageLocation storage provider that has a prefix of /<OADP_namespace>.

Restoring data using OADP 1.2 Data Mover and CephFS storage

You can use OpenShift API for Data Protection (OADP) 1.2 Data Mover to restore data using CephFS storage if the shallow copy feature of CephFS storage was enabled for the back up procedure. The shallow copy feature is not used in the restore procedure.

Procedure

  1. Delete the application namespace by running the following command:

    1. $ oc delete vsb -n <app_namespace> --all

  2. Delete any VolumeSnapshotContent CRs that were created during backup by running the following command:

    1. $ oc delete volumesnapshotcontent --all

  3. Create a Restore CR as in the following example:

    Example Restore CR

    1. apiVersion: velero.io/v1
    2. kind: Restore
    3. metadata:
    4.   name: <restore_name>
    5.   namespace: <protected_ns>
    6. spec:
    7.   backupName: <previous_backup_name>

  4. Monitor the progress of the VolumeSnapshotRestore CRs by doing the following:

    1. To check the progress of all the VolumeSnapshotRestore CRs, run the following command:

      1. $ oc get vsr -n <app_ns>

    2. To check the progress of a specific VolumeSnapshotRestore CR, run the following command:

      1. $ oc get vsr <vsr_name> -n <app_ns> -ojsonpath="{.status.phase}

  5. Verify that your application data has been restored by running the following command:

    1. $ oc get route <route_name> -n <app_ns> -ojsonpath="{.spec.host}"

Backing up and restoring data using OADP 1.2 Data Mover and split volumes (CephFS and Ceph RBD)

You can use OpenShift API for Data Protection (OADP) 1.2 Data Mover to back up and restore data in an environment that has split volumes, that is, an environment that uses both CephFS and CephRBD.

Prerequisites

  • A stateful application is running in a separate namespace with persistent volume claims (PVCs) using CephFS as the provisioner.

  • The StorageClass and VolumeSnapshotClass custom resources (CRs) are defined for CephFS and OADP 1.2 Data Mover.

  • There is a secret cloud-credentials in the openshift-adp namespace.

Creating a DPA for use with split volumes

You must create a Data Protection Application (DPA) CR before you use the OpenShift API for Data Protection (OADP) 1.2 Data Mover to back up and restore data using split volumes.

Procedure

  • Create a Data Protection Application (DPA) CR as in the following example:

    Example DPA CR for environment with split volumes

    1. apiVersion: oadp.openshift.io/v1alpha1
    2. kind: DataProtectionApplication
    3. metadata:
    4.   name: velero-sample
    5.   namespace: openshift-adp
    6. spec:
    7.   backupLocations:
    8.     - velero:
    9.         config:
    10.           profile: default
    11.           region: us-east-1
    12.         credential:
    13.           key: cloud
    14.           name: cloud-credentials
    15.         default: true
    16.         objectStorage:
    17.           bucket: <my-bucket>
    18.           prefix: velero
    19.         provider: aws
    20.   configuration:
    21.     restic:
    22.       enable: false
    23.     velero:
    24.       defaultPlugins:
    25.         - openshift
    26.         - aws
    27.         - csi
    28.         - vsm
    29.   features:
    30.     dataMover:
    31.       credentialName: <restic_secret_name> (1)
    32.       enable: true
    33.       volumeOptionsForStorageClasses: (2)
    34.         ocs-storagecluster-cephfs:
    35.           sourceVolumeOptions:
    36.             accessMode: ReadOnlyMany
    37.             cacheAccessMode: ReadWriteMany
    38.             cacheStorageClassName: ocs-storagecluster-cephfs
    39.             storageClassName: ocs-storagecluster-cephfs-shallow
    40.         ocs-storagecluster-ceph-rbd:
    41.           sourceVolumeOptions:
    42.             storageClassName: ocs-storagecluster-ceph-rbd
    43.             cacheStorageClassName: ocs-storagecluster-ceph-rbd
    44.         destinationVolumeOptions:
    45.             storageClassName: ocs-storagecluster-ceph-rbd
    46.             cacheStorageClassName: ocs-storagecluster-ceph-rbd
    1Use the Restic Secret that you created when you prepared your environment for working with OADP 1.2 Data Mover and Ceph. If you do not, then the CR will use the default value dm-credential for this parameter.
    2A different set of VolumeOptionsForStorageClass labels can be defined for each storageClass volume, thus allowing a backup to volumes with different providers.

Backing up data using OADP 1.2 Data Mover and split volumes

You can use OpenShift API for Data Protection (OADP) 1.2 Data Mover to back up data in an environment that has split volumes.

Procedure

  1. Create a Backup CR as in the following example:

    Example Backup CR

    1. apiVersion: velero.io/v1
    2. kind: Backup
    3. metadata:
    4.   name: <backup_name>
    5.   namespace: <protected_ns>
    6. spec:
    7.   includedNamespaces:
    8.   - <app_ns>
    9.   storageLocation: velero-sample-1

  2. Monitor the progress of the VolumeSnapshotBackup CRs by completing the following steps:

    1. To check the progress of all the VolumeSnapshotBackup CRs, run the following command:

      1. $ oc get vsb -n <app_ns>

    2. To check the progress of a specific VolumeSnapshotBackup CR, run the following command:

      1. $ oc get vsb <vsb_name> -n <app_ns> -ojsonpath="{.status.phase}`

  3. Wait several minutes until the VolumeSnapshotBackup CR has the status Completed.

  4. Verify that there is at least one snapshot in the object store that is given in the Restic Secret. You can check for this snapshot in your targeted BackupStorageLocation storage provider that has a prefix of /<OADP_namespace>.

Restoring data using OADP 1.2 Data Mover and split volumes

You can use OpenShift API for Data Protection (OADP) 1.2 Data Mover to restore data in an environment that has split volumes, if the shallow copy feature of CephFS storage was enabled for the back up procedure. The shallow copy feature is not used in the restore procedure.

Procedure

  1. Delete the application namespace by running the following command:

    1. $ oc delete vsb -n <app_namespace> --all

  2. Delete any VolumeSnapshotContent CRs that were created during backup by running the following command:

    1. $ oc delete volumesnapshotcontent --all

  3. Create a Restore CR as in the following example:

    Example Restore CR

    1. apiVersion: velero.io/v1
    2. kind: Restore
    3. metadata:
    4.   name: <restore_name>
    5.   namespace: <protected_ns>
    6. spec:
    7.   backupName: <previous_backup_name>

  4. Monitor the progress of the VolumeSnapshotRestore CRs by doing the following:

    1. To check the progress of all the VolumeSnapshotRestore CRs, run the following command:

      1. $ oc get vsr -n <app_ns>

    2. To check the progress of a specific VolumeSnapshotRestore CR, run the following command:

      1. $ oc get vsr <vsr_name> -n <app_ns> -ojsonpath="{.status.phase}

  5. Verify that your application data has been restored by running the following command:

    1. $ oc get route <route_name> -n <app_ns> -ojsonpath="{.spec.host}"