Advanced OADP features and functionalities

This document provides information about advanced features and functionalities of OpenShift API for Data Protection (OADP).

Working with different Kubernetes API versions on the same cluster

Listing the Kubernetes API group versions on a cluster

A source cluster might offer multiple versions of an API, where one of these versions is the preferred API version. For example, a source cluster with an API named Example might be available in the example.com/v1 and example.com/v1beta2 API groups.

If you use Velero to back up and restore such a source cluster, Velero backs up only the version of that resource that uses the preferred version of its Kubernetes API.

To return to the above example, if example.com/v1 is the preferred API, then Velero only backs up the version of a resource that uses example.com/v1. Moreover, the target cluster needs to have example.com/v1 registered in its set of available API resources in order for Velero to restore the resource on the target cluster.

Therefore, you need to generate a list of the Kubernetes API group versions on your target cluster to be sure the preferred API version is registered in its set of available API resources.

Procedure

• Enter the following command:

$ oc api-resources

About Enable API Group Versions

By default, Velero only backs up resources that use the preferred version of the Kubernetes API. However, Velero also includes a feature, Enable API Group Versions, that overcomes this limitation. When enabled on the source cluster, this feature causes Velero to back up all Kubernetes API group versions that are supported on the cluster, not only the preferred one. After the versions are stored in the backup .tar file, they are available to be restored on the destination cluster.

For example, a source cluster with an API named Example might be available in the example.com/v1 and example.com/v1beta2 API groups, with example.com/v1 being the preferred API.

Without the Enable API Group Versions feature enabled, Velero backs up only the preferred API group version for Example, which is example.com/v1. With the feature enabled, Velero also backs up example.com/v1beta2.

When the Enable API Group Versions feature is enabled on the destination cluster, Velero selects the version to restore on the basis of the order of priority of API group versions.

Velero uses the following algorithm to assign priorities to API versions, with 1 as the top priority:

1. Preferred version of the destination cluster

2. Preferred version of the source_ cluster

3. Common non-preferred supported version with the highest Kubernetes version priority

Additional resources

• Enable API Group Versions Feature

Using Enable API Group Versions

You can use Velero’s Enable API Group Versions feature to back up all Kubernetes API group versions that are supported on a cluster, not only the preferred one.

Procedure

• Configure the EnableAPIGroupVersions feature flag:

apiVersion: oadp.openshift.io/vialpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      featureFlags:
      - EnableAPIGroupVersions

Additional resources

• Enable API Group Versions Feature

Backing up data from one cluster and restoring it to another cluster

About backing up data from one cluster and restoring it on another cluster

OpenShift API for Data Protection (OADP) is designed to back up and restore application data in the same OKD cluster. Migration Toolkit for Containers (MTC) is designed to migrate containers, including application data, from one OKD cluster to another cluster.

You can use OADP to back up application data from one OKD cluster and restore it on another cluster. However, doing so is more complicated than using MTC or using OADP to back up and restore on the same cluster.

To successfully use OADP to back up data from one cluster and restore it to another cluster, you must take into account the following factors, in addition to the prerequisites and procedures that apply to using OADP to back up and restore data on the same cluster:

• Operators

• Use of Velero

• UID and GID ranges

Operators

You must exclude Operators from the backup of an application for backup and restore to succeed.

Use of Velero

Velero, which OADP is built upon, does not natively support migrating persistent volume snapshots across cloud providers. To migrate volume snapshot data between cloud platforms, you must either enable the Velero Restic file system backup option, which backs up volume contents at the file system level, or use the OADP Data Mover for CSI snapshots.

• You must also use Velero’s File System Backup to migrate data between AWS regions or between Microsoft Azure regions.

• Velero does not support restoring data to a cluster with an earlier Kubernetes version than the source cluster.

• It is theoretically possible to migrate workloads to a destination with a later Kubernetes version than the source, but you must consider the compatibility of API groups between clusters for each custom resource. If a Kubernetes version upgrade breaks the compatibility of core or native API groups, you must first update the impacted custom resources.

About determining which pod volumes to back up

Before you start a backup operation by using File System Backup (FSB), you must specify which pods contain a volume that you want to back up. Velero refers to this process as “discovering” the appropriate pod volumes.

Velero supports two approaches for determining pod volumes:

• Opt-in approach: The opt-in approach requires that you actively indicate that you want to include - opt-in - a volume in a backup. You do this by labelling each pod that contains a volume to be backed up with the name of the volume. When Velero finds a persistent volume (PV), it checks the pod that mounted the volume. If the pod is labelled with the name of the volume, Velero backs up the pod.

• Opt-out approach: With the opt-out approach, you must actively specify that you want to exclude a volume from a backup. You do this by labelling each pod that contains a volume you do not want to back up with the name of the volume. When Velero finds a PV, it checks the pod that mounted the volume. If the pod is labelled with the volume’s name, Velero does not back up the pod.

Limitations

• FSB does not support backing up and restoring hostpath volumes. However, FSB does support backing up and restoring local volumes.

• Velero uses a static, common encryption key for all backup repositories it creates. This static key means that anyone who can access your backup storage can also decrypt your backup data. It is essential that you limit access to backup storage.

• For PVCs, every incremental backup chain is maintained across pod reschedules.

  For pod volumes that are not PVCs, such as emptyDir volumes, if a pod is deleted or recreated, for example, by a ReplicaSet or a deployment, the next backup of those volumes will be a full backup and not an incremental backup. It is assumed that the lifecycle of a pod volume is defined by its pod.

• Even though backup data can be kept incrementally, backing up large files, such as a database, can take a long time. This is because FSB uses deduplication to find the difference that needs to be backed up.

• FSB reads and writes data from volumes by accessing the file system of the node on which the pod is running. For this reason, FSB can only back up volumes that are mounted from a pod and not directly from a PVC. Some Velero users have overcome this limitation by running a staging pod, such as a BusyBox or Alpine container with an infinite sleep, to mount these PVC and PV pairs before performing a Velero backup..

• FSB expects volumes to be mounted under <hostPath>/<pod UID>, with <hostPath> being configurable. Some Kubernetes systems, for example, vCluster, do not mount volumes under the <pod UID> subdirectory, and VFSB does not work with them as expected.

Backing up pod volumes by using the opt-in method

You can use the opt-in method to specify which volumes need to be backed up by File System Backup (FSB). You can do this by using the backup.velero.io/backup-volumes command.

Procedure

• On each pod that contains one or more volumes that you want to back up, enter the following command:

  $ oc -n <your_pod_namespace> annotate pod/<your_pod_name> \
    backup.velero.io/backup-volumes=<your_volume_name_1>, \ <your_volume_name_2>>,...,<your_volume_name_n>

  where:

  <your_volume_name_x>

  specifies the name of the xth volume in the pod specification.

Backing up pod volumes by using the opt-out method

When using the opt-out approach, all pod volumes are backed up by using File System Backup (FSB), although there are some exceptions:

• Volumes that mount the default service account token, secrets, and configuration maps.

• hostPath volumes

You can use the opt-out method to specify which volumes not to back up. You can do this by using the backup.velero.io/backup-volumes-excludes command.

Procedure

• On each pod that contains one or more volumes that you do not want to back up, run the following command:

  $ oc -n <your_pod_namespace> annotate pod/<your_pod_name> \
    backup.velero.io/backup-volumes-excludes=<your_volume_name_1>, \ <your_volume_name_2>>,...,<your_volume_name_n>

  where:

  <your_volume_name_x>

  specifies the name of the xth volume in the pod specification.

UID and GID ranges

If you back up data from one cluster and restore it to another cluster, problems might occur with UID (User ID) and GID (Group ID) ranges. The following section explains these potential issues and mitigations:

Summary of the issues

The namespace UID and GID ranges might change depending on the destination cluster. OADP does not back up and restore OpenShift UID range metadata. If the backed up application requires a specific UID, ensure the range is availableupon restore. For more information about OpenShift’s UID and GID ranges, see A Guide to OpenShift and UIDs.

Detailed description of the issues

When you create a namespace in OKD by using the shell command oc create namespace, OKD assigns the namespace a unique User ID (UID) range from its available pool of UIDs, a Supplemental Group (GID) range, and unique SELinux MCS labels. This information is stored in the metadata.annotations field of the cluster. This information is part of the Security Context Constraints (SCC) annotations, which comprise of the following components:

• openshift.io/sa.scc.mcs

• openshift.io/sa.scc.supplemental-groups

• openshift.io/sa.scc.uid-range

When you use OADP to restore the namespace, it automatically uses the information in metadata.annotations without resetting it for the destination cluster. As a result, the workload might not have access to the backed up data if any of the following is true:

• There is an existing namespace with other SCC annotations, for example, on another cluster. In this case, OADP uses the existing namespace during the backup instead of the namespace you want to restore.

• A label selector was used during the backup, but the namespace in which the workloads are executed does not have the label. In this case, OADP does not back up the namespace, but creates a new namespace during the restore that does not contain the annotations of the backed up namespace. This results in a new UID range being assigned to the namespace.

  This can be an issue for customer workloads if OKD assigns a pod a securityContext UID to a pod based on namespace annotations that have changed since the persistent volume data was backed up.

• The UID of the container no longer matches the UID of the file owner.

• An error occurs because OKD has not changed the UID range of the destination cluster to match the backup cluster data. As a result, the backup cluster has a different UID than the destination cluster, which means that the application cannot read or write data on the destination cluster.

  Mitigations

  You can use one or more of the following mitigations to resolve the UID and GID range issues:

• Simple mitigations:

  • If you use a label selector in the Backup CR to filter the objects to include in the backup, be sure to add this label selector to the namespace that contains the workspace.

  • Remove any pre-existing version of a namespace on the destination cluster before attempting to restore a namespace with the same name.

• Advanced mitigations:

  • Fix UID ranges after migration by Resolving overlapping UID ranges in OpenShift namespaces after migration. Step 1 is optional.

For an in-depth discussion of UID and GID ranges in OKD with an emphasis on overcoming issues in backing up data on one cluster and restoring it on another, see A Guide to OpenShift and UIDs.

Backing up data from one cluster and restoring it to another cluster

In general, you back up data from one OKD cluster and restore it on another OKD cluster in the same way that you back up and restore data to the same cluster. However, there are some additional prerequisites and differences in the procedure when backing up data from one OKD cluster and restoring it on another.

Prerequisites

• All relevant prerequisites for backing up and restoring on your platform (for example, AWS, Microsoft Azure, GCP, and so on), especially the prerequisites for the Data Protection Application (DPA), are described in the relevant sections of this guide.

Procedure

• Make the following additions to the procedures given for your platform:

  • Ensure that the backup store location (BSL) and volume snapshot location have the same names and paths to restore resources to another cluster.

  • Share the same object storage location credentials across the clusters.

  • For best results, use OADP to create the namespace on the destination cluster.

  • If you use the Velero file-system-backup option, enable the --default-volumes-to-fs-backup flag for use during backup by running the following command:

    $ velero backup create <backup_name> --default-volumes-to-fs-backup <any_other_options>

Additional resources

For more information about API group versions, see Working with different Kubernetes API versions on the same cluster.

For more information about OADP Data Mover, see Using Data Mover for CSI snapshots.

For more information about using Restic with OADP, see Backing up applications with File System Backup: Kopia or Restic.