Backing up and restoring etcd on a hosted cluster

If you use hosted control planes on OKD, the process to back up and restore etcd is different from the usual etcd backup process.

Taking a snapshot of etcd on a hosted cluster

As part of the process to back up etcd for a hosted cluster, you take a snapshot of etcd. After you take the snapshot, you can restore it, for example, as part of a disaster recovery operation.

Procedure

1. Pause reconciliation of the hosted cluster by entering this command:

   $ oc patch -n clusters hostedclusters/${CLUSTER_NAME} -p '{"spec":{"pausedUntil":"'${PAUSED_UNTIL}'"}}' --type=merge

2. Stop all etcd-writer deployments by entering this command:

   $ oc scale deployment -n ${HOSTED_CLUSTER_NAMESPACE} --replicas=0 kube-apiserver openshift-apiserver openshift-oauth-apiserver

3. Take an etcd snapshot by using the exec command in each etcd container:

   $ oc exec -it etcd-0 -n ${HOSTED_CLUSTER_NAMESPACE} -- env ETCDCTL_API=3 /usr/bin/etcdctl --cacert /etc/etcd/tls/client/etcd-client-ca.crt --cert /etc/etcd/tls/client/etcd-client.crt --key /etc/etcd/tls/client/etcd-client.key --endpoints=localhost:2379 snapshot save /var/lib/data/snapshot.db
   $ oc exec -it etcd-0 -n ${HOSTED_CLUSTER_NAMESPACE} -- env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /var/lib/data/snapshot.db

4. Copy the snapshot data to a location where you can retrieve it later, such as an S3 bucket, as shown in the following example.

   The following example uses signature version 2. If you are in a region that supports signature version 4, such as the us-east-2 region, use signature version 4. Otherwise, if you use signature version 2 to copy the snapshot to an S3 bucket, the upload fails and signature version 2 is deprecated.

   Example

   BUCKET_NAME=somebucket
   FILEPATH="/${BUCKET_NAME}/${CLUSTER_NAME}-snapshot.db"
   CONTENT_TYPE="application/x-compressed-tar"
   DATE_VALUE=`date -R`
   SIGNATURE_STRING="PUT\n\n${CONTENT_TYPE}\n${DATE_VALUE}\n${FILEPATH}"
   ACCESS_KEY=accesskey
   SECRET_KEY=secret
   SIGNATURE_HASH=`echo -en ${SIGNATURE_STRING} | openssl sha1 -hmac ${SECRET_KEY} -binary | base64`
   oc exec -it etcd-0 -n ${HOSTED_CLUSTER_NAMESPACE} -- curl -X PUT -T "/var/lib/data/snapshot.db" \
     -H "Host: ${BUCKET_NAME}.s3.amazonaws.com" \
     -H "Date: ${DATE_VALUE}" \
     -H "Content-Type: ${CONTENT_TYPE}" \
     -H "Authorization: AWS ${ACCESS_KEY}:${SIGNATURE_HASH}" \
     https://${BUCKET_NAME}.s3.amazonaws.com/${CLUSTER_NAME}-snapshot.db

5. If you want to be able to restore the snapshot on a new cluster later, save the encryption secret that the hosted cluster references, as shown in this example:

   Example

   oc get hostedcluster $CLUSTER_NAME -o=jsonpath='{.spec.secretEncryption.aescbc}'
   {"activeKey":{"name":"CLUSTER_NAME-etcd-encryption-key"}}
   # Save this secret, or the key it contains so the etcd data can later be decrypted
   oc get secret ${CLUSTER_NAME}-etcd-encryption-key -o=jsonpath='{.data.key}'

Next steps

Restore the etcd snapshot.

Restoring an etcd snapshot on a hosted cluster

If you have a snapshot of etcd from your hosted cluster, you can restore it. Currently, you can restore an etcd snapshot only during cluster creation.

To restore an etcd snapshot, you modify the output from the create cluster --render command and define a restoreSnapshotURL value in the etcd section of the HostedCluster specification.

Prerequisites

You took an etcd snapshot on a hosted cluster.

Procedure

1. On the aws command-line interface (CLI), create a pre-signed URL so that you can download your etcd snapshot from S3 without passing credentials to the etcd deployment:

   ETCD_SNAPSHOT=${ETCD_SNAPSHOT:-"s3://${BUCKET_NAME}/${CLUSTER_NAME}-snapshot.db"}
   ETCD_SNAPSHOT_URL=$(aws s3 presign ${ETCD_SNAPSHOT})

2. Modify the HostedCluster specification to refer to the URL:

   spec:
     etcd:
       managed:
         storage:
           persistentVolume:
             size: 4Gi
           type: PersistentVolume
           restoreSnapshotURL:
           - "${ETCD_SNAPSHOT_URL}"
       managementType: Managed

3. Ensure that the secret that you referenced from the spec.secretEncryption.aescbc value contains the same AES key that you saved in the previous steps.

Additional resources

• Disaster recovery for a hosted cluster within an AWS region