Creating an environment-wide backup

Creating an environment-wide backup involves copying important data to assist with restoration in the case of crashing instances, or corrupt data. After backups have been created, they can be restored onto a newly installed version of the relevant component.

In OKD, you can back up, saving state to separate storage, at the cluster level. The full state of an environment backup includes:

  • Cluster data files

  • etcd data on each master

  • API objects

  • Registry storage

  • Volume storage

Perform a back up on a regular basis to prevent data loss.

The following process describes a generic way of backing up applications and the OKD cluster. It cannot take into account custom requirements. Use these steps as a foundation for a full backup and restoration procedure for your cluster. You must take all necessary precautions to prevent data loss.

Backup and restore is not guaranteed. You are responsible for backing up your own data.

Creating a master host backup

Perform this backup process before any change to the OKD infrastructure, such as a system update, upgrade, or any other significant modification. Back up data regularly to ensure that recent data is available if a failure occurs.

OKD files

The master instances run important services, such as the API, controllers. The /etc/origin/master directory stores many important files:

  • The configuration, the API, controllers, services, and more

  • Certificates generated by the installation

  • All cloud provider-related configuration

  • Keys and other authentication files, such as htpasswd if you use htpasswd

  • And more

You can customize OKD services, such as increasing the log level or using proxies. The configuration files are stored in the /etc/sysconfig directory.

Because the masters are also nodes, back up the entire /etc/origin directory.

Procedure

You must perform the following steps on each master node.

  1. Create a backup of the pod definitions, located here.

  2. Create a backup of the master host configuration files:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
    3. $ sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
    4. $ sudo cp -aR /etc/sysconfig/ ${MYBACKUPDIR}/etc/sysconfig/

    The master configuration file is/etc/origin/master/master-config.yaml.

    The /etc/origin/master/ca.serial.txt file is generated on only the first master listed in the Ansible host inventory. If you deprecate the first master host, copy the /etc/origin/master/ca.serial.txt file to the rest of master hosts before the process.

    In OKD 3.11 clusters running multiple masters, one of the master nodes includes additional CA certificates in /etc/origin/master, /etc/etcd/ca and /etc/etcd/generated_certs. These are required for application node and etcd node scale-up operations and would need to be restored on another master node should the originating master become permanently unavailable. These directories are included by default within the backup procedures documented here.

  3. Other important files that need to be considered when planning a backup include:

    File

    Description

    /etc/cni/

    Container Network Interface configuration (if used)

    /etc/sysconfig/iptables

    Where the iptables rules are stored

    /etc/sysconfig/docker-storage-setup

    The input file for container-storage-setup command

    /etc/sysconfig/docker

    The docker configuration file

    /etc/sysconfig/docker-network

    docker networking configuration (i.e. MTU)

    /etc/sysconfig/docker-storage

    docker storage configuration (generated by container-storage-setup)

    /etc/dnsmasq.conf

    Main configuration file for dnsmasq

    /etc/dnsmasq.d/

    Different dnsmasq configuration files

    /etc/sysconfig/flanneld

    flannel configuration file (if used)

    /etc/pki/ca-trust/source/anchors/

    Certificates added to the system (i.e. for external registries)

    Create a backup of those files:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
    3. $ sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
    4. $ sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    5.     ${MYBACKUPDIR}/etc/sysconfig/
    6. $ sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
    7. $ sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    8.     ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

  4. If a package is accidentally removed or you need to resore a file that is included in an rpm package, having a list of rhel packages installed on the system can be useful.

    If you use Red Hat Satellite features, such as content views or the facts store, provide a proper mechanism to reinstall the missing packages and a historical data of packages installed in the systems.

    To create a list of the current rhel packages installed in the system:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo mkdir -p ${MYBACKUPDIR}
    3. $ rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt

  5. If you used the previous steps, the following files are present in the backup directory:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'
    3. etc/sysconfig/flanneld
    4. etc/sysconfig/iptables
    5. etc/sysconfig/docker-network
    6. etc/sysconfig/docker-storage
    7. etc/sysconfig/docker-storage-setup
    8. etc/sysconfig/docker-storage-setup.rpmnew
    9. etc/origin/master/ca.crt
    10. etc/origin/master/ca.key
    11. etc/origin/master/ca.serial.txt
    12. etc/origin/master/ca-bundle.crt
    13. etc/origin/master/master.proxy-client.crt
    14. etc/origin/master/master.proxy-client.key
    15. etc/origin/master/service-signer.crt
    16. etc/origin/master/service-signer.key
    17. etc/origin/master/serviceaccounts.private.key
    18. etc/origin/master/serviceaccounts.public.key
    19. etc/origin/master/openshift-master.crt
    20. etc/origin/master/openshift-master.key
    21. etc/origin/master/openshift-master.kubeconfig
    22. etc/origin/master/master.server.crt
    23. etc/origin/master/master.server.key
    24. etc/origin/master/master.kubelet-client.crt
    25. etc/origin/master/master.kubelet-client.key
    26. etc/origin/master/admin.crt
    27. etc/origin/master/admin.key
    28. etc/origin/master/admin.kubeconfig
    29. etc/origin/master/etcd.server.crt
    30. etc/origin/master/etcd.server.key
    31. etc/origin/master/master.etcd-client.key
    32. etc/origin/master/master.etcd-client.csr
    33. etc/origin/master/master.etcd-client.crt
    34. etc/origin/master/master.etcd-ca.crt
    35. etc/origin/master/policy.json
    36. etc/origin/master/scheduler.json
    37. etc/origin/master/htpasswd
    38. etc/origin/master/session-secrets.yaml
    39. etc/origin/master/openshift-router.crt
    40. etc/origin/master/openshift-router.key
    41. etc/origin/master/registry.crt
    42. etc/origin/master/registry.key
    43. etc/origin/master/master-config.yaml
    44. etc/origin/generated-configs/master-master-1.example.com/master.server.crt
    45. ...[OUTPUT OMITTED]...
    46. etc/origin/cloudprovider/openstack.conf
    47. etc/origin/node/system:node:master-0.example.com.crt
    48. etc/origin/node/system:node:master-0.example.com.key
    49. etc/origin/node/ca.crt
    50. etc/origin/node/system:node:master-0.example.com.kubeconfig
    51. etc/origin/node/server.crt
    52. etc/origin/node/server.key
    53. etc/origin/node/node-dnsmasq.conf
    54. etc/origin/node/resolv.conf
    55. etc/origin/node/node-config.yaml
    56. etc/origin/node/flannel.etcd-client.key
    57. etc/origin/node/flannel.etcd-client.csr
    58. etc/origin/node/flannel.etcd-client.crt
    59. etc/origin/node/flannel.etcd-ca.crt
    60. etc/pki/ca-trust/source/anchors/openshift-ca.crt
    61. etc/pki/ca-trust/source/anchors/registry-ca.crt
    62. etc/dnsmasq.conf
    63. etc/dnsmasq.d/origin-dns.conf
    64. etc/dnsmasq.d/origin-upstream-dns.conf
    65. etc/dnsmasq.d/node-dnsmasq.conf
    66. packages.txt

    If needed, you can compress the files to save space:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
    3. $ sudo rm -Rf ${MYBACKUPDIR}

To create any of these files from scratch, the openshift-ansible-contrib repository contains the backup_master_node.sh script, which performs the previous steps. The script creates a directory on the host where you run the script and copies all the files previously mentioned.

The openshift-ansible-contrib script is not supported by Red Hat, but the reference architecture team performs testing to ensure the code operates as defined and is secure.

You can run the script on every master host with:

  1. $ mkdir ~/git
  2. $ cd ~/git
  3. $ git clone https://github.com/openshift/openshift-ansible-contrib.git
  4. $ cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
  5. $ ./backup_master_node.sh -h

Creating a node host backup

Creating a backup of a node host is a different use case from backing up a master host. Because master hosts contain many important files, creating a backup is highly recommended. However, the nature of nodes is that anything special is replicated over the nodes in case of failover, and they typically do not contain data that is necessary to run an environment. If a backup of a node contains something necessary to run an environment, then a creating a backup is recommended.

The backup process is to be performed before any change to the infrastructure, such as a system update, upgrade, or any other significant modification. Backups should be performed on a regular basis to ensure the most recent data is available if a failure occurs.

OKD files

Node instances run applications in the form of pods, which are based on containers. The /etc/origin/ and /etc/origin/node directories house important files, such as:

  • The configuration of the node services

  • Certificates generated by the installation

  • Cloud provider-related configuration

  • Keys and other authentication files, such as the dnsmasq configuration

The OKD services can be customized to increase the log level, use proxies, and more, and the configuration files are stored in the /etc/sysconfig directory.

Procedure

  1. Create a backup of the node configuration files:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
    3. $ sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
    4. $ sudo cp -aR /etc/sysconfig/atomic-openshift-node ${MYBACKUPDIR}/etc/sysconfig/

  2. OKD uses specific files that must be taken into account when planning the backup policy, including:

    File

    Description

    /etc/cni/

    Container Network Interface configuration (if used)

    /etc/sysconfig/iptables

    Where the iptables rules are stored

    /etc/sysconfig/docker-storage-setup

    The input file for container-storage-setup command

    /etc/sysconfig/docker

    The docker configuration file

    /etc/sysconfig/docker-network

    docker networking configuration (i.e. MTU)

    /etc/sysconfig/docker-storage

    docker storage configuration (generated by container-storage-setup)

    /etc/dnsmasq.conf

    Main configuration file for dnsmasq

    /etc/dnsmasq.d/

    Different dnsmasq configuration files

    /etc/sysconfig/flanneld

    flannel configuration file (if used)

    /etc/pki/ca-trust/source/anchors/

    Certificates added to the system (i.e. for external registries)

    To create those files:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
    3. $ sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
    4. $ sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    5.     ${MYBACKUPDIR}/etc/sysconfig/
    6. $ sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
    7. $ sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    8.     ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

  3. If a package is accidentally removed, or a file included in an rpm package should be restored, having a list of rhel packages installed on the system can be useful.

    If using Red Hat Satellite features, such as content views or the facts store, provide a proper mechanism to reinstall the missing packages and a historical data of packages installed in the systems.

    To create a list of the current rhel packages installed in the system:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo mkdir -p ${MYBACKUPDIR}
    3. $ rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt

  4. The following files should now be present in the backup directory:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'
    3. etc/sysconfig/atomic-openshift-node
    4. etc/sysconfig/flanneld
    5. etc/sysconfig/iptables
    6. etc/sysconfig/docker-network
    7. etc/sysconfig/docker-storage
    8. etc/sysconfig/docker-storage-setup
    9. etc/sysconfig/docker-storage-setup.rpmnew
    10. etc/origin/node/system:node:app-node-0.example.com.crt
    11. etc/origin/node/system:node:app-node-0.example.com.key
    12. etc/origin/node/ca.crt
    13. etc/origin/node/system:node:app-node-0.example.com.kubeconfig
    14. etc/origin/node/server.crt
    15. etc/origin/node/server.key
    16. etc/origin/node/node-dnsmasq.conf
    17. etc/origin/node/resolv.conf
    18. etc/origin/node/node-config.yaml
    19. etc/origin/node/flannel.etcd-client.key
    20. etc/origin/node/flannel.etcd-client.csr
    21. etc/origin/node/flannel.etcd-client.crt
    22. etc/origin/node/flannel.etcd-ca.crt
    23. etc/origin/cloudprovider/openstack.conf
    24. etc/pki/ca-trust/source/anchors/openshift-ca.crt
    25. etc/pki/ca-trust/source/anchors/registry-ca.crt
    26. etc/dnsmasq.conf
    27. etc/dnsmasq.d/origin-dns.conf
    28. etc/dnsmasq.d/origin-upstream-dns.conf
    29. etc/dnsmasq.d/node-dnsmasq.conf
    30. packages.txt

    If needed, the files can be compressed to save space:

    1. $ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
    2. $ sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
    3. $ sudo rm -Rf ${MYBACKUPDIR}

To create any of these files from scratch, the openshift-ansible-contrib repository contains the backup_master_node.sh script, which performs the previous steps. The script creates a directory on the host running the script and copies all the files previously mentioned.

The openshift-ansible-contrib script is not supported by Red Hat, but the reference architecture team performs testing to ensure the code operates as defined and is secure.

The script can be executed on every master host with:

  1. $ mkdir ~/git
  2. $ cd ~/git
  3. $ git clone https://github.com/openshift/openshift-ansible-contrib.git
  4. $ cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
  5. $ ./backup_master_node.sh -h

Backing up registry certificates

If you use an external secured registry, you must save all the registry certificates. The registry is secured by default.

You must perform the following steps on each cluster node.

Procedure

  1. Back up the registry certificates:

    1. # cd /etc/docker/certs.d/
    2. # tar cf /tmp/docker-registry-certs-$(hostname).tar *

  2. Move the backup to an external location.

When working with one or more external secured registry, any host that pulls or pushes images must trust the registry certificates to run pods.

Backing up other installation files

Back up the files that you used to install OKD.

Procedure

  1. Because the restoration procedure involves a complete reinstallation, save all the files used in the initial installation. These files might include:

  1. Backup the procedures for post-installation steps. Some installations might involve steps that are not included in the installer. These steps might include changes to the services outside of the control of OKD or the installation of extra services like monitoring agents. Additional configuration that is not yet supported by the advanced installer might also be affected, such as using multiple authentication providers.

Backing up application data

In many cases, you can back up application data by using the oc rsync command, assuming rsync is installed within the container image. The Red Hat rhel7 base image contains rsync. Therefore, all images that are based on rhel7 contain it as well. See Troubleshooting and Debugging CLI Operations - rsync.

This is a generic backup of application data and does not take into account application-specific backup procedures, for example, special export and import procedures for database systems.

Other means of backup might exist depending on the type of the persistent volume you use, for example, Cinder, NFS, or Gluster.

The paths to back up are also application specific. You can determine what path to back up by looking at the **mountPath** for volumes in the **deploymentconfig**.

You can perform this type of application data backup only while the application pod is running.

Procedure

Example of backing up a Jenkins deployment’s application data

  1. Get the application data **mountPath** from the **deploymentconfig**:

    1. $ oc get dc/jenkins -o jsonpath='{ .spec.template.spec.containers[?(@.name=="jenkins")].volumeMounts[?(@.name=="jenkins-data")].mountPath }'
    2. /var/lib/jenkins

  2. Get the name of the pod that is currently running:

    1. $ oc get pod --selector=deploymentconfig=jenkins -o jsonpath='{ .metadata.name }'
    2. jenkins-1-37nux

  3. Use the oc rsync command to copy application data:

    1. $ oc rsync jenkins-1-37nux:/var/lib/jenkins /tmp/

etcd backup

etcd is the key value store for all object definitions, as well as the persistent master state. Other components watch for changes, then bring themselves into the desired state.

OKD versions prior to 3.5 use etcd version 2 (v2), while 3.5 and later use version 3 (v3). The data model between the two versions of etcd is different. etcd v3 can use both the v2 and v3 data models, whereas etcd v2 can only use the v2 data model. In an etcd v3 server, the v2 and v3 data stores exist in parallel and are independent.

For both v2 and v3 operations, you can use the ETCDCTL_API environment variable to use the correct API:

  1. $ etcdctl -v
  2. etcdctl version: 3.2.5
  3. API version: 2
  4. $ ETCDCTL_API=3 etcdctl version
  5. etcdctl version: 3.2.5
  6. API version: 3.2

See Migrating etcd Data (v2 to v3) section in the OKD 3.7 documentation for information about how to migrate to v3.

In OKD version 3.10 and later, you can either install etcd on separate hosts or run it as a static pod on your master hosts. If you do not specify separate etcd hosts, etcd runs as a static pod on master hosts. Because of this difference, the backup process is different if you use static pods.

The etcd backup process is composed of two different procedures:

  • Configuration backup: Including the required etcd configuration and certificates

  • Data backup: Including both v2 and v3 data model.

You can perform the data backup process on any host that has connectivity to the etcd cluster, where the proper certificates are provided, and where the etcdctl tool is installed.

The backup files must be copied to an external system, ideally outside the OKD environment, and then encrypted.

Note that the etcd backup still has all the references to current storage volumes. When you restore etcd, OKD starts launching the previous pods on nodes and reattaching the same storage. This process is no different than the process of when you remove a node from the cluster and add a new one back in its place. Anything attached to that node is reattached to the pods on whatever nodes they are rescheduled to.

Backing up etcd

When you back up etcd, you must back up both the etcd configuration files and the etcd data.

Backing up etcd configuration files

The etcd configuration files to be preserved are all stored in the /etc/etcd directory of the instances where etcd is running. This includes the etcd configuration file (/etc/etcd/etcd.conf) and the required certificates for cluster communication. All those files are generated at installation time by the Ansible installer.

Procedure

For each etcd member of the cluster, back up the etcd configuration.

  1. $ ssh master-0
  2. # mkdir -p /backup/etcd-config-$(date +%Y%m%d)/
  3. # cp -R /etc/etcd/ /backup/etcd-config-$(date +%Y%m%d)/

The certificates and configuration files on each etcd cluster member are unique.

Backing up etcd data

Prerequisites

The OKD installer creates aliases to avoid typing all the flags named etcdctl2 for etcd v2 tasks and etcdctl3 for etcd v3 tasks.

However, the etcdctl3 alias does not provide the full endpoint list to the etcdctl command, so you must specify the —endpoints option and list all the endpoints.

Before backing up etcd:

  • etcdctl binaries must be available or, in containerized installations, the rhel7/etcd container must be available.

  • Ensure that the OKD API service is running.

  • Ensure connectivity with the etcd cluster (port 2379/tcp).

  • Ensure the proper certificates to connect to the etcd cluster.

  • Ensure go is installed.

    1. To ensure the etcd cluster is working, check its health.

      • Run the following command:

        1. # ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
        2.           --key=/etc/etcd/peer.key \
        3.           --cacert="/etc/etcd/ca.crt" \
        4.           --endpoints="https://*master-0.example.com*:2379,\
        5.             https://*master-1.example.com*:2379,\
        6.             https://*master-2.example.com*:2379"
        7.             endpoint health
        8. https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 5.011358ms
        9. https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.305173ms
        10. https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.388772ms
  1. 2.  Check the member list.
  3.     ```
  4.     # etcdctl3 member list
  5.     2a371dd20f21ca8d, started, master-1.example.com, https://192.168.55.12:2380, https://192.168.55.12:2379
  6.     40bef1f6c79b3163, started, master-0.example.com, https://192.168.55.8:2380, https://192.168.55.8:2379
  7.     95dc17ffcce8ee29, started, master-2.example.com, https://192.168.55.13:2380, https://192.168.55.13:2379
  8.     ```
Procedure

While the etcdctl backup command is used to perform the backup, etcd v3 has no concept of a backup. Instead, you either take a snapshot from a live member with the etcdctl snapshot save command or copy the member/snap/db file from an etcd data directory.

The etcdctl backup command rewrites some of the metadata contained in the backup, specifically, the node ID and cluster ID, which means that in the backup, the node loses its former identity. To recreate a cluster from the backup, you create a new, single-node cluster, then add the rest of the nodes to the cluster. The metadata is rewritten to prevent the new node from joining an existing cluster.

Back up the etcd data:

+

Clusters upgraded from previous versions of OKD might contain v2 data stores. Back up all etcd data stores.

  1. Make a snapshot of the etcd node:

    1. # systemctl show etcd --property=ActiveState,SubState
    2. # mkdir -p /var/lib/etcd/backup/etcd-$(date +%Y%m%d) (1)
    3. # etcdctl3 snapshot save /var/lib/etcd/backup/etcd-$(date +%Y%m%d)/db
    1You must write the snapshot to a directory under /var/lib/etcd/.

    The etcdctl snapshot save command requires the etcd service to be running.

  2. Stop all etcd services by removing the etcd pod definition and rebooting the host:

    1. # mkdir -p /etc/origin/node/pods-stopped
    2. # mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/

  3. Create the etcd data backup and copy the etcd db file:

    1. # etcdctl2 backup \
    2.     --data-dir /var/lib/etcd \
    3.     --backup-dir /backup/etcd-$(date +%Y%m%d)

    A /backup/etcd-<date>/ directory is created, where <date> represents the current date, which must be an external NFS share, S3 bucket, or any external storage location.

    In the case of an all-in-one cluster, the etcd data directory is located in the /var/lib/origin/openshift.local.etcd directory.

    • If etcd runs as a static pod, run the following commands:

      If you use static pods, use the v3 API.

  1. Obtain the etcd endpoint IP address from the static pod manifest:

    1. $ export ETCD_POD_MANIFEST="/etc/origin/node/pods/etcd.yaml"
    2. $ export ETCD_EP=$(grep https ${ETCD_POD_MANIFEST} | cut -d '/' -f3)

  2. Obtain the etcd pod name:

    1. $ oc login -u system:admin
    2. $ export ETCD_POD=$(oc get pods -n kube-system | grep -o -m 1 '\S*etcd\S*')

  3. Take a snapshot of the etcd data in the pod and store it locally:

    1. $ oc project kube-system
    2. $ oc exec ${ETCD_POD} -c etcd -- /bin/bash -c "ETCDCTL_API=3 etcdctl \
    3.     --cert /etc/etcd/peer.crt \
    4.     --key /etc/etcd/peer.key \
    5.     --cacert /etc/etcd/ca.crt \
    6.     --endpoints $ETCD_EP \
    7.     snapshot save /var/lib/etcd/snapshot.db"

Backing up a project

Creating a backup of all relevant data involves exporting all important information, then restoring into a new project.

Because the oc get all command returns only certain project resources, you must separately back up other resources, including PVCs and Secrets, as shown in the following steps.

Procedure

  1. List the project data to back up:

    1. $ oc get all
    2. NAME         TYPE      FROM      LATEST
    3. bc/ruby-ex   Source    Git       1
    5. NAME               TYPE      FROM          STATUS     STARTED         DURATION
    6. builds/ruby-ex-1   Source    Git@c457001   Complete   2 minutes ago   35s
    8. NAME                 DOCKER REPO                                     TAGS      UPDATED
    9. is/guestbook         10.111.255.221:5000/myproject/guestbook         latest    2 minutes ago
    10. is/hello-openshift   10.111.255.221:5000/myproject/hello-openshift   latest    2 minutes ago
    11. is/ruby-22-centos7   10.111.255.221:5000/myproject/ruby-22-centos7   latest    2 minutes ago
    12. is/ruby-ex           10.111.255.221:5000/myproject/ruby-ex           latest    2 minutes ago
    14. NAME                 REVISION   DESIRED   CURRENT   TRIGGERED BY
    15. dc/guestbook         1          1         1         config,image(guestbook:latest)
    16. dc/hello-openshift   1          1         1         config,image(hello-openshift:latest)
    17. dc/ruby-ex           1          1         1         config,image(ruby-ex:latest)
    19. NAME                   DESIRED   CURRENT   READY     AGE
    20. rc/guestbook-1         1         1         1         2m
    21. rc/hello-openshift-1   1         1         1         2m
    22. rc/ruby-ex-1           1         1         1         2m
    24. NAME                  CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
    25. svc/guestbook         10.111.105.84    <none>        3000/TCP            2m
    26. svc/hello-openshift   10.111.230.24    <none>        8080/TCP,8888/TCP   2m
    27. svc/ruby-ex           10.111.232.117   <none>        8080/TCP            2m
    29. NAME                         READY     STATUS      RESTARTS   AGE
    30. po/guestbook-1-c010g         1/1       Running     0          2m
    31. po/hello-openshift-1-4zw2q   1/1       Running     0          2m
    32. po/ruby-ex-1-build           0/1       Completed   0          2m
    33. po/ruby-ex-1-rxc74           1/1       Running     0          2m

  2. Export the project objects to a .yaml or .json file.

    • To export the project objects into a project.yaml file:

      1. $ oc get -o yaml --export all > project.yaml

    • To export the project objects into a project.json file:

      1. $ oc get -o json --export all > project.json

  1. Export the project’s role bindings, secrets, service accounts, and persistent volume claims:

    1. $ for object in rolebindings serviceaccounts secrets imagestreamtags cm egressnetworkpolicies rolebindingrestrictions limitranges resourcequotas pvc templates cronjobs statefulsets hpa deployments replicasets poddisruptionbudget endpoints
    2. do
    3.   oc get -o yaml --export $object > $object.yaml
    4. done

  2. To list all the namespaced objects:

    1. $ oc api-resources --namespaced=true -o name

  3. Some exported objects can rely on specific metadata or references to unique IDs in the project. This is a limitation on the usability of the recreated objects.

    When using imagestreams, the image parameter of a deploymentconfig can point to a specific sha checksum of an image in the internal registry that would not exist in a restored environment. For instance, running the sample “ruby-ex” as oc new-app centos/ruby-22-centos7~https://github.com/sclorg/ruby-ex.git creates an imagestream ruby-ex using the internal registry to host the image:

    1. $ oc get dc ruby-ex -o jsonpath="{.spec.template.spec.containers[].image}"
    2. 10.111.255.221:5000/myproject/ruby-ex@sha256:880c720b23c8d15a53b01db52f7abdcbb2280e03f686a5c8edfef1a2a7b21cee

    If importing the deploymentconfig as it is exported with oc get --export it fails if the image does not exist.

Backing up persistent volume claims

You can synchronize persistent data from inside of a container to a server.

Depending on the provider that is hosting the OKD environment, the ability to launch third party snapshot services for backup and restore purposes also exists. As OKD does not have the ability to launch these services, this guide does not describe these steps.

Consult any product documentation for the correct backup procedures of specific applications. For example, copying the mysql data directory itself does not create a usable backup. Instead, run the specific backup procedures of the associated application and then synchronize any data. This includes using snapshot solutions provided by the OKD hosting platform.

Procedure

  1. View the project and pods:

    1. $ oc get pods
    2. NAME           READY     STATUS      RESTARTS   AGE
    3. demo-1-build   0/1       Completed   0          2h
    4. demo-2-fxx6d   1/1       Running     0          1h

  2. Describe the desired pod to find the volumes that are currently used by a persistent volume:

    1. $ oc describe pod demo-2-fxx6d
    2. Name:            demo-2-fxx6d
    3. Namespace:        test
    4. Security Policy:    restricted
    5. Node:            ip-10-20-6-20.ec2.internal/10.20.6.20
    6. Start Time:        Tue, 05 Dec 2017 12:54:34 -0500
    7. Labels:            app=demo
    8.             deployment=demo-2
    9.             deploymentconfig=demo
    10. Status:            Running
    11. IP:            172.16.12.5
    12. Controllers:        ReplicationController/demo-2
    13. Containers:
    14.   demo:
    15.     Container ID:    docker://201f3e55b373641eb36945d723e1e212ecab847311109b5cee1fd0109424217a
    16.     Image:        docker-registry.default.svc:5000/test/demo@sha256:0a9f2487a0d95d51511e49d20dc9ff6f350436f935968b0c83fcb98a7a8c381a
    17.     Image ID:        docker-pullable://docker-registry.default.svc:5000/test/demo@sha256:0a9f2487a0d95d51511e49d20dc9ff6f350436f935968b0c83fcb98a7a8c381a
    18.     Port:        8080/TCP
    19.     State:        Running
    20.       Started:        Tue, 05 Dec 2017 12:54:52 -0500
    21.     Ready:        True
    22.     Restart Count:    0
    23.     Volume Mounts:
    24.       */opt/app-root/src/uploaded from persistent-volume (rw)*
    25.       /var/run/secrets/kubernetes.io/serviceaccount from default-token-8mmrk (ro)
    26.     Environment Variables:    <none>
    27. ...omitted...

    This output shows that the persistent data is in the /opt/app-root/src/uploaded directory.

  3. Copy the data locally:

    1. $ oc rsync demo-2-fxx6d:/opt/app-root/src/uploaded ./demo-app
    2. receiving incremental file list
    3. uploaded/
    4. uploaded/ocp_sop.txt
    5. uploaded/lost+found/
    7. sent 38 bytes  received 190 bytes  152.00 bytes/sec
    8. total size is 32  speedup is 0.14

    The ocp_sop.txt file is downloaded to the local system to be backed up by backup software or another backup mechanism.

    You can also use the previous steps if a pod starts without needing to use a pvc, but you later decide that a pvc is necessary. You can preserve the data and then use the restorate process to populate the new storage.