Restoring Volumes for Kubernetes Stateful Sets

Longhorn supports restoring backups, and one of the use cases for this feature is to restore data for use in a Kubernetes Stateful Set, which requires restoring a volume for each replica that was backed up. To restore, follow the below instructions based on which plugin you have deployed. The example below uses a Stateful Set with one volume attached to each Pod and two replicas.

CSI Instructions

  1. Connect to the Longhorn UI page in your web browser. Under the Backup tab, select the name of the Stateful Set volume. Click the dropdown menu of the volume entry and restore it. Name the volume something that can easily be referenced later for the Persistent Volumes.
  • Repeat this step for each volume you need restored.
  • For example, if restoring a Stateful Set with two replicas that had volumes named pvc-01a and pvc-02b, the restore could look like this:
Backup NameRestored Volume
pvc-01astatefulset-vol-0
pvc-02bstatefulset-vol-1
  1. In Kubernetes, create a Persistent Volume for each Longhorn volume that was created. Name the volumes something that can easily be referenced later for the Persistent Volume Claims. storage capacity, numberOfReplicas, storageClassName, and volumeHandle must be replaced below. In the example, we’re referencing statefulset-vol-0 and statefulset-vol-1 in Longhorn and using longhorn as our storageClassName.
  1. apiVersion: v1
  2. kind: PersistentVolume
  3. metadata:
  4. spec:
  5.   capacity:
  6.     storage: <size> # must match size of Longhorn volume
  7.   volumeMode: Filesystem
  8.   accessModes:
  9.     - ReadWriteOnce
  10.   persistentVolumeReclaimPolicy: Delete
  11.   csi:
  12.     driver: io.rancher.longhorn # driver must match this
  13.     fsType: ext4
  14.     volumeAttributes:
  15.       numberOfReplicas: <replicas> # must match Longhorn volume value
  16.       staleReplicaTimeout: '30'
  17.     volumeHandle: statefulset-vol-0 # must match volume name from Longhorn
  18.   storageClassName: longhorn # must be same name that we will use later
  19. ---
  20. apiVersion: v1
  21. kind: PersistentVolume
  22. metadata:
  23. spec:
  24.   capacity:
  25.     storage: <size>  # must match size of Longhorn volume
  26.   volumeMode: Filesystem
  27.   accessModes:
  28.     - ReadWriteOnce
  29.   persistentVolumeReclaimPolicy: Delete
  30.   csi:
  31.     driver: io.rancher.longhorn # driver must match this
  32.     fsType: ext4
  33.     volumeAttributes:
  34.       numberOfReplicas: <replicas> # must match Longhorn volume value
  35.       staleReplicaTimeout: '30'
  36.     volumeHandle: statefulset-vol-1 # must match volume name from Longhorn
  37.   storageClassName: longhorn # must be same name that we will use later
  1. Go to General Instructions.

Flexvolume Instructions

Because of the implementation of Flexvolume, creating the Longhorn volumes from the Longhorn UI manually can be skipped. Instead, follow these instructions:

  1. Connect to the Longhorn UI page in your web browser. Under the Backup tab, select the name of the Stateful Set volume. Click the dropdown menu of the volume entry and select Get URL.
  • Repeat this step for each volume you need restored. Save these URLs for the next step.
  • If using NFS backups, the URL will appear similar to:
    • nfs://longhorn-nfs-svc.default:/opt/backupstore?backup=backup-c57844b68923408f&volume=pvc-59b20247-99bf-11e8-8a92-be8835d7412a.
  • If using S3 backups, the URL will appear similar to:
    • s3://backupbucket@us-east-1/backupstore?backup=backup-1713a64cd2774c43&volume=longhorn-testvol-g1n1de
  1. Similar to Step 2 for CSI, create a Persistent Volume for each volume you want to restore. storage capacity, storageClassName, and the Flexvolume options must be replaced. This example uses longhorn as the storageClassName.
  1. apiVersion: v1
  2. kind: PersistentVolume
  3. metadata:
  4. spec:
  5.   capacity:
  6.     storage: <size> # must match "size" parameter below
  7.   accessModes:
  8.     - ReadWriteOnce
  9.   storageClassName: longhorn # must be same name that we will use later
  10.   flexVolume:
  11.     driver: "rancher.io/longhorn" # driver must match this
  12.     fsType: "ext4"
  13.     options:
  14.       size: <size> # must match "storage" parameter above
  15.       numberOfReplicas: <replicas>
  16.       staleReplicaTimeout: <timeout>
  17.       fromBackup: <backup URL> # must be set to Longhorn backup URL
  18. ---
  19. apiVersion: v1
  20. kind: PersistentVolume
  21. metadata:
  22. spec:
  23.   capacity:
  24.     storage: <size> # must match "size" parameter below
  25.   accessModes:
  26.     - ReadWriteOnce
  27.   storageClassName: longhorn # must be same name that we will use later
  28.   flexVolume:
  29.     driver: "rancher.io/longhorn" # driver must match this
  30.     fsType: "ext4"
  31.     options:
  32.       size: <size> # must match "storage" parameter above
  33.       numberOfReplicas: <replicas>
  34.       staleReplicaTimeout: <timeout>
  35.       fromBackup: <backup URL> # must be set to Longhorn backup URL
  1. Go to General Instructions.

General Instructions

Make sure you have followed either the CSI or Flexvolume instructions before following the steps in this section.

  1. In the namespace the Stateful Set will be deployed in, create Persistent Volume Claims for each Persistent Volume.
  • The name of the Persistent Volume Claim must follow this naming scheme: <name of Volume Claim Template>-<name of Stateful Set>-<index>. Stateful Set Pods are zero-indexed. In this example, the name of the Volume Claim Template is data, the name of the Stateful Set is webapp, and there are two replicas, which are indexes 0 and 1.
  1. apiVersion: v1
  2. kind: PersistentVolumeClaim
  3. metadata:
  4.   spec:
  5.     accessModes:
  6.       - ReadWriteOnce
  7.     resources:
  8.       requests:
  9.         storage: 2Gi # must match size from earlier
  10.   storageClassName: longhorn # must match name from earlier
  11.   volumeName: statefulset-vol-0 # must reference Persistent Volume
  12. ---
  13. apiVersion: v1
  14. kind: PersistentVolumeClaim
  15. metadata:
  16.   spec:
  17.     accessModes:
  18.       - ReadWriteOnce
  19.     resources:
  20.       requests:
  21.         storage: 2Gi # must match size from earlier
  22.   storageClassName: longhorn # must match name from earlier
  23.   volumeName: statefulset-vol-1 # must reference Persistent Volume
  1. Create the Stateful Set:
  1. apiVersion: apps/v1beta2
  2. kind: StatefulSet
  3. metadata:
  4. spec:
  5.   selector:
  6.     matchLabels:
  7.       app: nginx # has to match .spec.template.metadata.labels
  8.   serviceName: "nginx"
  9.   replicas: 2 # by default is 1
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx # has to match .spec.selector.matchLabels
  14.     spec:
  15.       terminationGracePeriodSeconds: 10
  16.       containers:
  17.         image: k8s.gcr.io/nginx-slim:0.8
  18.         ports:
  19.         - containerPort: 80
  20.         volumeMounts:
  21.           mountPath: /usr/share/nginx/html
  22.   volumeClaimTemplates:
  23.   - metadata:
  24.     spec:
  25.       accessModes: [ "ReadWriteOnce" ]
  26.       storageClassName: longhorn # must match name from earlier
  27.       resources:
  28.         requests:
  29.           storage: 2Gi # must match size from earlier

The restored data should now be accessible from inside the Stateful Set Pods.