Restore Longhorn System

Longhorn System Restore Rollouts

  • Longhorn restores the resource from the Longhorn System Backup Bundle.
  • Longhorn does not restore existing Volumes and their associated PersistentVolume and PersistentVolumeClaim.
  • Longhorn automatically restores a Volume from its latest backup.

Limitations

Restore Path

Longhorn does not support cross-major/minor version system restore except for upgrade failures, ex: 1.4.x -> 1.5.

Create Longhorn System Restore

You can restore the Longhorn system using Longhorn UI. Or with the kubectl command.

Prerequisite

  • A running Longhorn cluster for Longhorn to roll out the resources in the system backup bundle.

  • Set up the Nodes and disk tags for StorageClass.

  • Have a Longhorn system backup.

    See Backup Longhorn System - Create Longhorn System Backup for instructions.

  • Have volume BackingImages available in the cluster.

    In case of the BackingImage absence, Longhorn will skip the restoration for that Volume and its PersistentVolume and PersistentVolumeClaim.

  • All existing Volumes are detached.

Using Longhorn UI

  1. Go to the System Backup page in the Setting.
  2. Select a system backup to restore.
  3. Click Restore in the Operation drop-down menu.
  4. Give a Name for the system restore.
  5. The system restore starts and show the Completed state when done.

Using kubectl Command

  1. Find the Longhorn SystemBackup to restore.

    1. > kubectl -n longhorn-system get systembackup
    2. NAME     VERSION   STATE   CREATED
    3. demo     v1.4.0    Ready   2022-11-24T04:23:24Z
    4. demo-2   v1.4.0    Ready   2022-11-24T05:00:59Z

  2. Execute kubectl create to create a Longhorn SystemRestore of the SystemBackup.

    1. apiVersion: longhorn.io/v1beta2
    2. kind: SystemRestore
    3. metadata:
    4.   name: restore-demo
    5.   namespace: longhorn-system
    6. spec:
    7.   systemBackup: demo

  3. The system restore starts.

  4. The SystemRestore change to state Completed when done.

    1. > kubectl -n longhorn-system get systemrestore
    2. NAME           STATE       AGE
    3. restore-demo   Completed   59s

Delete Longhorn System Restore

Warning: Deleting the SystemRestore also deletes the associated job and will abort the remaining resource rollouts. You can Restart the Longhorn System Restore to roll out the remaining resources.

You can abort or remove a completed Longhorn system restore using Longhorn UI. Or with the kubectl command.

Using Longhorn UI

  1. Go to the System Backup page in the Setting.
  2. Delete a single system restore in the Operation drop-down menu next to the system restore. Or delete in batch with the Delete button.

Using kubectl Command

  1. Execute kubectl delete to delete a Longhorn SystemRestore.

    1. > kubectl -n longhorn-system get systemrestore
    2. NAME           STATE       AGE
    3. restore-demo   Completed   2m37s
    5. > kubectl -n longhorn-system delete systemrestore/restore-demo
    6. systemrestore.longhorn.io "restore-demo" deleted

Restart Longhorn System Restore

  1. Delete Longhorn System Restore that is in progress.
  2. Create Longhorn System Restore.

Configurable Settings

Some settings are excluded as configurable before the Longhorn system restore.

Troubleshoot

System Restore Hangs

  1. Check the longhorn-system-rollout Pod log for any errors.
  1. > kubectl -n longhorn-system logs --selector=job-name=longhorn-system-rollout-<SYSTEM-RESTORE-NAME>
  1. Resolve if the issue is identifiable, ex: remove the problematic restoring resource.
  2. Restart the Longhorn system restore.

History

Original Feature Request

Available since v1.4.0