Restore Longhorn System

• What does the Longhorn system restore rollout to the cluster
• What are the limitations
  • Restore Path
• How to restore from Longhorn system backup
  • Prerequisite
  • Using Longhorn UI
  • Using kubectl command
• How to delete Longhorn system restore
  • Using Longhorn UI
  • Using kubectl command
• How to restart Longhorn System Restore
• What settings are configurable
• How to troubleshoot
• History

Longhorn System Restore Rollouts

• Longhorn restores the resource from the Longhorn System Backup Bundle.
• Longhorn does not restore existing Volumes and their associated PersistentVolume and PersistentVolumeClaim.
• Longhorn automatically restores a Volume from its latest backup.
• To prevent overwriting eligible settings, Longhorn does not restore the ConfigMap/longhorn-default-setting.
• Longhorn does not restore configurable settings.

Limitations

Restore Path

Longhorn does not support cross-major/minor version system restore except for upgrade failures, ex: 1.4.x -> 1.5.

Create Longhorn System Restore

You can restore the Longhorn system using Longhorn UI. Or with the kubectl command.

Prerequisite

• A running Longhorn cluster for Longhorn to roll out the resources in the system backup bundle.

• Set up the Nodes and disk tags for StorageClass.

• Have a Longhorn system backup.

  See Backup Longhorn System - Create Longhorn System Backup for instructions.

• All existing Volumes are detached.

Using Longhorn UI

1. Go to the System Backup page in the Setting.
2. Select a system backup to restore.
3. Click Restore in the Operation drop-down menu.
4. Give a Name for the system restore.
5. The system restore starts and show the Completed state when done.

Using kubectl Command

1. Find the Longhorn SystemBackup to restore.

   > kubectl -n longhorn-system get systembackup
   NAME     VERSION   STATE   CREATED
   demo     v1.4.0    Ready   2022-11-24T04:23:24Z
   demo-2   v1.4.0    Ready   2022-11-24T05:00:59Z

2. Execute kubectl create to create a Longhorn SystemRestore of the SystemBackup.

   apiVersion: longhorn.io/v1beta2
   kind: SystemRestore
   metadata:
     name: restore-demo
     namespace: longhorn-system
   spec:
     systemBackup: demo

3. The system restore starts.

4. The SystemRestore change to state Completed when done.

   > kubectl -n longhorn-system get systemrestore
   NAME           STATE       AGE
   restore-demo   Completed   59s

Delete Longhorn System Restore

Warning: Deleting the SystemRestore also deletes the associated job and will abort the remaining resource rollouts. You can Restart the Longhorn System Restore to roll out the remaining resources.

You can abort or remove a completed Longhorn system restore using Longhorn UI. Or with the kubectl command.

Using Longhorn UI

1. Go to the System Backup page in the Setting.
2. Delete a single system restore in the Operation drop-down menu next to the system restore. Or delete in batch with the Delete button.

Using kubectl Command

1. Execute kubectl delete to delete a Longhorn SystemRestore.

   > kubectl -n longhorn-system get systemrestore
   NAME           STATE       AGE
   restore-demo   Completed   2m37s
   > kubectl -n longhorn-system delete systemrestore/restore-demo
   systemrestore.longhorn.io "restore-demo" deleted

Restart Longhorn System Restore

1. Delete Longhorn System Restore that is in progress.
2. Create Longhorn System Restore.

Configurable Settings

Some settings are excluded as configurable before the Longhorn system restore.

• Concurrent volume backup restore per node limit
• Concurrent replica rebuild per node limit
• Backup Target
• Backup Target Credential Secret

Troubleshoot

System Restore Hangs

1. Check the longhorn-system-rollout Pod log for any errors.

> kubectl -n longhorn-system logs --selector=job-name=longhorn-system-rollout-<SYSTEM-RESTORE-NAME>

1. Resolve if the issue is identifiable, ex: remove the problematic restoring resource.
2. Restart the Longhorn system restore.

History

Original Feature Request

Available since v1.4.0