Upgrading Longhorn Manager

Upgrading from v1.7.x (< v1.7.2) or v1.6.x

We only support upgrading to v1.7.2 from v1.7.x or v1.6.x. For other versions, please upgrade to v1.7.x or v1.6.x first.

Engine live upgrade is supported from v1.6.x to v1.7.2.

For airgap upgrades when Longhorn is installed as a Rancher app, you will need to modify the image names and remove the registry URL part.

For example, the image registry.example.com/longhorn/longhorn-manager:v1.7.2 is changed to longhorn/longhorn-manager:v1.7.2 in Longhorn images section. For more information, see the air gap installation steps here.

Preparing for the Upgrade

If Longhorn was installed using a Helm Chart, or if it was installed as Rancher catalog app, check to make sure the parameters in the default StorageClass weren’t changed. Changing the default StorageClass’s parameter might result in a chart upgrade failure. if you want to reconfigure the parameters in the StorageClass, you can copy the default StorageClass’s configuration to create another StorageClass.

  1. The current default StorageClass has the following parameters:
  3.     parameters:
  4.       numberOfReplicas: <user specified replica count, 3 by default>
  5.       staleReplicaTimeout: "30"
  6.       fromBackup: ""
  7.       baseImage: ""

Upgrade

Prerequisite: Always back up volumes before upgrading. If anything goes wrong, you can restore the volume using the backup.

Upgrade as a Rancher Catalog App

To upgrade the Longhorn App, make sure which Rancher UI the existing Longhorn App was installed with. There are two Rancher UIs, one is the Cluster Manager (old UI), and the other one is the Cluster Explorer (new UI). The Longhorn App in different UIs considered as two different applications by Rancher. They cannot upgrade to each other. If you installed Longhorn in the Cluster Manager, you need to use the Cluster Manager to upgrade Longhorn to a newer version, and vice versa for the Cluster Explorer.

Note: Because the Cluster Manager (old UI) is being deprecated, we provided the instruction to migrate the existing Longhorn installation to the Longhorn chart in the Cluster Explorer (new UI) here

Different Rancher UIs screenshots.

  • The Cluster Manager (old UI)

    Upgrading Longhorn Manager - 图1

  • The Cluster Explorer (new UI)

    Upgrading Longhorn Manager - 图2

On Kubernetes clusters managed by Rancher 2.1 or newer, the steps to upgrade the catalog app longhorn-system are the similar to the installation steps.

Upgrade with Kubectl

To upgrade with kubectl, run this command:

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml

Upgrade with Helm

To upgrade with Helm, run this command:

  1. helm upgrade longhorn longhorn/longhorn --namespace longhorn-system --version 1.7.2

Upgrade with Helm Controller

Update the value of spec.version in the HelmChart YAML file:

  1. spec:
  2.   version: v1.7.2 # Replace with the Longhorn version you'd like to upgrade to
  3.   chart: longhorn
  4.   repo: https://charts.longhorn.io
  5.   failurePolicy: abort

Alternatively, if using the spec.chartContent key, create a patch file with

  1. spec:
  2.   chartContent: <base64 content> # tar cz of longhorn charts directory for release | base64 -w 0

and then apply it with

  1. kubectl  patch helmchart longhorn -n <namespace> --type merge --patch-file <name of patch file>

IMPORTANT! In both cases, ensure that spec.failurePolicy is set to “abort”. The only other value is the default: “reinstall”, which performs an uninstall of Longhorn if the pre-upgrade check or the upgrade fails. With “abort”, it retries periodically, giving the user a chance to fix the problem.

Upgrade with Fleet

Update the value of helm.version in the fleet YAML file of your GitOps repository.

  1. helm:
  2.   repo: https://charts.longhorn.io
  3.   chart: longhorn
  4.   version: v1.7.2 # Replace with the Longhorn version you'd like to upgrade to
  5.   releaseName: longhorn

Upgrade with Flux

Update the value of spec.chart.spec.version in the HelmRelease YAML file of your GitOps repository.

  1. spec:
  2.   chart:
  3.     spec:
  4.       chart: longhorn
  5.       reconcileStrategy: ChartVersion
  6.       sourceRef:
  7.         kind: HelmRepository
  8.         name: longhorn
  9.       version: v1.7.2 # Replace with the Longhorn version you'd like to upgrade to

Upgrade with Argo CD

Update the value of targetRevision in the Application YAML file of your GitOps repository.

  1. spec:
  2.   project: default
  3.   sources:
  4.     - chart: longhorn
  5.       repoURL: https://charts.longhorn.io
  6.       targetRevision: v1.7.2 # Replace with the Longhorn version you'd like to upgrade to

Then wait for all the pods to become running and Longhorn UI working. e.g.:

  1. $ kubectl -n longhorn-system get pod
  2. NAME                                                  READY   STATUS    RESTARTS      AGE
  3. engine-image-ei-4dbdb778-nw88l                        1/1     Running   0             4m29s
  4. longhorn-ui-b7c844b49-jn5g6                           1/1     Running   0             75s
  5. longhorn-manager-z2p8h                                1/1     Running   0             71s
  6. instance-manager-b34d5db1fe1e2d52bcfb308be3166cfc     1/1     Running   0             65s
  7. longhorn-driver-deployer-6bd59c9f76-jp6pg             1/1     Running   0             75s
  8. engine-image-ei-df38d2e5-zccq5                        1/1     Running   0             65s
  9. csi-snapshotter-588457fcdf-h2lgc                      1/1     Running   0             30s
  10. csi-resizer-6d8cf5f99f-8v4sp                          1/1     Running   1 (30s ago)   37s
  11. csi-snapshotter-588457fcdf-6pgf4                      1/1     Running   0             30s
  12. csi-provisioner-869bdc4b79-7ddwd                      1/1     Running   1 (30s ago)   44s
  13. csi-snapshotter-588457fcdf-p4kkn                      1/1     Running   0             30s
  14. csi-attacher-7bf4b7f996-mfbdn                         1/1     Running   1 (30s ago)   50s
  15. csi-provisioner-869bdc4b79-4dc7n                      1/1     Running   1 (30s ago)   43s
  16. csi-resizer-6d8cf5f99f-vnspd                          1/1     Running   1 (30s ago)   37s
  17. csi-attacher-7bf4b7f996-hrs7w                         1/1     Running   1 (30s ago)   50s
  18. csi-attacher-7bf4b7f996-rt2s9                         1/1     Running   1 (30s ago)   50s
  19. csi-resizer-6d8cf5f99f-7vv89                          1/1     Running   1 (30s ago)   37s
  20. csi-provisioner-869bdc4b79-sn6zr                      1/1     Running   1 (30s ago)   43s
  21. longhorn-csi-plugin-b2zzj                             2/2     Running   0             24s

Next, upgrade Longhorn engine.

Upgrading from Unsupported Versions

We only support upgrading to v1.7.2 from from v1.7.x or v1.6.x. For other versions, please upgrade to v1.7.x or v1.6.x first.

If you attempt to upgrade from an unsupported version, the upgrade will fail. When encountering an upgrade failure, please consider the following scenarios to recover the state based on different upgrade methods.

Upgrade with Kubectl

When you upgrade with kubectl by running this command:

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml

Longhorn will block the upgrade process and provide the failure reason in the logs of the longhorn-manager pod. During the upgrade failure, the user’s Longhorn system should remain intact without any impacts except longhorn-manager daemon set.

To recover, you need to apply the manifest of the previously installed version using the following command:

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/[previous installed version]/deploy/longhorn.yaml

Besides, users might need to delete new components introduced by the new version manually.

Upgrade with Helm or Rancher App Marketplace

To prevent any impact caused by failed upgrades from unsupported versions, Longhorn will automatically initiate a new job (pre-upgrade) to verify if the upgrade path is supported before upgrading when upgrading through Helm or Rancher App Marketplace.

The pre-upgrade job will block the upgrade process and provide the failure reason in the logs of the pod. During the upgrade failure, the user’s Longhorn system should remain intact without any impacts.

To recover, you need to run the below commands to rollback to the previously installed revision:

  1. # get previous installed Longhorn REVISION
  2. helm history longhorn
  3. helm rollback longhorn [REVISION]
  5. # or
  6. helm upgrade longhorn longhorn/longhorn --namespace longhorn-system --version [previous installed version]

To recover, you need to upgrade to the previously installed revision at Rancher App Marketplace again.

TroubleShooting

  1. Error: "longhorn" is invalid: provisioner: Forbidden: updates to provisioner are forbidden.

  • This means there are some modifications applied to the default storageClass and you need to clean up the old one before upgrade.

  • To clean up the deprecated StorageClass, run this command:

    1. kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/examples/storageclass.yaml