Upgrading Longhorn Manager

Upgrading from v1.3.x

We only support upgrading to v1.4.0 from v1.3.x. For other versions, please upgrade to v1.3.x first.

Engine live upgrade is supported from v1.3.x to v1.4.0.

For airgap upgrades when Longhorn is installed as a Rancher app, you will need to modify the image names and remove the registry URL part.

For example, the image registry.example.com/longhorn/longhorn-manager:v1.4.0 is changed to longhorn/longhorn-manager:v1.4.0 in Longhorn images section. For more information, see the air gap installation steps here.

Preparing for the Upgrade

If Longhorn was installed using a Helm Chart, or if it was installed as Rancher catalog app, check to make sure the parameters in the default StorageClass weren’t changed. Changing the default StorageClass’s parameter might result in a chart upgrade failure. if you want to reconfigure the parameters in the StorageClass, you can copy the default StorageClass’s configuration to create another StorageClass.

  1. The current default StorageClass has the following parameters:
  3.     parameters:
  4.       numberOfReplicas: <user specified replica count, 3 by default>
  5.       staleReplicaTimeout: "30"
  6.       fromBackup: ""
  7.       baseImage: ""

Upgrade

Prerequisite: Always back up volumes before upgrading. If anything goes wrong, you can restore the volume using the backup.

Upgrade as a Rancher Catalog App

To upgrade the Longhorn App, make sure which Rancher UI the existing Longhorn App was installed with. There are two Rancher UIs, one is the Cluster Manager (old UI), and the other one is the Cluster Explorer (new UI). The Longhorn App in different UIs considered as two different applications by Rancher. They cannot upgrade to each other. If you installed Longhorn in the Cluster Manager, you need to use the Cluster Manager to upgrade Longhorn to a newer version, and vice versa for the Cluster Explorer.

Note: Because the Cluster Manager (old UI) is being deprecated, we provided the instruction to migrate the existing Longhorn installation to the Longhorn chart in the Cluster Explorer (new UI) here

Different Rancher UIs screenshots.

  • The Cluster Manager (old UI)

    Upgrading Longhorn Manager - 图1

  • The Cluster Explorer (new UI)

    Upgrading Longhorn Manager - 图2

Upgrade with Kubectl

To upgrade with kubectl, run this command:

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.4.0/deploy/longhorn.yaml

Upgrade with Helm

To upgrade with Helm, run this command:

  1. helm upgrade longhorn longhorn/longhorn --namespace longhorn-system --version 1.4.0

On Kubernetes clusters managed by Rancher 2.1 or newer, the steps to upgrade the catalog app longhorn-system are the similar to the installation steps.

Then wait for all the pods to become running and Longhorn UI working. e.g.:

  1. $ kubectl -n longhorn-system get pod
  2. NAME                                           READY   STATUS    RESTARTS      AGE
  3. engine-image-ei-4dbdb778-nw88l                 1/1     Running   0             4m29s
  4. longhorn-conversion-webhook-5dc58756b6-8vf8g   1/1     Running   0             75s
  5. longhorn-conversion-webhook-5dc58756b6-jqq6n   1/1     Running   0             75s
  6. longhorn-ui-b7c844b49-jn5g6                    1/1     Running   0             75s
  7. longhorn-admission-webhook-8b7f74576-898c4     1/1     Running   0             75s
  8. longhorn-admission-webhook-8b7f74576-t7jqk     1/1     Running   0             75s
  9. longhorn-manager-z2p8h                         1/1     Running   0             71s
  10. instance-manager-r-de8337e2                    1/1     Running   0             65s
  11. instance-manager-e-c812d56c                    1/1     Running   0             65s
  12. longhorn-driver-deployer-6bd59c9f76-jp6pg      1/1     Running   0             75s
  13. engine-image-ei-df38d2e5-zccq5                 1/1     Running   0             65s
  14. csi-snapshotter-588457fcdf-h2lgc               1/1     Running   0             30s
  15. csi-resizer-6d8cf5f99f-8v4sp                   1/1     Running   1 (30s ago)   37s
  16. csi-snapshotter-588457fcdf-6pgf4               1/1     Running   0             30s
  17. csi-provisioner-869bdc4b79-7ddwd               1/1     Running   1 (30s ago)   44s
  18. csi-snapshotter-588457fcdf-p4kkn               1/1     Running   0             30s
  19. csi-attacher-7bf4b7f996-mfbdn                  1/1     Running   1 (30s ago)   50s
  20. csi-provisioner-869bdc4b79-4dc7n               1/1     Running   1 (30s ago)   43s
  21. csi-resizer-6d8cf5f99f-vnspd                   1/1     Running   1 (30s ago)   37s
  22. csi-attacher-7bf4b7f996-hrs7w                  1/1     Running   1 (30s ago)   50s
  23. csi-attacher-7bf4b7f996-rt2s9                  1/1     Running   1 (30s ago)   50s
  24. csi-resizer-6d8cf5f99f-7vv89                   1/1     Running   1 (30s ago)   37s
  25. csi-provisioner-869bdc4b79-sn6zr               1/1     Running   1 (30s ago)   43s
  26. longhorn-csi-plugin-b2zzj                      2/2     Running   0             24s

Next, upgrade Longhorn engine.

Post upgrade

To avoid crashing existing volumes, as well as switch from the deprecated setting Guaranteed Engine CPU to the new instance manager CPU reservation mechanism, Longhorn will automatically set Engine Manager CPU Request and Replica Manager CPU Request from each node based on the deprecated setting value during upgrade. Then, the new global instance manager CPU settings Guaranteed Engine Manager CPU and Guaranteed Replica Manager CPU won’t take effect. You may need to check the new mechanism and the setting descriptions to see if you need any adjustments.

TroubleShooting

  1. Error: "longhorn" is invalid: provisioner: Forbidden: updates to provisioner are forbidden.

  • This means there are some modifications applied to the default storageClass and you need to clean up the old one before upgrade.

  • To clean up the deprecated StorageClass, run this command:

    1. kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/v1.4.0/examples/storageclass.yaml

  1. Error: ...proto: duplicate proto type registered: VersionResponse panic: Unrecognized command: conversion-webhook ... in the longhorn-conversion-webhook pods in CrashLoopBackOff state.

    • Check if the longhorn-conversion-webhook image tag is v1.4.0 by

      1. kubectl -n longhorn-system get deployments longhorn-conversion-webhook -o yaml | grep image

    • It indicates Helm uses the previously configured image tag value if the image tag is different than expected, i.e. v1.4.0. Then, you need to reset the values by --reset-values.

      • You can simply upgrade Longhorn without updating your customized default settings.

        1. helm upgrade longhorn longhorn/longhorn -n longhorn-system --reset-values

      • You can customize the default settings in as described in Customize Default Settings and then run

        1. helm upgrade longhorn longhorn/longhorn -n longhorn-system --values ./values.yaml --reset-values