Requirements:

  • Deploy Longhorn Components images to your own registry.
  • Deploy Kuberntes CSI driver components images to your own registry.

Note:

  • CSI driver components images names and tags can be found here.
  • We recommend using a short registry URL due to a Kubernetes limitation on the length of pod metadata labels. For more information, refer to this section

Using manifest file.

  1. Get Longhorn Deployment manifest file

    wget https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

  2. Create Longhorn namespace

    kubectl create namespace longhorn-system

  3. If private registry require authentication, Create docker-registry secret in longhorn-system namespace:

    kubectl -n longhorn-system create secret docker-registry <SECRET_NAME> --docker-server=<REGISTRY_URL> --docker-username=<REGISTRY_USER> --docker-password=<REGISTRY_PASSWORD>

    • Add your secret name to longhorn-default-setting ConfigMap
    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: longhorn-default-setting
    5.   namespace: longhorn-system
    6. data:
    7.   default-setting.yaml: |-
    8.     backup-target:
    9.     backup-target-credential-secret:
    10.     create-default-disk-labeled-nodes:
    11.     default-data-path:
    12.     replica-soft-anti-affinity:
    13.     storage-over-provisioning-percentage:
    14.     storage-minimal-available-percentage:
    15.     upgrade-checker:
    16.     default-replica-count:
    17.     guaranteed-engine-cpu:
    18.     default-longhorn-static-storage-class:
    19.     backupstore-poll-interval:
    20.     taint-toleration:
    21.     registry-secret:  <SECRET_NAME>

    • Add your secret name SECRET_NAME to imagePullSecrets.name in the following resources

      • longhorn-driver-deployer Deployment
      • longhorn-manager DaemonSet
      • longhorn-ui Deployment

      Example:

      1. apiVersion: apps/v1
      2. kind: Deployment
      3. metadata:
      4.   labels:
      5.     app: longhorn-ui
      6.   name: longhorn-ui
      7.   namespace: longhorn-system
      8. spec:
      9.   replicas: 1
      10.   selector:
      11.     matchLabels:
      12.       app: longhorn-ui
      13.   template:
      14.     metadata:
      15.       labels:
      16.         app: longhorn-ui
      17.     spec:
      18.       containers:
      19.       - name: longhorn-ui
      20.         image: longhornio/longhorn-ui:v0.8.0
      21.         ports:
      22.         - containerPort: 8000
      23.         env:
      24.           - name: LONGHORN_MANAGER_IP
      25.             value: "http://longhorn-backend:9500"
      26.       imagePullSecrets:
      27.       - name: <SECRET_NAME>                          ## Add SECRET_NAME here
      28.       serviceAccountName: longhorn-service-account

  4. Apply the following modifications to the manifest file

    • Modify Kubernetes CSI driver components environment variables in longhorn-driver-deployer Deployment point to your private registry images

      • CSI_ATTACHER_IMAGE
      • CSI_PROVISIONER_IMAGE
      • CSI_NODE_DRIVER_REGISTRAR_IMAGE
      • CSI_RESIZER_IMAGE
      1. - name: CSI_ATTACHER_IMAGE
      2.   value: <REGISTRY_URL>/csi-attacher:<CSI_ATTACHER_IMAGE_TAG>
      3. - name: CSI_PROVISIONER_IMAGE
      4.   value: <REGISTRY_URL>/csi-provisioner:<CSI_PROVISIONER_IMAGE_TAG>
      5. - name: CSI_NODE_DRIVER_REGISTRAR_IMAGE
      6.   value: <REGISTRY_URL>/csi-node-driver-registrar:<CSI_NODE_DRIVER_REGISTRAR_IMAGE_TAG>
      7. - name: CSI_RESIZER_IMAGE
      8.   value: <REGISTRY_URL>/csi-resizer:<CSI_RESIZER_IMAGE_TAG>

    • Modify Longhorn images to point to your private registry images

      • longhornio/longhorn-manager

        image: <REGISTRY_URL>/longhorn-manager:<LONGHORN_MANAGER_IMAGE_TAG>

      • longhornio/longhorn-engine

        image: <REGISTRY_URL>/longhorn-engine:<LONGHORN_ENGINE_IMAGE_TAG>

      • longhornio/longhorn-instance-manager

        image: <REGISTRY_URL>/longhorn-instance-manager:<LONGHORN_INSTANCE_MANAGER_IMAGE_TAG>

      • longhornio/longhorn-ui

        image: <REGISTRY_URL>/longhorn-ui:<LONGHORN_UI_IMAGE_TAG>

  1. Example:
  3. ```
  4. apiVersion: apps/v1
  5. kind: Deployment
  6. metadata:
  7.   labels:
  8.     app: longhorn-ui
  9.   name: longhorn-ui
  10.   namespace: longhorn-system
  11. spec:
  12.   replicas: 1
  13.   selector:
  14.     matchLabels:
  15.       app: longhorn-ui
  16.   template:
  17.     metadata:
  18.       labels:
  19.         app: longhorn-ui
  20.     spec:
  21.       containers:
  22.       - name: longhorn-ui
  23.         image: <REGISTRY_URL>/longhorn-ui:<LONGHORN_UI_IMAGE_TAG>   ## Add image name and tag here
  24.         ports:
  25.         - containerPort: 8000
  26.         env:
  27.           - name: LONGHORN_MANAGER_IP
  28.             value: "http://longhorn-backend:9500"
  29.       imagePullSecrets:
  30.       - name: <SECRET_NAME>
  31.       serviceAccountName: longhorn-service-account
  32. ```
  1. Deploy Longhorn using modified manifest file kubectl apply -f longhorn.yaml

Using Helm Chart

  1. Clone longhorn repo

    git clone https://github.com/longhorn/longhorn.git

  2. In chart/values.yaml

  • Specify Longhorn images:

    1. image:
    2.   longhorn:
    3.     engine: <REGISTRY_URL>/longhorn-engine
    4.     engineTag: <LONGHORN_ENGINE_IMAGE_TAG>
    5.     manager: <REGISTRY_URL>/longhorn-manager
    6.     managerTag: LONGHORN_MANAGER_IMAGE_TAG<>
    7.     ui: <REGISTRY_URL>/longhorn-ui
    8.     uiTag: <LONGHORN_UI_IMAGE_TAG>
    9.     instanceManager: <REGISTRY_URL>/longhorn-instance-manager
    10.     instanceManagerTag: <LONGHORN_INSTANCE_MANAGER_IMAGE_TAG>

  • Specify CSI Driver components images:

    1. csi:
    2.   attacherImage: <REGISTRY_URL>/csi-attacher:<CSI_ATTACHER_IMAGE_TAG>
    3.   provisionerImage: <REGISTRY_URL>/csi-provisioner:<CSI_PROVISIONER_IMAGE_TAG>
    4.   driverRegistrarImage: <REGISTRY_URL>/csi-node-driver-registrar:<CSI_NODE_DRIVER_REGISTRAR_IMAGE_TAG>
    5.   resizerImage: <REGISTRY_URL>/csi-resizer:<CSI_RESIZER_IMAGE_TAG>

  • Specify registry secret Name, URL, and Credentials:

    1. defaultSettings:
    2.   registrySecret: <SECRET_NAME>
    4. privateRegistry: 
    5.     registryUrl: <REGISTRY_URL>
    6.     registryUser: <REGISTRY_USER>
    7.     registryPasswd: <REGISTRY_PASSWORD>

  1. Install Longhorn

    • Helm2

      helm install ./chart --name longhorn --namespace longhorn-system

    • Helm3

      kubectl create namespace longhorn-system

      helm install longhorn ./chart --namespace longhorn-system

Using Rancher App

  • In Longhorn Images Settings section specify

    • Longhorn Manager Image Name e.g. <REGISTRY_URL>/longhorn-manager
    • Longhorn Manager Image Tag
    • Longhorn Engine Image Name e.g. <REGISTRY_URL>/longhorn-engine
    • Longhorn Engine Image Tag
    • Longhorn UI Image Name e.g. <REGISTRY_URL>/longhorn-ui
    • Longhorn UI Image Tag
    • Longhorn Instance Manager Image Name e.g. <REGISTRY_URL>/longhorn-instance-manager
    • Longhorn Instance Manager Image Tag

  • In Longhorn CSI Driver Setting section specify

    • Longhorn CSI Attacher Image e.g. <REGISTRY_URL>/csi-attacher:<CSI_ATTACHER_IMAGE_TAG>
    • Longhorn CSI Provisioner Image e.g. <REGISTRY_URL>/csi-provisioner:<CSI_PROVISIONER_IMAGE_TAG>
    • Longhorn CSI Driver Registrar Image e.g. <REGISTRY_URL>/csi-node-driver-registrar:<CSI_NODE_DRIVER_REGISTRAR_IMAGE_TAG>
    • Longhorn CSI Driver Resizer Image e.g. <REGISTRY_URL>/csi-resizer:<CSI_RESIZER_IMAGE_TAG>

  • In Longhorn Default Settings secton specify

    • Private registry secret

  • In Private Registry Settings secton specify

    • Private registry URL
    • Private registry user
    • Private registry password

Troubleshooting

For Helm/Rancher installation, if user forgot to submit a secret to authenticate to private registry, longhorn-manager DaemonSet will fail to create.

  1. Create the Kubernetes secret

    kubectl -n longhorn-system create secret docker-registry <SECRET_NAME> --docker-server=<REGISTRY_URL> --docker-username=<REGISTRY_USER> --docker-password=<REGISTRY_PASSWORD>

  2. Create registry-secret setting object manually.

    1. apiVersion: longhorn.io/v1beta1
    2. kind: Setting
    3. metadata:
    4.   name: registry-secret
    5.   namespace: longhorn-system
    6. value: <SECRET_NAME>

    kubectl apply -f registry-secret.yml

  3. Delete Longhorn and re-install it again.

    • Helm2

      helm uninstall ./chart --name longhorn --namespace longhorn-system

      helm install ./chart --name longhorn --namespace longhorn-system

    • Helm3

      helm uninstall longhorn ./chart --namespace longhorn-system

      helm install longhorn ./chart --namespace longhorn-system

longhorn-driver-deployer error: Node is not support mount propagation

If longhorn-instance-manager image name is more than 63 characters long, it will fail to deploy, and longhorn-driver-deployer pod will be in CrashLoopBackOff.

Checking Longhorn driver deployer logs will report the following:

  1. time="2020-03-13T22:49:22Z" level=warning msg="Got an error when checking MountPropagation with node status, Node XXX is not support mount propagation"
  2. time="2020-03-13T22:49:22Z" level=fatal msg="Error deploying driver: CSI cannot be deployed because MountPropagation is not set: Node <NODE_NAME> is not support mount propagation"

Issue can be conformed by checking Longhorn manager log, you should be able to see the following logs:

“Dropping Longhorn node longhorn-system/NODE_NAME out of the queue: fail to sync node for longhorn-system/NODE_NAME: InstanceManager.longhorn.io “instance-manager-e-605e9473” is invalid: metadata.labels: Invalid value: “PRIVATE_REGISTRY_URL-PREFIX-longhorn-instance-manager-v1_20200301”: must be no more than 63 characters”

Longhorn instance manager: metadata.labels must be no more than 63 characters

Using a long registry URL may cause Longhorn installation error

Longhorn manager would report errors in the log when this happened:

  1. "instance-manager-e-xxxxxxxx" is invalid: metadata.labels: Invalid value: "<PRIVATE_REGISTRY_URL>-longhornio-longhorn-instance-manager-v1_20200301": must be no more than 63 characters

Longhorn instance manager pods have labels with key longhorn.io/instance-manager-image and value REGISTRY_URL-USER-IMAGE_NAME-TAG

e.g

  1. metadata:
  2.   labels:
  3.     longhorn.io/component: instance-manager
  4.     longhorn.io/instance-manager-image: <PRIVATE_REGISTRY_URL>-longhornio-longhorn-instance-manager-v1_20200301
  5.     longhorn.io/instance-manager-type: engine
  6.     longhorn.io/node: <NODE_NAME>
  7.   name: instance-manager-e-XXXXXXXX

it’s known Kubernetes limitaion that label value should be no more than 63 characters here

Recommendation:

It’s highly recommended not to manipulate image tags, especially instance manager image tags such as v1_20200301, because we intentionally use the date to avoid associating it with a Longhorn version.

e.g

  • Longhorn components images

    • longhorn-instance-manager: hub.example.com/lh/ins-mgr:v1_20200301
    • longhorn-manager: hub.example.com/lh/mgr:v0.8.1
    • longhorn-engine: hub.example.com/lh/eng:v0.8.1
    • longhorn-ui: hub.examples.com/lh/ui:v0.8.1

  • Kubernetes CSI images

    • CSI Attacher: hub.example.com/csi/attacher:v2.0.0
    • CSI Provisioner: hub.example.com/csi/provisioner:v1.4.0
    • CSI Node Driver Registrar: hub.example.com/csi/node-driver-reg:v1.2.0
    • CSI Resizer: hub.example.com/csi/resizer:v0.3.0