Quick Installation

Install Longhorn on Kubernetes

Note: This quick installation guide uses some configurations which are not for production usage. Please see Best Practices for how to configure Longhorn for production usage.

Longhorn can be installed on a Kubernetes cluster in several ways:

To install Longhorn in an air gapped environment, refer to this section.

For information on customizing Longhorn’s default settings, refer to this section.

For information on deploying Longhorn on specific nodes and rejecting general workloads for those nodes, refer to the section on taints and tolerations.

Installation Requirements

Each node in the Kubernetes cluster where Longhorn is installed must fulfill the following requirements:

  • A container runtime compatible with Kubernetes (Docker v1.13+, containerd v1.3.7+, etc.)
  • Kubernetes >= v1.21
  • open-iscsi is installed, and the iscsid daemon is running on all the nodes. This is necessary, since Longhorn relies on iscsiadm on the host to provide persistent volumes to Kubernetes. For help installing open-iscsi, refer to this section.
  • RWX support requires that each node has a NFSv4 client installed.
  • The host filesystem supports the file extents feature to store the data. Currently we support:
    • ext4
    • XFS
  • bash, curl, findmnt, grep, awk, blkid, lsblk must be installed.
  • Mount propagation must be enabled.

The Longhorn workloads must be able to run as root in order for Longhorn to be deployed and operated properly.

This script can be used to check the Longhorn environment for potential issues.

For the minimum recommended hardware, refer to the best practices guide.

OS/Distro Specific Configuration

  • Google Kubernetes Engine (GKE) requires some additional setup for Longhorn to function properly. If you’re a GKE user, refer to this section for details.
  • K3s clusters require some extra setup. Refer to this section
  • RKE clusters with CoreOS need this configuration.

Using the Environment Check Script

We’ve written a script to help you gather enough information about the factors.

Note jq maybe required to be installed locally prior to running env check script.

To run script:

  1. curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.4.1/scripts/environment_check.sh | bash

Example result:

  1. [INFO]  Required dependencies are installed.
  2. [INFO]  Waiting for longhorn-environment-check pods to become ready (0/3)...
  3. [INFO]  All longhorn-environment-check pods are ready (3/3).
  4. [INFO]  Required packages are installed.
  5. [INFO]  Cleaning up longhorn-environment-check pods...
  6. [INFO]  Cleanup completed.

Pod Security Policy

Starting with v1.0.2, Longhorn is shipped with a default Pod Security Policy that will give Longhorn the necessary privileges to be able to run properly.

No special configuration is needed for Longhorn to work properly on clusters with Pod Security Policy enabled.

Notes on Mount Propagation

If your Kubernetes cluster was provisioned by Rancher v2.0.7+ or later, the MountPropagation feature is enabled by default.

If MountPropagation is disabled, Base Image feature will be disabled.

Root and Privileged Permission

Longhorn components require root access with privileged permissions to achieve volume operations and management, because Longhorn relies on system resources on the host across different namespaces, for example, Longhorn uses nsenter to understand block devices’ usage or encrypt/decrypt volumes on the host.

Below are the directories Longhorn components requiring access with root and privileged permissions :

  • Longhorn Manager
    • /dev: Block devices created by Longhorn are under the /dev path.
    • /proc: Find the recognized host process like container runtime, then use nsenter to access the mounts on the host to understand disks usage.
    • /var/lib/longhorn: The default path for storing volume data on a host.
  • Longhorn Engine Image
    • /var/lib/longhorn/engine-binaries: The default path for storing the Longhorn engine binaries.
  • Longhorn Instance Manager (Engine)
    • /dev: Block devices created by Longhorn are under the /dev path.
    • /proc: Find the recognized host process like container runtime, then use nsenter to manage iSCSI targets and initiators, also some file system operations like sync before snapshotting.
    • /var/lib/longhorn/engine-binaries: The default path for storing the Longhorn engine binaries.
  • Longhorn Instance Manager (Replica)
    • /: Access any data path on this node and access Longhorn engine binaries.
  • Longhorn Share Manager
    • /dev: Block devices created by Longhorn are under the /dev path.
    • /lib/modules: Kernel modules required by cryptsetup for volume encryption.
    • /proc: Find the recognized host process like container runtime, then use nsenter for volume encryption.
    • /sys: Support volume encryption by cryptsetup.
  • Longhorn CSI Plugin
    • /: For host checks via the NFS customer mounter (deprecated). Note that, this will be removed in the future release.
    • /dev: Block devices created by Longhorn are under the /dev path.
    • /lib/modules: Kernel modules required by Longhorn CSI plugin.
    • /sys: Support volume encryption by cryptsetup.
    • /var/lib/kubelet/plugins/kubernetes.io/csi: The path where the Longhorn CSI plugin creates the staging path (via NodeStageVolume) of a block device. The staging path will be bind-mounted to the target path /var/lib/kubelet/pods (via NodePublishVolume) for support single volume could be mounted to multiple Pods.
    • /var/lib/kubelet/plugins_registry: The path where the node-driver-registrar registers the CSI plugin with kubelet.
    • /var/lib/kubelet/plugins/driver.longhorn.io: The path where the socket for the communication between kubelet Longhorn CSI driver.
    • /var/lib/kubelet/pods: The path where the Longhorn CSI driver mounts volume from the target path (via NodePublishVolume).
  • Longhorn CSI Attacher/Provisioner/Resizer/Snapshotter
    • /var/lib/kubelet/plugins/driver.longhorn.io: The path where the socket for the communication between kubelet Longhorn CSI driver.
  • Longhorn Backing Image Manager
    • /var/lib/longhorn: The default path for storing data on the host.
  • Longhorn Backing Image Data Source
    • /var/lib/longhorn: The default path for storing data on the host.
  • Longhorn System Restore Rollout
    • /var/lib/longhorn/engine-binaries: The default path for storing the Longhorn engine binaries.

Installing open-iscsi

The command used to install open-iscsi differs depending on the Linux distribution.

For GKE, we recommend using Ubuntu as the guest OS image since it containsopen-iscsi already.

You may need to edit the cluster security group to allow SSH access.

For SUSE and openSUSE, use this command:

  1. zypper install open-iscsi

For Debian and Ubuntu, use this command:

  1. apt-get install open-iscsi

For RHEL, CentOS, and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use below commands:

  1. yum --setopt=tsflags=noscripts install iscsi-initiator-utils
  2. echo "InitiatorName=$(/sbin/iscsi-iname)" > /etc/iscsi/initiatorname.iscsi
  3. systemctl enable iscsid
  4. systemctl start iscsid

Please ensure iscsi_tcp module has been loaded before iscsid service starts. Generally, it should be automatically loaded along with the package installation.

  1. modprobe iscsi_tcp

We also provide an iscsi installer to make it easier for users to install open-iscsi automatically:

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.4.1/deploy/prerequisite/longhorn-iscsi-installation.yaml

After the deployment, run the following command to check pods’ status of the installer:

  1. kubectl get pod | grep longhorn-iscsi-installation
  2. longhorn-iscsi-installation-49hd7   1/1     Running   0          21m
  3. longhorn-iscsi-installation-pzb7r   1/1     Running   0          39m

And also can check the log with the following command to see the installation result:

  1. kubectl logs longhorn-iscsi-installation-pzb7r -c iscsi-installation
  2. ...
  3. Installed:
  4.   iscsi-initiator-utils.x86_64 0:6.2.0.874-7.amzn2
  6. Dependency Installed:
  7.   iscsi-initiator-utils-iscsiuio.x86_64 0:6.2.0.874-7.amzn2
  9. Complete!
  10. Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service.
  11. iscsi install successfully

Installing NFSv4 client

In Longhorn system, backup feature requires NFSv4, v4.1 or v4.2, and ReadWriteMany (RWX) volume feature requires NFSv4.1. Before installing NFSv4 client userspace daemon and utilities, make sure the client kernel support is enabled on each Longhorn node.

  • Check NFSv4.1 support is enabled in kernel

    1. cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_1

  • Check NFSv4.2 support is enabled in kernel

    1. cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_2

The command used to install a NFSv4 client differs depending on the Linux distribution.

  • For Debian and Ubuntu, use this command:

    1. apt-get install nfs-common

  • For RHEL, CentOS, and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command:

    1. yum install nfs-utils

  • For SUSE/OpenSUSE you can install a NFSv4 client via:

    1. zypper install nfs-client

We also provide an nfs installer to make it easier for users to install nfs-client automatically:

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.4.1/deploy/prerequisite/longhorn-nfs-installation.yaml

After the deployment, run the following command to check pods’ status of the installer:

  1. kubectl get pod | grep longhorn-nfs-installation
  2. NAME                                  READY   STATUS    RESTARTS   AGE
  3. longhorn-nfs-installation-t2v9v   1/1     Running   0          143m
  4. longhorn-nfs-installation-7nphm   1/1     Running   0          143m

And also can check the log with the following command to see the installation result:

  1. kubectl logs longhorn-nfs-installation-t2v9v -c nfs-installation
  2. ...
  3. nfs install successfully

Checking the Kubernetes Version

Use the following command to check your Kubernetes server version

  1. kubectl version

Result:

  1. Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
  2. Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0+k3s1", GitCommit:"2705431d9645d128441c578309574cd262285ae6", GitTreeState:"clean", BuildDate:"2021-04-26T21:45:52Z", GoVersion:"go1.16.2", Compiler:"gc", Platform:"linux/amd64"}

The Server Version should be >= v1.21.