Quick Start

Table of Contents

Longhorn’s V2 Data Engine harnesses the power of the Storage Performance Development Kit (SPDK) to elevate its overall performance. The integration significantly reduces I/O latency while simultaneously boosting IOPS and throughput. The enhancement provides a high-performance storage solution capable of meeting diverse workload demands.

V2 Data Engine is currently a PREVIEW feature and should NOT be utilized in a production environment. At present, a volume with V2 Data Engine only supports

  • Volume lifecycle (creation, attachment, detachment and deletion)
  • Degraded volume
  • Offline replica rebuilding
  • Block disk management
  • Orphaned replica management

In addition to the features mentioned above, additional functionalities such as replica number adjustment, online replica rebuilding, snapshot, backup, restore and so on will be introduced in future versions.

This tutorial will guide you through the process of configuring the environment and create Kubernetes persistent storage resources of persistent volumes (PVs) and persistent volume claims (PVCs) that correspond to Longhorn volumes using V2 Data Engine.

Prerequisites

Configure Kernel Modules and Huge Pages

For Debian and Ubuntu, please install Linux kernel extra modules before loading the kernel modules

  1. apt install -y linux-modules-extra-`uname -r`

We provide a manifest that helps you configure the kernel modules and huge pages automatically, making it easier to set up.

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/deploy/prerequisite/longhorn-spdk-setup.yaml

And also can check the log with the following command to see the installation result.

  1. Cloning into '/tmp/spdk'...
  2. INFO: Requested 512 hugepages but 512 already allocated on node0
  3. SPDK environment is configured successfully

Or, you can install them manually by following these steps.

  • Load the kernel modules on the each Longhorn node

    1. modprobe uio
    2. modprobe uio_pci_generic

  • Configure huge pages SPDK utilizes huge pages to enhance performance and minimize memory overhead. To enable the usage of huge pages, it is necessary to configure 2MiB-sized huge pages on each Longhorn node. Specifically, 512 pages (equivalent to a total of 1 GiB) need to be available on each Longhorn node. To allocate the huge pages, run the following commands on each node.

    1. echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

    To make the change permanent, add the following line to the file /etc/sysctl.conf.

    1. echo "vm.nr_hugepages=512" >> /etc/sysctl.conf

Install NVMe Userspace Tool and Load nvme-tcp Kernel Module

NOTICE:

Make sure that the version of nvme-cli is equal to or greater than 1.12.

If the version of nvme-cli installed by the below steps is not equal to or greater than 1.12., you will need to compile the utility from the source codes and install it on each Longhorn node by manual.

Also, install the uuid development library before compiling to support the show-hostnqn subcommand.

For SUSE/OpenSUSE you can install it use this command:

  1. zypper install uuid-devel

For Debian and Ubuntu, use this command:

  1. apt install uuid-dev

For RHEL, CentOS, and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command:

  1. yum install uuid-devel

And also can check the log with the following command to see the installation result

  1. nvme-cli install successfully

We provide a manifest that helps you finish the deployment on each Longhorn node.

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/deploy/prerequisite/longhorn-nvme-cli-installation.yaml

Or, you can manually install them.

  • Install nvme-cli on each node and make sure that the version of nvme-cli is equal to or greater than 1.12.

    For SUSE/OpenSUSE you can install it use this command:

    1. zypper install nvme-cli

    For Debian and Ubuntu, use this command:

    1. apt install nvme-cli

    For RHEL, CentOS, and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command:

    1. yum install nvme-cli

    To check the version of nvme-cli, execute the following command.

    1. nvme version

  • Load nvme-tcp kernel module on the each Longhorn node

    1. modprobe nvme-tcp

Load Kernel Modules Automatically on Boot

Rather than manually loading kernel modules uio, uio_pci_generic and nvme-tcp each time after reboot, you can streamline the process by configuring automatic module loading during the boot sequence. For detailed instructions, please consult the manual provided by your operating system.

Reference:

Restart kubelet

After finishing the above steps, restart kubelet on each node.

Check Environment

Make sure everything is correctly configured and installed by

  1. bash -c "$(curl -sfL https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/scripts/environment_check.sh)" -s -s

Installation

Install Longhorn System

Follow the steps in Quick Installation to install Longhorn system.

Enable V2 Data Engine

Enable the V2 Data Engine by changing the v2-data-engine setting to true after installation. Following this, the instance-manager pods will be automatically restarted.

Or, you can enable it in Setting > General > V2 Data Engine.

CPU and Memory Usage

When the V2 Data Engine is enabled, each instance-manager pod utilizes 1 CPU core. This high CPU usage is attributed to the spdk_tgt process running within each instance-manager pod. The spdk_tgt process is responsible for handling input/output (IO) operations and requires intensive polling. As a result, it consumes 100% of a dedicated CPU core to efficiently manage and process the IO requests, ensuring optimal performance and responsiveness for storage operations.

  1. NAME                                                CPU(cores)   MEMORY(bytes)
  2. csi-attacher-6488f75fb4-48pnb                       1m           19Mi
  3. csi-attacher-6488f75fb4-94m6r                       1m           16Mi
  4. csi-attacher-6488f75fb4-zmwfm                       1m           15Mi
  5. csi-provisioner-6785d78459-6tps7                    1m           18Mi
  6. csi-provisioner-6785d78459-bj89g                    1m           23Mi
  7. csi-provisioner-6785d78459-c5dzt                    1m           17Mi
  8. csi-resizer-d9bb7b7fc-25m8b                         1m           17Mi
  9. csi-resizer-d9bb7b7fc-fncjf                         1m           15Mi
  10. csi-resizer-d9bb7b7fc-t5dw7                         1m           17Mi
  11. csi-snapshotter-5b89555c8f-76ptq                    1m           15Mi
  12. csi-snapshotter-5b89555c8f-7vgtv                    1m           19Mi
  13. csi-snapshotter-5b89555c8f-vkhd8                    1m           17Mi
  14. engine-image-ei-b907910b-5vp8h                      12m          15Mi
  15. engine-image-ei-b907910b-9krcz                      17m          15Mi
  16. instance-manager-b3735b3e6d0a9e27d1464f548bdda5ec   1000m        29Mi
  17. instance-manager-cbe60909512c58798690f692b883e5a9   1001m        27Mi
  18. longhorn-csi-plugin-qf9kt                           1m           61Mi
  19. longhorn-csi-plugin-zk6sm                           1m           60Mi
  20. longhorn-driver-deployer-7d46fd5945-8tfmk           1m           24Mi
  21. longhorn-manager-nm925                              6m           137Mi
  22. longhorn-manager-np849                              6m           126Mi
  23. longhorn-ui-54df99bfc-2lc8w                         0m           2Mi
  24. longhorn-ui-54df99bfc-w6dts                         0m           2Mi

You can observe the utilization of allocated huge pages on each node by running the command kubectl get node <node name> -o yaml.

  1. # kubectl get node sles-pool1-07437316-4jw8f -o yaml
  2. ...
  4. status:
  5.   ...
  6.   allocatable:
  7.     cpu: "8"
  8.     ephemeral-storage: "203978054087"
  9.     hugepages-1Gi: "0"
  10.     hugepages-2Mi: 1Gi
  11.     memory: 31813168Ki
  12.     pods: "110"
  13.   capacity:
  14.     cpu: "8"
  15.     ephemeral-storage: 209681388Ki
  16.     hugepages-1Gi: "0"
  17.     hugepages-2Mi: 1Gi
  18.     memory: 32861744Ki
  19.     pods: "110"
  20. ...

Add block-type Disks in Longhorn Nodes

Unlike filesystem-type disks that are designed for legacy volumes, volumes using V2 Data Engine are persistent on block-type disks. Therefore, it is necessary to equip Longhorn nodes with block-type disks.

Prepare disks

If there are no additional disks available on the Longhorn nodes, you can create loop block devices to test the feature. To accomplish this, execute the following command on each Longhorn node to create a 10 GiB block device.

  1. dd if=/dev/zero of=blockfile bs=1M count=10240
  2. losetup -f blockfile

To display the path of the block device when running the command losetup -f blockfile, use the following command.

  1. losetup -j blockfile

Add disks to node.longhorn.io

You can add the disk by navigating to the Node UI page and specify the Disk Type as Block. Next, provide the block device’s path in the Path field.

Or, edit the node.longhorn.io resource.

  1. kubectl -n longhorn-system edit node.longhorn.io <NODE NAME>

Add the disk to Spec.Disks

  1. <DISK NAME>:
  2.   allowScheduling: true
  3.   evictionRequested: false
  4.   path: /PATH/TO/BLOCK/DEVICE
  5.   storageReserved: 0
  6.   tags: []
  7.   diskType: block

Wait for a while, you will see the disk is displayed in the Status.DiskStatus.

Application Deployment

After the installation and configuration, we can dyamically provision a Persistent Volume using V2 Data Engine as the following steps.

Create a StorageClass

Use following command to create a StorageClass called longhorn-spdk. Set parameters.backendStoreDriver to spdk to utilize V2 Data Engine.

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/examples/v2/storageclass.yaml

Create Longhorn Volumes

Create a Pod that uses Longhorn volumes using V2 Data Engine by running this command:

  1. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/examples/v2/pod_with_pvc.yaml

Or, if you are creating a volume on Longhorn UI, please specify the Backend Data Engine as v2.