Node Disk Support

Longhorn now supports the addition and management of various disk types (AIO, NVMe, and VirtIO) on nodes, enhancing filesystem operations, storage performance, and compatibility.

  • Enhanced Storage Performance

    Utilizing NVMe and VirtIO disks allows for faster disk operations, significantly improving overall performance.

  • Filesystem Compatibility

    Disks managed with NVMe or VirtIO drivers offer better filesystem support, including advanced operations like trimming.

  • Flexibility

    Users can select the disk type that best fits their environment: AIO for traditional setups, NVMe for high-performance needs, or VirtIO for virtualized environments.

  • Ease of Management

    Automatic detection of disk drivers simplifies the addition and management of disks, reducing administrative overhead.

Configure a Disk on Longhorn Node

Longhorn can automatically detect the disk type if node.spec.disks[i].diskDriver is set to auto, optimizing storage performance. The detection and management will be as follows:

  • NVMe Disk: managed by spdk_tgt using the nvme bdev driver, and node.status.diskStatus[i].diskDriver will be set to nvme.
  • VirtIO Disk: managed by spdk_tgt using the virtio bdev driver, and node.status.diskStatus[i].diskDriver will be set to virtio-blk.
  • Other Disks: managed by spdk_tgt using the aio bdev driver, and node.status.diskStatus[i].diskDriver will be set to aio.

Alternatively, users can manually set node.spec.disks[i].diskDriver to aio to force the use of the aio bdev driver.

To support NVMe and VirtIO disks, you need to find the BDF (Bus, Device, Function) of the disk as a disk path that will be added to the Longhorn node. The following examples provide an introduction to configuring NVMe disks, VirtIO disks, and others.

Note

Once these disks are managed by the NVMe bdev driver or VirtIO bdev driver, instead of the Linux kernel driver, they will no be listed under /dev/nvmeXnY or /dev/vdbX.

Using NVMe Disks

  1. List the disks

    First, identify the NVMe disks available on your system by running the following command:

    1. # ls -al /sys/block/

    Example output:

    1. lrwxrwxrwx  1 root root 0  Jul  30 12:20 loop0 -> ../devices/virtual/block/loop0
    2. lrwxrwxrwx  1 root root 0  Jul  30 12:20 nvme0n1 -> ../devices/pci0000:00/0000:00:01.2/0000:02:00.0/nvme/nvme0/nvme0n1
    3. lrwxrwxrwx  1 root root 0  Jul  30 12:20 nvme0n1 -> ../devices/pci0000:00/0000:00:01.2/0000:05:00.0/nvme/nvme1/nvme1n1

  2. Get the BDF of the NVMe disk

    Identify the BDF of the NVMe disk /dev/nvme1n1. From the example above, the BDF is 0000:05:00.0.

  3. Add the NVMe disk to spec.disks of node.longhorn.io

    1. nvme-disk:
    2.   allowScheduling: true
    3.   diskType: block
    4.   diskDriver: auto
    5.   evictionRequested: false
    6.   path: 0000:05:00.0
    7.   storageReserved: 0
    8.   tags: []

  4. Check the status.diskStatus. The disk should be detected without errors, and the diskDriver should be set to nvme.

Note: Alternative Disk Configuration

If you add the disk using a different path, such as:

  1. nvme-disk:
  2.   allowScheduling: true
  3.   diskType: block
  4.   diskDriver: auto
  5.   evictionRequested: false
  6.   path: /dev/nvme1n1
  7.   storageReserved: 0
  8.   tags: []

In this case, the disk will be managed by the aio bdev driver, and the node.status.diskStatus[i].diskDriver will be set to aio.

Using VirtIO Disks

The steps are similar to NVMe disks.

  1. List the disks

    First, identify the VirtIO disks available on your system by running the following command:

    1. # ls -al /sys/block/

    Example output:

    1. lrwxrwxrwx  1 root root 0  Jul  30 12:20 loop0 -> ../devices/virtual/block/loop0
    2. lrwxrwxrwx  1 root root 0 Feb 22 14:04 vda -> ../devices/pci0000:00/0000:00:02.3/0000:04:00.0/virtio2/block/vda
    3. lrwxrwxrwx  1 root root 0 Feb 22 14:24 vdb -> ../devices/pci0000:00/0000:00:02.6/0000:07:00.0/virtio5/block/vdb

  2. Get the BDF of the VirtIO disk

    Identify the BDF of the VirtIO disk /dev/vdb. From the example above, the BDF is 0000:07:00.0.

  3. Add the NVMe disk to spec.disks of node.longhorn.io

    1. nvme-disk:
    2.   allowScheduling: true
    3.   diskType: block
    4.   diskDriver: auto
    5.   evictionRequested: false
    6.   path: 0000:07:00.0
    7.   storageReserved: 0
    8.   tags: []

  4. Check the status.diskStatus. The disk should be detected without errors, and the diskDriver should be set to virtio-blk.

Note: Alternative Disk Configuration

If you add the disk using a different path, such as:

  1. nvme-disk:
  2.   allowScheduling: true
  3.   diskType: block
  4.   diskDriver: auto
  5.   evictionRequested: false
  6.   path: /dev/vdb
  7.   storageReserved: 0
  8.   tags: []

In this case, the disk will be managed by the aio bdev driver, and the node.status.diskStatus[i].diskDriver will be set to aio.

Using AIO Disks

When neither NVMe nor VirtIO drivers can manage a disk, Longhorn will default to using the aio bdev driver. Users can also manually configure this.

  1. Add the disk to spec.disks of node.longhorn.io

    1. default-disk-loop:
    2.   allowScheduling: true
    3.   diskDriver: aio
    4.   diskType: block
    5.   evictionRequested: false
    6.   path: /dev/loop12
    7.   storageReserved: 0
    8.   tags: []

  2. Check node.status.diskStatus. The disk should be detected without errors, and the node.status.diskStatus[i].diskDriver will be set to aio.

History

Original Feature Request