Disks and Volumes

Making persistent storage in the cluster (volumes) accessible to VMs consists of three parts. First, volumes are specified in spec.volumes. Second, disks are added to the VM by specifying them in spec.domain.devices.disks. Finally, a reference to the specified volume is added to the disk specification by name.

Disks

Like all other vmi devices a spec.domain.devices.disks element has a mandatory name, and furthermore, the disk’s name must reference the name of a volume inside spec.volumes.

A disk can be made accessible via four different types:

All possible configuration options are available in the Disk API Reference.

All types, with the exception of floppy, allow you to specify the bus attribute. The bus attribute determines how the disk will be presented to the guest operating system. floppy disks don’t support the bus attribute: they are always attached to the fdc bus.

lun

A lun disk will expose the volume as a LUN device to the VM. This allows the VM to execute arbitrary iSCSI command passthrough.

A minimal example which attaches a PersistentVolumeClaim named mypvc as a lun device to the VM:

  1. metadata:
  2.   name: testvmi-lun
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mypvcdisk
  13.         # This makes it a lun device
  14.         lun: {}
  15.   volumes:
  16.     - name: mypvcdisk
  17.       persistentVolumeClaim:
  18.         claimName: mypvc

disk

A disk disk will expose the volume as an ordinary disk to the VM.

A minimal example which attaches a PersistentVolumeClaim named mypvc as a disk device to the VM:

  1. metadata:
  2.   name: testvmi-disk
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mypvcdisk
  13.         # This makes it a disk
  14.         disk: {}
  15.   volumes:
  16.     - name: mypvcdisk
  17.       persistentVolumeClaim:
  18.         claimName: mypvc

You can set the disk bus type, overriding the defaults, which in turn depends on the chipset the VM is configured to use:

  1. metadata:
  2.   name: testvmi-disk
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mypvcdisk
  13.         # This makes it a disk
  14.         disk:
  15.           # This makes it exposed as /dev/vda, being the only and thus first
  16.           # disk attached to the VM
  17.           bus: virtio
  18.   volumes:
  19.     - name: mypvcdisk
  20.       persistentVolumeClaim:
  21.         claimName: mypvc

floppy

Note: Starting with version 0.16.0, floppy disks are deprecated and will be rejected. They will be removed from the API in a future version.

A floppy disk will expose the volume as a floppy drive to the VM.

A minimal example which attaches a PersistentVolumeClaim named mypvc as a floppy device to the VM:

  1. metadata:
  2.   name: testvmi-floppy
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mypvcdisk
  13.         # This makes it a floppy
  14.         floppy: {}
  15.   volumes:
  16.     - name: mypvcdisk
  17.       persistentVolumeClaim:
  18.         claimName: mypvc

cdrom

A cdrom disk will expose the volume as a cdrom drive to the VM. It is read-only by default.

A minimal example which attaches a PersistentVolumeClaim named mypvc as a cdrom device to the VM:

  1. metadata:
  2.   name: testvmi-cdrom
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mypvcdisk
  13.         # This makes it a cdrom
  14.         cdrom:
  15.           # This makes the cdrom writeable
  16.           readOnly: false
  17.           # This makes the cdrom be exposed as SATA device
  18.           bus: sata
  19.   volumes:
  20.     - name: mypvcdisk
  21.       persistentVolumeClaim:
  22.         claimName: mypvc

Volumes

Supported volume sources are

All possible configuration options are available in the Volume API Reference.

cloudInitNoCloud

Allows attaching cloudInitNoCloud data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.

A simple example which attaches a Secret as a cloud-init disk datasource may look like this:

  1. metadata:
  2.   name: testvmi-cloudinitnocloud
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mybootdisk
  13.         lun: {}
  14.       - name: mynoclouddisk
  15.         disk: {}
  16.   volumes:
  17.     - name: mybootdisk
  18.       persistentVolumeClaim:
  19.         claimName: mypvc
  20.     - name: mynoclouddisk
  21.       cloudInitNoCloud:
  22.         secretRef:
  23.           name: testsecret

cloudInitConfigDrive

Allows attaching cloudInitConfigDrive data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.

A simple example which attaches a Secret as a cloud-init disk datasource may look like this:

  1. metadata:
  2.   name: testvmi-cloudinitconfigdrive
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mybootdisk
  13.         lun: {}
  14.       - name: myconfigdrivedisk
  15.         disk: {}
  16.   volumes:
  17.     - name: mybootdisk
  18.       persistentVolumeClaim:
  19.         claimName: mypvc
  20.     - name: myconfigdrivedisk
  21.       cloudInitConfigDrive:
  22.         secretRef:
  23.           name: testsecret

persistentVolumeClaim

Allows connecting a PersistentVolumeClaim to a VM disk.

Use a PersistentVolumeClaim when the VirtualMachineInstance’s disk needs to persist after the VM terminates. This allows for the VM’s data to remain persistent between restarts.

A PersistentVolume can be in “filesystem” or “block” mode:

  • Filesystem: For KubeVirt to be able to consume the disk present on a PersistentVolume’s filesystem, the disk must be named disk.img and be placed in the root path of the filesystem. Currently the disk is also required to be in raw format. > Important: The disk.img image file needs to be owned by the user-id 107 in order to avoid permission issues.

    Note: If the disk.img image file has not been created manually before starting a VM then it will be created automatically with the PersistentVolumeClaim size. Since not every storage provisioner provides volumes with the exact usable amount of space as requested (e.g. due to filesystem overhead), KubeVirt tolerates up to 10% less available space. This can be configured with the developerConfiguration.pvcTolerateLessSpaceUpToPercent value in the KubeVirt CR (kubectl edit kubevirt kubevirt -n kubevirt).

  • Block: Use a block volume for consuming raw block devices. Note: you need to enable the BlockVolume feature gate.

A simple example which attaches a PersistentVolumeClaim as a disk may look like this:

  1. metadata:
  2.   name: testvmi-pvc
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mypvcdisk
  13.         lun: {}
  14.   volumes:
  15.     - name: mypvcdisk
  16.       persistentVolumeClaim:
  17.         claimName: mypvc

dataVolume

DataVolumes are a way to automate importing virtual machine disks onto PVCs during the virtual machine’s launch flow. Without using a DataVolume, users have to prepare a PVC with a disk image before assigning it to a VM or VMI manifest. With a DataVolume, both the PVC creation and import is automated on behalf of the user.

DataVolume VM Behavior

DataVolumes can be defined in the VM spec directly by adding the DataVolumes to the dataVolumeTemplates list. Below is an example.

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachine
  3. metadata:
  4.   labels:
  5.     kubevirt.io/vm: vm-alpine-datavolume
  6.   name: vm-alpine-datavolume
  7. spec:
  8.   running: false
  9.   template:
  10.     metadata:
  11.       labels:
  12.         kubevirt.io/vm: vm-alpine-datavolume
  13.     spec:
  14.       domain:
  15.         devices:
  16.           disks:
  17.           - disk:
  18.               bus: virtio
  19.             name: datavolumedisk1
  20.         resources:
  21.           requests:
  22.             memory: 64M
  23.       volumes:
  24.       - dataVolume:
  25.           name: alpine-dv
  26.         name: datavolumedisk1
  27.   dataVolumeTemplates:
  28.   - metadata:
  29.       name: alpine-dv
  30.     spec:
  31.       pvc:
  32.         accessModes:
  33.         - ReadWriteOnce
  34.         resources:
  35.           requests:
  36.             storage: 2Gi
  37.       source:
  38.         http:
  39.           url: http://cdi-http-import-server.kubevirt/images/alpine.iso

You can see the DataVolume defined in the dataVolumeTemplates section has two parts. The source and pvc

The source part declares that there is a disk image living on an http server that we want to use as a volume for this VM. The pvc part declares the spec that should be used to create the PVC that hosts the source data.

When this VM manifest is posted to the cluster, as part of the launch flow a PVC will be created using the spec provided and the source data will be automatically imported into that PVC before the VM starts. When the VM is deleted, the storage provisioned by the DataVolume will automatically be deleted as well.

DataVolume VMI Behavior

For a VMI object, DataVolumes can be referenced as a volume source for the VMI. When this is done, it is expected that the referenced DataVolume exists in the cluster. The VMI will consume the DataVolume, but the DataVolume’s life-cycle will not be tied to the VMI.

Below is an example of a DataVolume being referenced by a VMI. It is expected that the DataVolume alpine-datavolume was created prior to posting the VMI manifest to the cluster. It is okay to post the VMI manifest to the cluster while the DataVolume is still having data imported. KubeVirt knows not to start the VMI until all referenced DataVolumes have finished their clone and import phases.

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-alpine-datavolume
  6.   name: vmi-alpine-datavolume
  7. spec:
  8.   domain:
  9.     devices:
  10.       disks:
  11.       - disk:
  12.           bus: virtio
  13.         name: disk1
  14.     machine:
  15.       type: ""
  16.     resources:
  17.       requests:
  18.         memory: 64M
  19.   terminationGracePeriodSeconds: 0
  20.   volumes:
  21.   - name: disk1
  22.     dataVolume:
  23.       name: alpine-datavolume

Enabling DataVolume support.

A DataVolume is a custom resource provided by the Containerized Data Importer (CDI) project. KubeVirt integrates with CDI in order to provide users a workflow for dynamically creating PVCs and importing data into those PVCs.

In order to take advantage of the DataVolume volume source on a VM or VMI, the DataVolumes feature gate must be enabled in the kubevirt CR. CDI must also be installed.

Installing CDI

Go to the CDI release page

Pick the latest stable release and post the corresponding cdi-controller-deployment.yaml manifest to your cluster.

ephemeral

An ephemeral volume is a local COW (copy on write) image that uses a network volume as a read-only backing store. With an ephemeral volume, the network backing store is never mutated. Instead all writes are stored on the ephemeral image which exists on local storage. KubeVirt dynamically generates the ephemeral images associated with a VM when the VM starts, and discards the ephemeral images when the VM stops.

Ephemeral volumes are useful in any scenario where disk persistence is not desired. The COW image is discarded when VM reaches a final state (e.g., succeeded, failed).

Currently, only PersistentVolumeClaim may be used as a backing store of the ephemeral volume.

Up-to-date information on supported backing stores can be found in the KubeVirt API.

  1. metadata:
  2.   name: testvmi-ephemeral-pvc
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: mypvcdisk
  13.         lun: {}
  14.   volumes:
  15.     - name: mypvcdisk
  16.       ephemeral:
  17.         persistentVolumeClaim:
  18.           claimName: mypvc

containerDisk

containerDisk was originally registryDisk, please update your code when needed.

The containerDisk feature provides the ability to store and distribute VM disks in the container image registry. containerDisks can be assigned to VMs in the disks section of the VirtualMachineInstance spec.

No network shared storage devices are utilized by containerDisks. The disks are pulled from the container registry and reside on the local node hosting the VMs that consume the disks.

When to use a containerDisk

containerDisks are ephemeral storage devices that can be assigned to any number of active VirtualMachineInstances. This makes them an ideal tool for users who want to replicate a large number of VM workloads that do not require persistent data. containerDisks are commonly used in conjunction with VirtualMachineInstanceReplicaSets.

When Not to use a containerDisk

containerDisks are not a good solution for any workload that requires persistent root disks across VM restarts.

containerDisk Workflow Example

Users can inject a VirtualMachineInstance disk into a container image in a way that is consumable by the KubeVirt runtime. Disks must be placed into the /disk directory inside the container. Raw and qcow2 formats are supported. Qcow2 is recommended in order to reduce the container image’s size. containerdisks can and should be based on scratch. No content except the image is required.

Note: Prior to kubevirt 0.20, the containerDisk image needed to have kubevirt/container-disk-v1alpha as base image.

Note: The containerDisk needs to be readable for the user with the UID 107 (qemu).

Example: Inject a local VirtualMachineInstance disk into a container image.

  1. cat << END > Dockerfile
  2. FROM scratch
  3. ADD --chown=107:107 fedora25.qcow2 /disk/
  4. END
  6. docker build -t vmidisks/fedora25:latest .

Example: Inject a remote VirtualMachineInstance disk into a container image.

  1. cat << END > Dockerfile
  2. FROM scratch
  3. ADD --chown=107:107 https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 /disk/
  4. END

Example: Upload the ContainerDisk container image to a registry.

  1. docker push vmidisks/fedora25:latest

Example: Attach the ContainerDisk as an ephemeral disk to a VM.

  1. metadata:
  2.   name: testvmi-containerdisk
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: containerdisk
  13.         disk: {}
  14.   volumes:
  15.     - name: containerdisk
  16.       containerDisk:
  17.         image: vmidisks/fedora25:latest

Note that a containerDisk is file-based and therefore cannot be attached as a lun device to the VM.

Custom disk image path

ContainerDisk also allows to store disk images in any folder, when required. The process is the same as previous. The main difference is, that in custom location, kubevirt does not scan for any image. It is your responsibility to provide full path for the disk image. Providing image path is optional. When no path is provided, kubevirt searches for disk images in default location: /disk.

Example: Build container disk image:

  1. cat << END > Dockerfile
  2. FROM scratch
  3. ADD fedora25.qcow2 /custom-disk-path/fedora25.qcow2
  4. END
  6. docker build -t vmidisks/fedora25:latest .
  7. docker push vmidisks/fedora25:latest

Create VMI with container disk pointing to the custom location:

  1. metadata:
  2.   name: testvmi-containerdisk
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.     devices:
  11.       disks:
  12.       - name: containerdisk
  13.         disk: {}
  14.   volumes:
  15.     - name: containerdisk
  16.       containerDisk:
  17.         image: vmidisks/fedora25:latest
  18.         path: /custom-disk-path/fedora25.qcow2

emptyDisk

An emptyDisk works similar to an emptyDir in Kubernetes. An extra sparse qcow2 disk will be allocated and it will live as long as the VM. Thus it will survive guest side VM reboots, but not a VM re-creation. The disk capacity needs to be specified.

Example: Boot cirros with an extra emptyDisk with a size of 2GiB:

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   name: testvmi-nocloud
  5. spec:
  6.   terminationGracePeriodSeconds: 5
  7.   domain:
  8.     resources:
  9.       requests:
  10.         memory: 64M
  11.     devices:
  12.       disks:
  13.       - name: containerdisk
  14.         disk:
  15.           bus: virtio
  16.       - name: emptydisk
  17.         disk:
  18.           bus: virtio
  19.   volumes:
  20.     - name: containerdisk
  21.       containerDisk:
  22.         image: kubevirt/cirros-registry-disk-demo:latest
  23.     - name: emptydisk
  24.       emptyDisk:
  25.         capacity: 2Gi

When to use an emptyDisk

Ephemeral VMs very often come with read-only root images and limited tmpfs space. In many cases this is not enough to install application dependencies and provide enough disk space for the application data. While this data is not critical and thus can be lost, it is still needed for the application to function properly during its lifetime. This is where an emptyDisk can be useful. An emptyDisk is often used and mounted somewhere in /var/lib or /var/run.

hostDisk

A hostDisk volume type provides the ability to create or use a disk image located somewhere on a node. It works similar to a hostPath in Kubernetes and provides two usage types:

  • DiskOrCreate if a disk image does not exist at a given location then create one

  • Disk a disk image must exist at a given location

Note: you need to enable the HostDisk feature gate.

Example: Create a 1Gi disk image located at /data/disk.img and attach it to a VM.

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-host-disk
  6.   name: vmi-host-disk
  7. spec:
  8.   domain:
  9.     devices:
  10.       disks:
  11.       - disk:
  12.           bus: virtio
  13.         name: host-disk
  14.     machine:
  15.       type: ""
  16.     resources:
  17.       requests:
  18.         memory: 64M
  19.   terminationGracePeriodSeconds: 0
  20.   volumes:
  21.   - hostDisk:
  22.       capacity: 1Gi
  23.       path: /data/disk.img
  24.       type: DiskOrCreate
  25.     name: host-disk
  26. status: {}

configMap

A configMap is a reference to a ConfigMap in Kubernetes. An extra iso disk will be allocated which has to be mounted on a VM. To mount the configMap users can use cloudInit and the disks serial number. The name needs to be set for a reference to the created kubernetes ConfigMap.

Note: Currently, ConfigMap update is not propagate into the VMI. If a ConfigMap is updated, only a pod will be aware of changes, not running VMIs.

Note: Due to a Kubernetes CRD issue, you cannot control the paths within the volume where ConfigMap keys are projected.

Example: Attach the configMap to a VM and use cloudInit to mount the iso disk:

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-fedora
  6.   name: vmi-fedora
  7. spec:
  8.   domain:
  9.     devices:
  10.       disks:
  11.       - disk:
  12.           bus: virtio
  13.         name: containerdisk
  14.       - disk:
  15.           bus: virtio
  16.         name: cloudinitdisk
  17.       - disk: {}
  18.         name: app-config-disk
  19.         # set serial
  20.         serial: CVLY623300HK240D
  21.     machine:
  22.       type: ""
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   terminationGracePeriodSeconds: 0
  27.   volumes:
  28.   - name: containerdisk
  29.     containerDisk:
  30.       image: kubevirt/fedora-cloud-container-disk-demo:latest
  31.   - cloudInitNoCloud:
  32.       userData: |-
  33.         #cloud-config
  34.         password: fedora
  35.         chpasswd: { expire: False }
  36.         bootcmd:
  37.           # mount the ConfigMap
  38.           - "mkdir /mnt/app-config"
  39.           - "mount /dev/$(lsblk --nodeps -no name,serial | grep CVLY623300HK240D | cut -f1 -d' ') /mnt/app-config"
  40.     name: cloudinitdisk
  41.   - configMap:
  42.       name: app-config
  43.     name: app-config-disk
  44. status: {}

secret

A secret is a reference to a Secret in Kubernetes. An extra iso disk will be allocated which has to be mounted on a VM. To mount the secret users can use cloudInit and the disks serial number. The secretName needs to be set for a reference to the created kubernetes Secret.

Note: Currently, Secret update propagation is not supported. If a Secret is updated, only a pod will be aware of changes, not running VMIs.

Note: Due to a Kubernetes CRD issue, you cannot control the paths within the volume where Secret keys are projected.

Example: Attach the secret to a VM and use cloudInit to mount the iso disk:

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-fedora
  6.   name: vmi-fedora
  7. spec:
  8.   domain:
  9.     devices:
  10.       disks:
  11.       - disk:
  12.           bus: virtio
  13.         name: containerdisk
  14.       - disk:
  15.           bus: virtio
  16.         name: cloudinitdisk
  17.       - disk: {}
  18.         name: app-secret-disk
  19.         # set serial
  20.         serial: D23YZ9W6WA5DJ487
  21.     machine:
  22.       type: ""
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   terminationGracePeriodSeconds: 0
  27.   volumes:
  28.   - name: containerdisk
  29.     containerDisk:
  30.       image: kubevirt/fedora-cloud-container-disk-demo:latest
  31.   - cloudInitNoCloud:
  32.       userData: |-
  33.         #cloud-config
  34.         password: fedora
  35.         chpasswd: { expire: False }
  36.         bootcmd:
  37.           # mount the Secret
  38.           - "mkdir /mnt/app-secret"
  39.           - "mount /dev/$(lsblk --nodeps -no name,serial | grep D23YZ9W6WA5DJ487 | cut -f1 -d' ') /mnt/app-secret"
  40.     name: cloudinitdisk
  41.   - secret:
  42.       secretName: app-secret
  43.     name: app-secret-disk
  44. status: {}

serviceAccount

A serviceAccount volume references a Kubernetes ServiceAccount. A new iso disk will be allocated with the content of the service account (namespace, token and ca.crt), which needs to be mounted in the VM. For automatic mounting, see the configMap and secret examples above.

Example:

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-fedora
  6.   name: vmi-fedora
  7. spec:
  8.   domain:
  9.     devices:
  10.       disks:
  11.       - disk:
  12.           bus: virtio
  13.         name: containerdisk
  14.       - disk:
  15.           bus: virtio
  16.         name: serviceaccountdisk
  17.     machine:
  18.       type: ""
  19.     resources:
  20.       requests:
  21.         memory: 1024M
  22.   terminationGracePeriodSeconds: 0
  23.   volumes:
  24.   - name: containerdisk
  25.     containerDisk:
  26.       image: kubevirt/fedora-cloud-container-disk-demo:latest
  27.   - name: serviceaccountdisk
  28.     serviceAccount:
  29.       serviceAccountName: default

downwardMetrics

A downwardMetrics volume exposes a limited set of VM and host metrics to the guest as a raw block volume. The format of the block volume is compatible with vhostmd.

Getting a limited set of host and VM metrics is in some cases required to allow third-parties diagnosing performance issues on their appliances. One prominent example is SAP HANA.

Example:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-fedora
  6.   name: vmi-fedora
  7. spec:
  8.   domain:
  9.     devices:
  10.       disks:
  11.       - disk:
  12.           bus: virtio
  13.         name: containerdisk
  14.       - disk:
  15.           bus: virtio
  16.         name: metrics
  17.     machine:
  18.       type: ""
  19.     resources:
  20.       requests:
  21.         memory: 1024M
  22.   terminationGracePeriodSeconds: 0
  23.   volumes:
  24.   - name: containerdisk
  25.     containerDisk:
  26.       image: kubevirt/fedora-cloud-container-disk-demo:latest
  27.   - name: metrics
  28.     downwardMetrics: {}

The vm-dump-metrics tool can be used to read the metrics:

  1. $ dnf install -y vm-dump-metrics
  2. $ vm-dump-metrics
  3. <metrics>
  4.   <metric type="string" context="host">
  5.     <name>HostName</name>
  6.     <value>node01</value>
  7. [...]
  8.   <metric type="int64" context="host" unit="s">
  9.     <name>Time</name>
  10.     <value>1619008605</value>
  11.   </metric>
  12.   <metric type="string" context="host">
  13.     <name>VirtualizationVendor</name>
  14.     <value>kubevirt.io</value>
  15.   </metric>
  16. </metrics>

Note: The DownwardMetrics feature gate must be enabled to use this volume. Available starting with KubeVirt v0.42.0.

High Performance Features

IOThreads

Libvirt has the ability to use IOThreads for dedicated disk access (for supported devices). These are dedicated event loop threads that perform block I/O requests and improve scalability on SMP systems. KubeVirt exposes this libvirt feature through the ioThreadsPolicy setting. Additionally, each Disk device exposes a dedicatedIOThread setting. This is a boolean that indicates the specified disk should be allocated an exclusive IOThread that will never be shared with other disks.

Currently valid policies are shared and auto. If ioThreadsPolicy is omitted entirely, use of IOThreads will be disabled. However, if any disk requests a dedicated IOThread, ioThreadsPolicy will be enabled and default to shared.

Shared

An ioThreadsPolicy of shared indicates that KubeVirt should use one thread that will be shared by all disk devices. This policy stems from the fact that large numbers of IOThreads is generally not useful as additional context switching is incurred for each thread.

Disks with dedicatedIOThread set to true will not use the shared thread, but will instead be allocated an exclusive thread. This is generally useful if a specific Disk is expected to have heavy I/O traffic, e.g. a database spindle.

Auto

auto IOThreads indicates that KubeVirt should use a pool of IOThreads and allocate disks to IOThreads in a round-robin fashion. The pool size is generally limited to twice the number of VCPU’s allocated to the VM. This essentially attempts to dedicate disks to separate IOThreads, but only up to a reasonable limit. This would come in to play for systems with a large number of disks and a smaller number of CPU’s for instance.

As a caveat to the size of the IOThread pool, disks with dedicatedIOThread will always be guaranteed their own thread. This effectively diminishes the upper limit of the number of threads allocated to the rest of the disks. For example, a VM with 2 CPUs would normally use 4 IOThreads for all disks. However if one disk had dedicatedIOThread set to true, then KubeVirt would only use 3 IOThreads for the shared pool.

There is always guaranteed to be at least one thread for disks that will use the shared IOThreads pool. Thus if a sufficiently large number of disks have dedicated IOThreads assigned, auto and shared policies would essentially result in the same layout.

IOThreads with Dedicated (pinned) CPUs

When guest’s vCPUs are pinned to a host’s physical CPUs, it is also best to pin the IOThreads to specific CPUs to prevent these from floating between the CPUs. KubeVirt will automatically calculate and pin each IOThread to a CPU or a set of CPUs, depending on the ration between them. In case there are more IOThreads than CPUs, each IOThread will be pinned to a CPU, in a round-robin fashion. Otherwise, when there are fewer IOThreads than CPU, each IOThread will be pinned to a set of CPUs.

IOThreads with QEMU Emulator thread and Dedicated (pinned) CPUs

To further improve the vCPUs latency, KubeVirt can allocate an additional dedicated physical CPU1, exclusively for the emulator thread, to which it will be pinned. This will effectively “isolate” the emulator thread from the vCPUs of the VMI. When ioThreadsPolicy is set to auto IOThreads will also be “isolated” from the vCPUs and placed on the same physical CPU as the QEMU emulator thread.

Examples

Shared IOThreads

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-shared
  6.   name: vmi-shared
  7. spec:
  8.   domain:
  9.     ioThreadsPolicy: shared
  10.     cpu:
  11.       cores: 2
  12.     devices:
  13.       disks:
  14.       - disk:
  15.           bus: virtio
  16.         name: vmi-shared_disk
  17.       - disk:
  18.           bus: virtio
  19.         name: emptydisk
  20.         dedicatedIOThread: true
  21.       - disk:
  22.           bus: virtio
  23.         name: emptydisk2
  24.         dedicatedIOThread: true
  25.       - disk:
  26.           bus: virtio
  27.         name: emptydisk3
  28.       - disk:
  29.           bus: virtio
  30.         name: emptydisk4
  31.       - disk:
  32.           bus: virtio
  33.         name: emptydisk5
  34.       - disk:
  35.           bus: virtio
  36.         name: emptydisk6
  37.     machine:
  38.       type: ""
  39.     resources:
  40.       requests:
  41.         memory: 64M
  42.   volumes:
  43.   - name: vmi-shared_disk
  44.     persistentVolumeClaim:
  45.       claimName: vmi-shared_pvc
  46.   - emptyDisk:
  47.       capacity: 1Gi
  48.     name: emptydisk
  49.   - emptyDisk:
  50.       capacity: 1Gi
  51.     name: emptydisk2
  52.   - emptyDisk:
  53.       capacity: 1Gi
  54.     name: emptydisk3
  55.   - emptyDisk:
  56.       capacity: 1Gi
  57.     name: emptydisk4
  58.   - emptyDisk:
  59.       capacity: 1Gi
  60.     name: emptydisk5
  61.   - emptyDisk:
  62.       capacity: 1Gi
  63.     name: emptydisk6

In this example, emptydisk and emptydisk2 both request a dedicated IOThread. vmi-shared_disk, and emptydisk 3 through 6 will all shared one IOThread.

  1. mypvc:        1
  2. emptydisk:    2
  3. emptydisk2:   3
  4. emptydisk3:   1
  5. emptydisk4:   1
  6. emptydisk5:   1
  7. emptydisk6:   1

Auto IOThreads

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-shared
  6.   name: vmi-shared
  7. spec:
  8.   domain:
  9.     ioThreadsPolicy: auto
  10.     cpu:
  11.       cores: 2
  12.     devices:
  13.       disks:
  14.       - disk:
  15.           bus: virtio
  16.         name: mydisk
  17.       - disk:
  18.           bus: virtio
  19.         name: emptydisk
  20.         dedicatedIOThread: true
  21.       - disk:
  22.           bus: virtio
  23.         name: emptydisk2
  24.         dedicatedIOThread: true
  25.       - disk:
  26.           bus: virtio
  27.         name: emptydisk3
  28.       - disk:
  29.           bus: virtio
  30.         name: emptydisk4
  31.       - disk:
  32.           bus: virtio
  33.         name: emptydisk5
  34.       - disk:
  35.           bus: virtio
  36.         name: emptydisk6
  37.     machine:
  38.       type: ""
  39.     resources:
  40.       requests:
  41.         memory: 64M
  42.   volumes:
  43.   - name: mydisk
  44.     persistentVolumeClaim:
  45.       claimName: mypvc
  46.   - emptyDisk:
  47.       capacity: 1Gi
  48.     name: emptydisk
  49.   - emptyDisk:
  50.       capacity: 1Gi
  51.     name: emptydisk2
  52.   - emptyDisk:
  53.       capacity: 1Gi
  54.     name: emptydisk3
  55.   - emptyDisk:
  56.       capacity: 1Gi
  57.     name: emptydisk4
  58.   - emptyDisk:
  59.       capacity: 1Gi
  60.     name: emptydisk5
  61.   - emptyDisk:
  62.       capacity: 1Gi
  63.     name: emptydisk6

This VM is identical to the first, except it requests auto IOThreads. emptydisk and emptydisk2 will still be allocated individual IOThreads, but the rest of the disks will be split across 2 separate iothreads (twice the number of CPU cores is 4).

Disks will be assigned to IOThreads like this:

  1. mypvc:        1
  2. emptydisk:    3
  3. emptydisk2:   4
  4. emptydisk3:   2
  5. emptydisk4:   1
  6. emptydisk5:   2
  7. emptydisk6:   1

Virtio Block Multi-Queue

Block Multi-Queue is a framework for the Linux block layer that maps Device I/O queries to multiple queues. This splits I/O processing up across multiple threads, and therefor multiple CPUs. libvirt recommends that the number of queues used should match the number of CPUs allocated for optimal performance.

This feature is enabled by the BlockMultiQueue setting under Devices:

  1. spec:
  2.   domain:
  3.     devices:
  4.       blockMultiQueue: true
  5.       disks:
  6.       - disk:
  7.           bus: virtio
  8.         name: mydisk

Note: Due to the way KubeVirt implements CPU allocation, blockMultiQueue can only be used if a specific CPU allocation is requested. If a specific number of CPUs hasn’t been allocated to a VirtualMachine, KubeVirt will use all CPU’s on the node on a best effort basis. In that case the amount of CPU allocation to a VM at the host level could change over time. If blockMultiQueue were to request a number of queues to match all the CPUs on a node, that could lead to over-allocation scenarios. To avoid this, KubeVirt enforces that a specific slice of CPU resources is requested in order to take advantage of this feature.

Example

  1. metadata:
  2.   name: testvmi-disk
  3. apiVersion: kubevirt.io/v1alpha3
  4. kind: VirtualMachineInstance
  5. spec:
  6.   domain:
  7.     resources:
  8.       requests:
  9.         memory: 64M
  10.         cpu: 4
  11.     devices:
  12.       blockMultiQueue: true
  13.       disks:
  14.       - name: mypvcdisk
  15.         disk:
  16.           bus: virtio
  17.   volumes:
  18.     - name: mypvcdisk
  19.       persistentVolumeClaim:
  20.         claimName: mypvc

This example will enable Block Multi-Queue for the disk mypvcdisk and allocate 4 queues (to match the number of CPUs requested).

Disk device cache

KubeVirt supports none and writethrough KVM/QEMU cache modes.

  • none I/O from the guest is not cached on the host. Use this option for guests with large I/O requirements. This option is generally the best choice.

  • writethrough I/O from the guest is cached on the host but written through to the physical medium.

Important: none cache mode is set as default if the file system supports direct I/O, otherwise, writethrough is used.

Note: It is possible to force a specific cache mode, although if none mode has been chosen and the file system does not support direct I/O then started VMI will return an error.

Example: force writethrough cache mode

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-pvc
  6.   name: vmi-pvc
  7. spec:
  8.   domain:
  9.     devices:
  10.       disks:
  11.       - disk:
  12.           bus: virtio
  13.         name: pvcdisk
  14.         cache: writethrough
  15.     machine:
  16.       type: ""
  17.     resources:
  18.       requests:
  19.         memory: 64M
  20.   terminationGracePeriodSeconds: 0
  21.   volumes:
  22.   - name: pvcdisk
  23.     persistentVolumeClaim:
  24.       claimName: disk-alpine
  25. status: {}