Instance types and preferences

FEATURE STATE:

  • instancetype.kubevirt.io/v1alpha1 (Experimental) as of the v0.56.0 KubeVirt release
  • instancetype.kubevirt.io/v1alpha2 (Experimental) as of the v0.58.0 KubeVirt release
  • instancetype.kubevirt.io/v1beta1 as of the v1.0.0 KubeVirt release

See the Version History section for more details.

Introduction

KubeVirt’s VirtualMachine API contains many advanced options for tuning the performance of a VM that goes beyond what typical users need to be aware of. Users have previously been unable to simply define the storage/network they want assigned to their VM and then declare in broad terms what quality of resources and kind of performance characteristics they need for their VM.

Instance types and preferences provide a way to define a set of resource, performance and other runtime characteristics, allowing users to reuse these definitions across multiple VirtualMachines.

VirtualMachineInstancetype

  1. ---
  2. apiVersion: instancetype.kubevirt.io/v1beta1
  3. kind: VirtualMachineInstancetype
  4. metadata:
  5.   name: example-instancetype
  6. spec:
  7.   cpu:
  8.     guest: 1
  9.   memory:
  10.     guest: 128Mi

KubeVirt provides two CRDs for instance types, a cluster wide VirtualMachineClusterInstancetype and a namespaced VirtualMachineInstancetype. These CRDs encapsulate the following resource related characteristics of a VirtualMachine through a shared VirtualMachineInstancetypeSpec:

  • CPU : Required number of vCPUs presented to the guest
  • Memory : Required amount of memory presented to the guest
  • GPUs : Optional list of vGPUs to passthrough
  • HostDevices : Optional list of HostDevices to passthrough
  • IOThreadsPolicy : Optional IOThreadsPolicy to be used
  • LaunchSecurity: Optional LaunchSecurity to be used

Anything provided within an instance type cannot be overridden within the VirtualMachine. For example, as CPU and Memory are both required attributes of an instance type, if a user makes any requests for CPU or Memory resources within the underlying VirtualMachine, the instance type will conflict and the request will be rejected during creation.

VirtualMachinePreference

  1. ---
  2. apiVersion: instancetype.kubevirt.io/v1beta1
  3. kind: VirtualMachinePreference
  4. metadata:
  5.   name: example-preference
  6. spec:
  7.   devices:
  8.     preferredDiskBus: virtio
  9.     preferredInterfaceModel: virtio

KubeVirt also provides two further preference based CRDs, again a cluster wide VirtualMachineClusterPreference and namespaced VirtualMachinePreference. These CRDsencapsulate the preferred value of any remaining attributes of a VirtualMachine required to run a given workload, again this is through a shared VirtualMachinePreferenceSpec.

Unlike instance types, preferences only represent the preferred values and as such, they can be overridden by values in the VirtualMachine provided by the user.

In the example shown below, a user has provided a VirtualMachine with a disk bus already defined within a DiskTarget and has also selected a set of preferences with DevicePreference and preferredDiskBus , so the user’s original choice within the VirtualMachine and DiskTarget are used:

  1. $ kubectl apply -f - << EOF
  2. ---
  3. apiVersion: instancetype.kubevirt.io/v1beta1
  4. kind: VirtualMachinePreference
  5. metadata:
  6.   name: example-preference-disk-virtio
  7. spec:
  8.   devices:
  9.     preferredDiskBus: virtio
  10. ---
  11. apiVersion: kubevirt.io/v1
  12. kind: VirtualMachine
  13. metadata:
  14.   name: example-preference-user-override
  15. spec:
  16.   preference:
  17.     kind: VirtualMachinePreference
  18.     name: example-preference-disk-virtio
  19.   running: false
  20.   template:
  21.     spec:
  22.       domain:
  23.         memory:
  24.           guest: 128Mi
  25.         devices:
  26.           disks:
  27.           - disk:
  28.               bus: sata
  29.             name: containerdisk
  30.           - disk: {}
  31.             name: cloudinitdisk
  32.         resources: {}
  33.       terminationGracePeriodSeconds: 0
  34.       volumes:
  35.       - containerDisk:
  36.           image: registry:5000/kubevirt/cirros-container-disk-demo:devel
  37.         name: containerdisk
  38.       - cloudInitNoCloud:
  39.           userData: |
  40.             #!/bin/sh
  42.             echo 'printed from cloud-init userdata'
  43.         name: cloudinitdisk
  44. EOF
  45. virtualmachinepreference.instancetype.kubevirt.io/example-preference-disk-virtio created
  46. virtualmachine.kubevirt.io/example-preference-user-override configured
  49. $ virtctl start example-preference-user-override
  50. VM example-preference-user-override was scheduled to start
  52. # We can see the original request from the user within the VirtualMachine lists `containerdisk` with a `SATA` bus
  53. $ kubectl get vms/example-preference-user-override -o json | jq .spec.template.spec.domain.devices.disks
  54. [
  55.   {
  56.     "disk": {
  57.       "bus": "sata"
  58.     },
  59.     "name": "containerdisk"
  60.   },
  61.   {
  62.     "disk": {},
  63.     "name": "cloudinitdisk"
  64.   }
  65. ]
  67. # This is still the case in the VirtualMachineInstance with the remaining disk using the `preferredDiskBus` from the preference of `virtio`
  68. $ kubectl get vmis/example-preference-user-override -o json | jq .spec.domain.devices.disks
  69. [
  70.   {
  71.     "disk": {
  72.       "bus": "sata"
  73.     },
  74.     "name": "containerdisk"
  75.   },
  76.   {
  77.     "disk": {
  78.       "bus": "virtio"
  79.     },
  80.     "name": "cloudinitdisk"
  81.   }
  82. ]

VirtualMachine

  1. ---
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachine
  4. metadata:
  5.   name: example-vm
  6. spec:
  7.   instancetype:
  8.     kind: VirtualMachineInstancetype
  9.     name: example-instancetype
  10.   preference:
  11.     kind: VirtualMachinePreference
  12.     name: example-preference

The previous instance type and preference CRDs are matched to a given VirtualMachine through the use of a matcher. Each matcher consists of the following:

  • Name (string): Name of the resource being referenced
  • Kind (string): Optional, defaults to the cluster wide CRD kinds of VirtualMachineClusterInstancetype or VirtualMachineClusterPreference if not provided
  • RevisionName (string) : Optional, name of a ControllerRevision containing a copy of the VirtualMachineInstancetypeSpec or VirtualMachinePreferenceSpec taken when the VirtualMachine is first created. See the Versioning section below for more details on how and why this is captured.
  • InferFromVolume (string): Optional, see the Inferring defaults from a Volume section below for more details.

Versioning

Versioning of these resources is required to ensure the eventual VirtualMachineInstance created when starting a VirtualMachine does not change between restarts if any referenced instance type or set of preferences are updated during the lifetime of the VirtualMachine.

This is currently achieved by using ControllerRevision to retain a copy of the VirtualMachineInstancetype or VirtualMachinePreference at the time the VirtualMachine is created. A reference to these ControllerRevisions are then retained in the InstancetypeMatcher and PreferenceMatcher within the VirtualMachine for future use.

  1. $ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml
  2. virtualmachineinstancetype.instancetype.kubevirt.io/csmall created
  3. virtualmachine.kubevirt.io/vm-cirros-csmall created
  5. $ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype
  6. {
  7.   "kind": "VirtualMachineInstancetype",
  8.   "name": "csmall",
  9.   "revisionName": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1"
  10. }
  12. $ kubectl get controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1 -o json | jq .
  13. {
  14.   "apiVersion": "apps/v1",
  15.   "data": {
  16.     "apiVersion": "instancetype.kubevirt.io/v1beta1",
  17.     "kind": "VirtualMachineInstancetype",
  18.     "metadata": {
  19.       "creationTimestamp": "2022-09-30T12:20:19Z",
  20.       "generation": 1,
  21.       "name": "csmall",
  22.       "namespace": "default",
  23.       "resourceVersion": "10303",
  24.       "uid": "72c3a35b-6e18-487d-bebf-f73c7d4f4a40"
  25.     },
  26.     "spec": {
  27.       "cpu": {
  28.         "guest": 1
  29.       },
  30.       "memory": {
  31.         "guest": "128Mi"
  32.       }
  33.     }
  34.   },
  35.   "kind": "ControllerRevision",
  36.   "metadata": {
  37.     "creationTimestamp": "2022-09-30T12:20:19Z",
  38.     "name": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1",
  39.     "namespace": "default",
  40.     "ownerReferences": [
  41.       {
  42.         "apiVersion": "kubevirt.io/v1",
  43.         "blockOwnerDeletion": true,
  44.         "controller": true,
  45.         "kind": "VirtualMachine",
  46.         "name": "vm-cirros-csmall",
  47.         "uid": "5216527a-1d31-4637-ad3a-b640cb9949a2"
  48.       }
  49.     ],
  50.     "resourceVersion": "10307",
  51.     "uid": "a7bc784b-4cea-45d7-8432-15418e1dd7d3"
  52.   },
  53.   "revision": 0
  54. }
  57. $ kubectl delete vm/vm-cirros-csmall
  58. virtualmachine.kubevirt.io "vm-cirros-csmall" deleted
  60. $ kubectl get controllerrevision/controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1
  61. Error from server (NotFound): controllerrevisions.apps "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1" not found

Users can opt in to moving to a newer generation of an instance type or preference by removing the referenced revisionName from the appropriate matcher within the VirtualMachine object. This will result in fresh ControllerRevisions being captured and used.

The following example creates a VirtualMachine using an initial version of the csmall instance type before increasing the number of vCPUs provided by the instance type:

  1. $ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml
  2. virtualmachineinstancetype.instancetype.kubevirt.io/csmall created
  3. virtualmachine.kubevirt.io/vm-cirros-csmall created
  5. $ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype
  6. {
  7.   "kind": "VirtualMachineInstancetype",
  8.   "name": "csmall",
  9.   "revisionName": "vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-1"
  10. }
  12. $ virtctl start vm-cirros-csmall
  13. VM vm-cirros-csmall was scheduled to start
  15. $ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu
  16. {
  17.   "cores": 1,
  18.   "model": "host-model",
  19.   "sockets": 1,
  20.   "threads": 1
  21. }
  23. $ kubectl patch VirtualMachineInstancetype/csmall --type merge -p '{"spec":{"cpu":{"guest":2}}}'
  24. virtualmachineinstancetype.instancetype.kubevirt.io/csmall patched

In order for this change to be picked up within the VirtualMachine, we need to stop the running VirtualMachine and clear the revisionName referenced by the InstancetypeMatcher:

  1. $ virtctl stop vm-cirros-csmall
  2. VM vm-cirros-csmall was scheduled to stop
  4. $ kubectl patch vm/vm-cirros-csmall --type merge -p '{"spec":{"instancetype":{"revisionName":""}}}'
  5. virtualmachine.kubevirt.io/vm-cirros-csmall patched
  7. $ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype
  8. {
  9.   "kind": "VirtualMachineInstancetype",
  10.   "name": "csmall",
  11.   "revisionName": "vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-2"
  12. }

As you can see above, the InstancetypeMatcher now references a new ControllerRevision containing generation 2 of the instance type. We can now start the VirtualMachine again and see the new number of vCPUs being used by the VirtualMachineInstance:

  1. $ virtctl start vm-cirros-csmall
  2. VM vm-cirros-csmall was scheduled to start
  4. $ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu
  5. {
  6.   "cores": 1,
  7.   "model": "host-model",
  8.   "sockets": 2,
  9.   "threads": 1
  10. }

inferFromVolume

The inferFromVolume attribute of both the InstancetypeMatcher and PreferenceMatcher allows a user to request that defaults are inferred from a volume. When requested, KubeVirt will look for the following labels on the underlying PVC, DataSource or DataVolume to determine the default name and kind:

  • instancetype.kubevirt.io/default-instancetype
  • instancetype.kubevirt.io/default-instancetype-kind (optional, defaults to VirtualMachineClusterInstancetype)
  • instancetype.kubevirt.io/default-preference
  • instancetype.kubevirt.io/default-preference-kind (optional, defaults to VirtualMachineClusterPreference)

These values are then written into the appropriate matcher by the mutation webhook and used during validation before the VirtualMachine is formally accepted.

Failure to find the referenced Volume or labels on an underlying resource will cause the request to be rejected.

  1. $ kubectl kustomize https://github.com/kubevirt/common-instancetypes.git | kubectl apply -f -
  2. [..]
  3. $ virtctl image-upload pvc cirros-pvc --size=1Gi --image-path=./cirros-0.5.2-x86_64-disk.img
  4. [..]
  5. $ kubectl label pvc/cirros-pvc \
  6.   instancetype.kubevirt.io/default-instancetype=server.tiny \
  7.   instancetype.kubevirt.io/default-preference=cirros
  8. [..]
  9. $ kubectl apply -f - << EOF
  10. ---
  11. apiVersion: cdi.kubevirt.io/v1beta1
  12. kind: DataSource
  13. metadata:
  14.   name: cirros-datasource
  15. spec:
  16.   source:
  17.     pvc:
  18.       name: cirros-pvc
  19.       namespace: default
  20. ---
  21. apiVersion: kubevirt.io/v1
  22. kind: VirtualMachine
  23. metadata:
  24.   name: cirros
  25. spec:
  26.   instancetype:
  27.     inferFromVolume: cirros-volume
  28.   preference:
  29.     inferFromVolume: cirros-volume
  30.   running: false
  31.   dataVolumeTemplates:
  32.     - metadata:
  33.         name: cirros-datavolume
  34.       spec:
  35.         pvc:
  36.           accessModes:
  37.             - ReadWriteOnce
  38.           resources:
  39.             requests:
  40.               storage: 1Gi
  41.           storageClassName: local
  42.         sourceRef:
  43.           kind: DataSource
  44.           name: cirros-datasource
  45.           namespace: default
  46.   template:
  47.     spec:
  48.       domain:
  49.         devices: {}
  50.       volumes:
  51.         - dataVolume:
  52.             name: cirros-datavolume
  53.           name: cirros-volume
  54. EOF
  55. [..]
  56. kubectl get vms/cirros -o json | jq '.spec.instancetype, .spec.preference'
  57. {
  58.   "kind": "virtualmachineclusterinstancetype",
  59.   "name": "server.tiny",
  60.   "revisionName": "cirros-server.tiny-76454433-3d82-43df-a7e5-586e48c71f68-1"
  61. }
  62. {
  63.   "kind": "virtualmachineclusterpreference",
  64.   "name": "cirros",
  65.   "revisionName": "cirros-cirros-85823ddc-9e8c-4d23-a94c-143571b5489c-1"
  66. }

common-instancetypes

A set of common instance types and preferences are provided through the kubevirt/common-instancetypes project. To install all resources provided by the project, simply build with kustomize and apply:

  1. $ kubectl kustomize https://github.com/kubevirt/common-instancetypes.git | kubectl apply -f -

Alternatively targets for each of the available custom resource types are available for example for VirtualMachineClusterInstancetype:

  1. $ kubectl kustomize https://github.com/kubevirt/common-instancetypes.git/VirtualMachineClusterInstancetypes | kubectl apply -f -

Examples

Various examples are available within the kubevirt repo under /examples. The following uses an example VirtualMachine provided by the containerdisk/fedora repo and replaces much of the DomainSpec with the equivalent instance type and preferences:

  1. $ kubectl apply -f - << EOF
  2. ---
  3. apiVersion: instancetype.kubevirt.io/v1beta1
  4. kind: VirtualMachineInstancetype
  5. metadata:
  6.   name: cmedium
  7. spec:
  8.   cpu:
  9.     guest: 1
  10.   memory:
  11.     guest: 1Gi
  12. ---
  13. apiVersion: instancetype.kubevirt.io/v1beta1
  14. kind: VirtualMachinePreference
  15. metadata:
  16.   name: fedora
  17. spec:
  18.   devices:
  19.     preferredDiskBus: virtio
  20.     preferredInterfaceModel: virtio
  21.     preferredRng: {}
  22.   features:
  23.     preferredAcpi: {}
  24.     preferredSmm: {}
  25.   firmware:
  26.     preferredUseEfi: true
  27.     preferredUseSecureBoot: true    
  28. ---
  29. apiVersion: kubevirt.io/v1
  30. kind: VirtualMachine
  31. metadata:
  32.   creationTimestamp: null
  33.   name: fedora
  34. spec:
  35.   instancetype:
  36.     name: cmedium
  37.     kind: virtualMachineInstancetype
  38.   preference:
  39.     name: fedora
  40.     kind: virtualMachinePreference
  41.   runStrategy: Always
  42.   template:
  43.     metadata:
  44.       creationTimestamp: null
  45.     spec:
  46.       domain:
  47.         devices: {}
  48.       volumes:
  49.       - containerDisk:
  50.           image: quay.io/containerdisks/fedora:latest
  51.         name: containerdisk
  52.       - cloudInitNoCloud:
  53.           userData: |-
  54.             #cloud-config
  55.             ssh_authorized_keys:
  56.               - ssh-rsa AAAA...
  57.         name: cloudinit
  58. EOF

Version History

instancetype.kubevirt.io/v1alpha1 (Experimental)

  • Initial development version.

instancetype.kubevirt.io/v1alpha2 (Experimental)

  • This version captured complete VirtualMachine{Instancetype,ClusterInstancetype,Preference,ClusterPreference} objects within the created ControllerRevisions

  • This version is backwardly compatible with instancetype.kubevirt.io/v1alpha1.

instancetype.kubevirt.io/v1beta1