CPU Hotplug

The CPU hotplug feature was introduced in KubeVirt v1.0, making it possible to configure the VM workload to allow for adding or removing virtual CPUs while the VM is running.

Abstract

A virtual CPU (vCPU) is the CPU that is seen to the Guest VM OS. A VM owner can manage the amount of vCPUs from the VM spec template using the CPU topology fields (spec.template.spec.domain.cpu). The cpu object has the integers cores,sockets,threads so that the virtual CPU is calculated by the following formula: cores * sockets * threads.

Before CPU hotplug was introduced, the VM owner could change these integers in the VM template while the VM is running, and they were staged until the next boot cycle. With CPU hotplug, it is possible to patch the sockets integer in the VM template and the change will take effect right away.

Per each new socket that is hot-plugged, the amount of new vCPUs that would be seen by the guest is cores * threads, since the overall calculation of vCPUs is cores * sockets * threads.

Configuration

Enable feature-gate

In order to enable CPU hotplug we need to add the VMLiveUpdateFeatures feature gate in Kubevirt CR:

  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   configuration:
  5.     developerConfiguration:
  6.       featureGates:
  7.         - VMLiveUpdateFeatures

Configure the workload update strategy

Current implementation of the hotplug process requires the VM to live-migrate. The migration will be triggered automatically by the workload updater. The workload update strategy in the KubeVirt CR must be configured with LiveMigrate, as follows:

  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   workloadUpdateStrategy:
  5.     workloadUpdateMethods:
  6.     - LiveMigrate

Configure the VM rollout strategy

Hotplug requires a VM rollout strategy of LiveUpdate, so that the changes made to the VM object propagate to the VMI without a restart. This is also done in the KubeVirt CR configuration:

  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   configuration:
  5.     vmRolloutStrategy: "LiveUpdate"

More information can be found on the VM Rollout Strategies page

[OPTIONAL] Set maximum sockets or hotplug ratio

You can explicitly set the maximum amount of sockets in three ways:

  1. with a value VM level
  2. with a value at the cluster level
  3. with a ratio at the cluster level (maxSockets = ratio * sockets).

Note: the third way (cluster-level ratio) will also affect other quantitative hotplug resources like memory.

VM level

Cluster level value

Cluster level ratio

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. spec:
  4.   template:
  5.     spec:
  6.       domain:
  7.         cpu:
  8.           maxSockets: 8
  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   configuration:
  5.     liveUpdateConfiguration:
  6.       maxCpuSockets: 8
  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   configuration:
  5.     liveUpdateConfiguration:
  6.       maxHotplugRatio: 4

The VM-level configuration will take precedence over the cluster-wide configuration.

Hotplug process

Let’s assume we have a running VM with the 4 vCPUs, which were configured with sockets:4 cores:1 threads:1 In the VMI status we can observe the current CPU topology the VM is running with:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. ...
  4. status:
  5.   currentCPUTopology:
  6.     cores: 1
  7.     sockets: 4
  8.     threads: 1

Now we want to hotplug another socket, by patching the VM object:

  1. kubectl patch vm vm-cirros --type='json' \
  2. -p='[{"op": "replace", "path": "/spec/template/spec/domain/cpu/sockets", "value": 5}]'

We can observe the CPU hotplug process in the VMI status:

  1. status:
  2.   conditions:
  3.   - lastProbeTime: null
  4.     lastTransitionTime: null
  5.     status: "True"
  6.     type: LiveMigratable
  7.   - lastProbeTime: null
  8.     lastTransitionTime: null
  9.     status: "True"
  10.     type: HotVCPUChange
  11.   currentCPUTopology:
  12.     cores: 1
  13.     sockets: 4
  14.     threads: 1

Please note the condition HotVCPUChange that indicates the hotplug process is taking place. Also you can notice the VirtualMachineInstanceMigration object that was created for the VM in subject:

  1. NAME                             PHASE     VMI
  2. kubevirt-workload-update-kflnl   Running   vm-cirros

When the hotplug process has completed, the currentCPUTopology will be updated with the new number of sockets and the migration is marked as successful.

  1. #kubectl get vmi vm-cirros -oyaml
  3. apiVersion: kubevirt.io/v1
  4. kind: VirtualMachineInstance
  5. metadata:
  6.   name: vm-cirros
  7. spec:
  8.   domain:
  9.     cpu:
  10.       cores: 1
  11.       sockets: 5
  12.       threads: 1
  13. ...
  14. ...
  15. status:
  16.   currentCPUTopology:
  17.     cores: 1
  18.     sockets: 5
  19.     threads: 1
  22. #kubectl get vmim -l kubevirt.io/vmi-name=vm-cirros
  23. NAME                             PHASE       VMI
  24. kubevirt-workload-update-cgdgd   Succeeded   vm-cirros

Limitations

  • VPCU hotplug is currently not supported by ARM64 architecture.
  • Current hotplug implementation involves live-migration of the VM workload.