Memory Hotplug

Memory hotplug was introduced in KubeVirt version 1.1, enabling the dynamic resizing of the amount of memory available to a running VM.

Limitations

  • Memory hotplug is currently only supported on the x86_64,arm64 architectures.
  • Linux guests running at least Linux v5.8 are fully supported.
  • Windows guests support has been added to virtio-win, but it should be considered unstable.
  • Current hotplug implementation involves live-migration of the VM workload.
  • VirtualMachines must have at least 1GiB of memory to support memory-hotplug.

Configuration

Enable feature-gate

To use memory hotplug we need to add the VMLiveUpdateFeatures feature gate in the KubeVirt CR:

  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   configuration:
  5.     developerConfiguration:
  6.       featureGates:
  7.         - VMLiveUpdateFeatures

Configure the Workload Update Strategy

Configure LiveMigrate as workloadUpdateStrategy in the KubeVirt CR, since the current implementation of the hotplug process requires the VM to live-migrate.

  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   workloadUpdateStrategy:
  5.     workloadUpdateMethods:
  6.     - LiveMigrate

Configure the VM rollout strategy

Finally, set the VM rollout strategy to LiveUpdate, so that the changes made to the VM object propagate to the VMI without a restart.
This is also done in the KubeVirt CR configuration:

  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   configuration:
  5.     vmRolloutStrategy: "LiveUpdate"

NOTE: If memory hotplug is enabled/disabled on an already running VM, a reboot is necessary for the changes to take effect.

More information can be found on the VM Rollout Strategies page.

[OPTIONAL] Set a cluster-wide maximum amount of memory

You can set the maximum amount of memory for the guest using a cluster level setting in the KubeVirt CR.

  1. apiVersion: kubevirt.io/v1
  2. kind: KubeVirt
  3. spec:
  4.   configuration:
  5.     liveUpdateConfiguration:
  6.       maxGuest: 8Gi

The VM-level configuration will take precedence over the cluster-wide one.

Memory Hotplug in Action

First we enable the VMLiveUpdateFeatures feature gate, set the rollout strategy to LiveUpdate and set LiveMigrate as workloadUpdateStrategy in the KubeVirt CR.

  1. $ kubectl --namespace kubevirt patch kv kubevirt -p='[{"op": "add", "path": "/spec/configuration/developerConfiguration/featureGates", "value": ["VMLiveUpdateFeatures"]}]' --type='json'
  2. $ kubectl --namespace kubevirt patch kv kubevirt -p='[{"op": "add", "path": "/spec/configuration/vmRolloutStrategy", "value": "LiveUpdate"}]' --type='json'
  3. $ kubectl --namespace kubevirt patch kv kubevirt -p='[{"op": "add", "path": "/spec/workloadUpdateStrategy/workloadUpdateMethods", "value": ["LiveMigrate"]}]' --type='json'

Now we create a VM with memory hotplug enabled.

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   name: vm-alpine
  5. spec:
  6.   runStrategy: Always
  7.   template:
  8.     spec:
  9.       domain:
  10.         memory:
  11.           guest: 1Gi
  12.         devices:
  13.           interfaces:
  14.             - masquerade: {}
  15.               model: virtio
  16.               name: default
  17.           disks:
  18.           - disk:
  19.               bus: virtio
  20.             name: containerdisk
  21.       networks:
  22.         - name: default
  23.           pod: {}
  24.       volumes:
  25.       - containerDisk:
  26.           image: registry:5000/kubevirt/alpine-container-disk-demo:devel
  27.         name: containerdisk

The Virtual Machine will automatically start and once booted it will report the currently available memory to the guest in the status.memory field inside the VMI.

  1. $ kubectl get vmi vm-cirros -o json | jq .status.memory
  1. {
  2.   "guestAtBoot": "1Gi",
  3.   "guestCurrent": "1Gi",
  4.   "guestRequested": "1Gi"
  5. }

Since the Virtual Machine is now running we can patch the VM object to double the available guest memory so that we’ll go from 1Gi to 2Gi.

  1. $ kubectl patch vm vm-cirros -p='[{"op": "replace", "path": "/spec/template/spec/domain/memory/guest", "value": "2Gi"}]' --type='json'

After the hotplug request is processed and the Virtual Machine is live migrated, the new amount of memory should be available to the guest and visible in the VMI object.

  1. $ kubectl get vmi vm-cirros -o json | jq .status.memory
  1. {
  2.   "guestAtBoot": "1Gi",
  3.   "guestCurrent": "2Gi",
  4.   "guestRequested": "2Gi"
  5. }