Hotplug Network Interfaces

Warning: The feature is in alpha stage and its API may be changed in the future.

KubeVirt now supports hotplugging network interfaces into a running Virtual Machine (VM). Hotplug is only supported for interfaces using the virtio model connected through bridge binding.

Requirements

Adding an interface to a KubeVirt Virtual Machine requires first an interface to be added to a running pod. This is not trivial, and has some requirements:

  • multus dynamic networks controller: this daemon will listen to annotation changes, and trigger multus to configure a new attachment for this pod.
  • multus running as a thick plugin: this multus version exposes an endpoint to create attachments for a given pod on demand.

Enabling network interface hotplug support

Network interface hotplug support must be enabled via a feature gate. The feature gates array in the KubeVirt CR must feature HotplugNICs.

Adding an interface to a running VM

First start a VM. You can refer to the following example:

  1. ---
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachine
  4. metadata:
  5.   name: vm-fedora
  6. spec:
  7.   running: true
  8.   template:
  9.     spec:
  10.       domain:
  11.         devices:
  12.           disks:
  13.           - disk:
  14.               bus: virtio
  15.             name: containerdisk
  16.           interfaces:
  17.           - masquerade: {}
  18.             name: defaultnetwork
  19.           rng: {}
  20.         resources:
  21.           requests:
  22.             memory: 1024M
  23.       networks:
  24.       - name: defaultnetwork
  25.         pod: {}
  26.       terminationGracePeriodSeconds: 0
  27.       volumes:
  28.       - containerDisk:
  29.           image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:devel
  30.         name: containerdisk

You should configure a network attachment definition - where the pod interface configuration is held. The snippet below shows an example of a very simple one:

  1. apiVersion: k8s.cni.cncf.io/v1
  2. kind: NetworkAttachmentDefinition
  3. metadata:
  4.   name: new-fancy-net
  5. spec:
  6.     config: '{
  7.       "cniVersion": "0.3.1",
  8.       "type": "bridge",
  9.       "mtu": 1300,
  10.       "name":"new-fancy-net"
  11.     }'

Please refer to the multus documentation for more info.

Once the virtual machine is running, and the attachment configuration provisioned, the user can request the interface hotplug operation. Please refer to the following snippet:

  1. virtctl addinterface vm-fedora --network-attachment-definition-name new-fancy-net --name dyniface1

This will add the interface and network to the VM spec template and to the running VMI object.

NOTE: You can use the --help parameter for more information on each parameter.

You can now check the VMI status for the presence of this new interface:

  1. kubectl get vmi vm-fedora -ojsonpath="{ @.status.interfaces }"

Removing an interface from a running VM

Following the example above, the user can request an interface unplug operation. Please refer to the following snippet:

  1. virtctl removeinterface vm-fedora --name dyniface1

The subject interface state will be set as absent in the VM spec template and the running VMI object.

Note: Existing VMs from version v0.59.0 and below does not support hot-unplug interfaces.

Migration based hotplug

In case your cluster doesn’t run Multus as thick plugin and Multus Dynamic Networks controller, it’s possible to hotplug an interface by migrating the VM.

The actual attachment won’t take place immediately, and the new interface will be available in the guest once the migration is completed.

Add new interface

  1. virtctl addinterface vm-fedora --network-attachment-definition-name new-fancy-net --name dyniface1

At this point the new interface is added to the spec but will not be attached to the running VM.

Migrate the VM

  1. cat <<EOF kubectl apply -f -
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachineInstanceMigration
  4. metadata:
  5.   name: migration-job
  6. spec:
  7.   vmiName: vmi-fedora
  8. EOF

See the Live Migration docs for more details.

Once the migration is completed the VM will have the new interface attached.

Note: It is recommended to avoid performing migrations in parallel to a hotplug operation. It is safer to assure hotplug succeeded or at least reached the VMI specification before issuing a migration.

Remove interface

  1. virtctl removeinterface vm-fedora --name dyniface1

At this point the subject interface should be detached from the guest but exist in the pod.

Migrate the VM

  1. cat <<EOF kubectl apply -f -
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachineInstanceMigration
  4. metadata:
  5.   name: migration-job
  6. spec:
  7.   vmiName: vmi-fedora
  8. EOF

See the Live Migration docs for more details.

Once the VM is migrated, the interface will not exist in the migration target pod.

Note: It is recommended to avoid performing migrations in parallel to an unplug operation. It is safer to assure unplug succeeded or at least reached the VMI specification before issuing a migration.

Virtio Limitations

The hotplugged interfaces have model: virtio. This imposes several limitations: each interface will consume a PCI slot in the VM, and there are a total maximum of 32. Furthermore, other devices will also use these PCI slots (e.g. disks, guest-agent, etc).

Kubevirt reserves resources for 4 interface to allow later hotplug operations. The actual maximum amount of available resources depends on the machine type (e.g. q35 adds another PCI slot). For more information on maximum limits, see libvirt documentation.

Yet, upon a VM restart, the hotplugged interface will become part of the standard networks; this mitigates the maximum hotplug interfaces (per machine type) limitation.

NOTE: the user can execute this command against a stopped VM - i.e. a VM without an associated VMI. When this happens, KubeVirt mutates the VM spec template on behalf of the user.