Hotplug Network Interfaces

Warning: The feature is in alpha stage and its API may be changed in the future.

KubeVirt supports hotplugging and unplugging network interfaces into a running Virtual Machine (VM).

Hotplug is supported for interfaces using the virtio model connected through bridge binding or SR-IOV binding.

Hot-unplug is supported only for interfaces connected through bridge binding.

Requirements

Adding an interface to a KubeVirt Virtual Machine requires first an interface to be added to a running pod. This is not trivial, and has some requirements:

  • Multus Dynamic Networks Controller: this daemon will listen to annotation changes, and trigger Multus to configure a new attachment for this pod.
  • Multus CNI running as a thick plugin: this Multus version exposes an endpoint to create attachments for a given pod on demand.

Enabling network interface hotplug support

Network interface hotplug support must be enabled via a feature gate. The feature gates array in the KubeVirt CR must feature HotplugNICs.

Adding an interface to a running VM

First start a VM. You can refer to the following example:

  1. ---
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachine
  4. metadata:
  5.   name: vm-fedora
  6. spec:
  7.   running: true
  8.   template:
  9.     spec:
  10.       domain:
  11.         devices:
  12.           disks:
  13.           - disk:
  14.               bus: virtio
  15.             name: containerdisk
  16.           interfaces:
  17.           - masquerade: {}
  18.             name: defaultnetwork
  19.           rng: {}
  20.         resources:
  21.           requests:
  22.             memory: 1024M
  23.       networks:
  24.       - name: defaultnetwork
  25.         pod: {}
  26.       terminationGracePeriodSeconds: 0
  27.       volumes:
  28.       - containerDisk:
  29.           image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:devel
  30.         name: containerdisk

You should configure a network attachment definition - where the pod interface configuration is held. The snippet below shows an example of a very simple one:

  1. apiVersion: k8s.cni.cncf.io/v1
  2. kind: NetworkAttachmentDefinition
  3. metadata:
  4.   name: new-fancy-net
  5. spec:
  6.     config: '{
  7.       "cniVersion": "0.3.1",
  8.       "type": "bridge",
  9.       "mtu": 1300,
  10.       "name":"new-fancy-net"
  11.     }'

Please refer to the Multus documentation for more information.

Once the virtual machine is running, and the attachment configuration provisioned, the user can request the interface hotplug operation by editing the VM spec template and adding the desired interface and network:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   name: vm-fedora
  5. template:
  6.   spec:
  7.     domain:
  8.       devices:
  9.         interfaces:
  10.         - name: defaultnetwork
  11.           masquerade: {}
  12.           # new interface
  13.         - name: dyniface1
  14.           bridge: {}
  15.     networks:
  16.     - name: defaultnetwork
  17.       pod: {}
  18.       # new network
  19.     - name: dyniface1
  20.       multus:
  21.         networkName: new-fancy-net
  22.  ...

Note: virtctl addinterface and removeinterface commands are no longer available, hotplug/unplug interfaces is done by editing the VM spec template.

The interface and network will be added to the corresponding VMI object as well by Kubevirt.

You can now check the VMI status for the presence of this new interface:

  1. kubectl get vmi vm-fedora -ojsonpath="{ @.status.interfaces }"

Removing an interface from a running VM

Following the example above, the user can request an interface unplug operation by editing the VM spec template and set the desired interface state to absent:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   name: vm-fedora
  5. template:
  6.   spec:
  7.     domain:
  8.       devices:
  9.         interfaces:
  10.           - name: defaultnetwork
  11.             masquerade: {}
  12.           # set the interface state to absent 
  13.           - name: dyniface1
  14.             state: absent
  15.             bridge: {}
  16.     networks:
  17.       - name: defaultnetwork
  18.         pod: {}
  19.       - name: dyniface1
  20.         multus:
  21.           networkName: new-fancy-net

The interface in the corresponding VMI object will be set with state ‘absent’ as well by Kubevirt.

Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.

Migration based hotplug

In case your cluster doesn’t run Multus as thick plugin and Multus Dynamic Networks controller, it’s possible to hotplug an interface by migrating the VM.

The actual attachment won’t take place immediately, and the new interface will be available in the guest once the migration is completed.

Add new interface

Add the desired interface and network to the VM spec template:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   name: vm-fedora
  5. template:
  6.   spec:
  7.     domain:
  8.       devices:
  9.         interfaces:
  10.         - name: defaultnetwork
  11.           masquerade: {}
  12.           # new interface
  13.         - name: dyniface1
  14.           bridge: {}
  15.     networks:
  16.     - name: defaultnetwork
  17.       pod: {}
  18.       # new network
  19.     - name: dyniface1
  20.       multus:
  21.         networkName: new-fancy-net
  22.  ...

At this point the interface and network will be added to the corresponding VMI object as well, but won’t be attached to the guest.

Migrate the VM

  1. cat <<EOF kubectl apply -f -
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachineInstanceMigration
  4. metadata:
  5.   name: migration-job
  6. spec:
  7.   vmiName: vmi-fedora
  8. EOF

Please refer to the Live Migration documentation for more information.

Once the migration is completed the VM will have the new interface attached.

Note: It is recommended to avoid performing migrations in parallel to a hotplug operation. It is safer to assure hotplug succeeded or at least reached the VMI specification before issuing a migration.

Remove interface

Set the desired interface state to absent in the VM spec template:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   name: vm-fedora
  5. template:
  6.   spec:
  7.     domain:
  8.       devices:
  9.         interfaces:
  10.           - name: defaultnetwork
  11.             masquerade: {}
  12.           # set the interface state to absent 
  13.           - name: dyniface1
  14.             state: absent
  15.             bridge: {}
  16.     networks:
  17.       - name: defaultnetwork
  18.         pod: {}
  19.       - name: dyniface1
  20.         multus:
  21.           networkName: new-fancy-net

At this point the subject interface should be detached from the guest but exist in the pod.

Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.

Migrate the VM

  1. cat <<EOF kubectl apply -f -
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachineInstanceMigration
  4. metadata:
  5.   name: migration-job
  6. spec:
  7.   vmiName: vmi-fedora
  8. EOF

Please refer to the Live Migration documentation for more information.

Once the VM is migrated, the interface will not exist in the migration target pod.

Note: It is recommended to avoid performing migrations in parallel to an unplug operation. It is safer to assure unplug succeeded or at least reached the VMI specification before issuing a migration.

SR-IOV interfaces

Kubevirt supports hot-plugging of SR-IOV interfaces to running VMs.

Similar to bridge binding interfaces, edit the VM spec template and add the desired SR-IOV interface and network:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   name: vm-fedora
  5. template:
  6.   spec:
  7.     domain:
  8.       devices:
  9.         interfaces:
  10.         - name: defaultnetwork
  11.           masquerade: {}
  12.           # new interface
  13.         - name: sriov-net
  14.           sriov: {}
  15.     networks:
  16.     - name: defaultnetwork
  17.       pod: {}
  18.       # new network
  19.     - name: sriov-net
  20.       multus:
  21.         networkName: sriov-net-1
  22.  ...

Please refer to the Interface and Networks documentation for more information about SR-IOV networking.

At this point the interface and network will be added to the corresponding VMI object as well, but won’t be attached to the guest.

Migrate the VM

  1. cat <<EOF kubectl apply -f -
  2. apiVersion: kubevirt.io/v1
  3. kind: VirtualMachineInstanceMigration
  4. metadata:
  5.   name: migration-job
  6. spec:
  7.   vmiName: vmi-fedora
  8. EOF

Please refer to the Live Migration documentation for more information.

Once the VM is migrated, the interface will not exist in the migration target pod. Due to limitation of Kubernetes device plugin API to allocate resources dynamically, the SR-IOV device plugin cannot allocate additional SR-IOV resources for Kubevirt to hotplug. Thus, SR-IOV interface hotplug is limited to migration based hotplug only, regardless of Multus “thick” version.

Virtio Limitations

The hotplugged interfaces have model: virtio. This imposes several limitations: each interface will consume a PCI slot in the VM, and there are a total maximum of 32. Furthermore, other devices will also use these PCI slots (e.g. disks, guest-agent, etc).

Kubevirt reserves resources for 4 interface to allow later hotplug operations. The actual maximum amount of available resources depends on the machine type (e.g. q35 adds another PCI slot). For more information on maximum limits, see libvirt documentation.

Yet, upon a VM restart, the hotplugged interface will become part of the standard networks; this mitigates the maximum hotplug interfaces (per machine type) limitation.

Note: The user can execute this command against a stopped VM - i.e. a VM without an associated VMI. When this happens, KubeVirt mutates the VM spec template on behalf of the user.