Liveness and Readiness Probes

It is possible to configure Liveness and Readiness Probes in a similar fashion like it is possible to configure Liveness and Readiness Probes on Containers.

Liveness Probes will effectively stop the VirtualMachineInstance if they fail, which will allow higher level controllers, like VirtualMachine or VirtualMachineInstanceReplicaSet to spawn new instances, which will hopefully be responsive again.

Readiness Probes are an indicator for Services and Endpoints if the VirtualMachineInstance is ready to receive traffic from Services. If Readiness Probes fail, the VirtualMachineInstance will be removed from the Endpoints which back services until the probe recovers.

Watchdogs focus on ensuring that an Operating System is still responsive. They complement the probes which are more workload centric. Watchdogs require kernel support from the guest and additional tooling like the commonly used watchdog binary.

Exec probes are Liveness or Readiness probes specifically intended for VMs. These probes run a command inside the VM and determine the VM ready/live state based on its success. For running commands inside the VMs, the qemu-guest-agent package is used. A command supplied to an exec probe will be wrapped by virt-probe in the operator and forwarded to the guest.

Define a HTTP Liveness Probe

The following VirtualMachineInstance configures a HTTP Liveness Probe via spec.livenessProbe.httpGet, which will query port 1500 of the VirtualMachineInstance, after an initial delay of 120 seconds. The VirtualMachineInstance itself installs and runs a minimal HTTP server on port 1500 via cloud-init.

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     kubevirt.io/vm: vmi-fedora-vmi
  6.   name: vmi-fedora
  7. spec:
  8.   template:
  9.     metadata:
  10.       labels:
  11.         kubevirt.io/domain: vmi-fedora
  12.         kubevirt.io/vm: vmi-fedora
  13.   domain:
  14.     devices:
  15.       disks:
  16.       - disk:
  17.           bus: virtio
  18.         name: containerdisk
  19.       - disk:
  20.           bus: virtio
  21.         name: cloudinitdisk
  22.       rng: {}
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   livenessProbe:
  27.     initialDelaySeconds: 120
  28.     periodSeconds: 20
  29.     httpGet:
  30.       port: 1500
  31.     timeoutSeconds: 10
  32.   terminationGracePeriodSeconds: 0
  33.   volumes:
  34.   - name: containerdisk
  35.     containerDisk:
  36.       image: quay.io/containerdisks/fedora:latest
  37.   - cloudInitNoCloud:
  38.       userData: |-
  39.         #cloud-config
  40.         password: fedora
  41.         user: fedora
  42.         chpasswd: { expire: False }
  43.         bootcmd:
  44.           - ["sudo", "dnf", "install", "-y", "nmap-ncat"]
  45.           - ["sudo", "systemd-run", "--unit=httpserver", "nc", "-klp", "1500", "-e", '/usr/bin/echo -e HTTP/1.1 200 OK\\nContent-Length: 12\\n\\nHello World!']
  46.     name: cloudinitdisk

Define a TCP Liveness Probe

The following VirtualMachineInstance configures a TCP Liveness Probe via spec.livenessProbe.tcpSocket, which will query port 1500 of the VirtualMachineInstance, after an initial delay of 120 seconds. The VirtualMachineInstance itself installs and runs a minimal HTTP server on port 1500 via cloud-init.

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     kubevirt.io/vm: vmi-fedora-vmi
  6.   name: vmi-fedora
  7. spec:
  8.   template:
  9.     metadata:
  10.       labels:
  11.         kubevirt.io/domain: vmi-fedora
  12.         kubevirt.io/vm: vmi-fedora
  13.   domain:
  14.     devices:
  15.       disks:
  16.       - disk:
  17.           bus: virtio
  18.         name: containerdisk
  19.       - disk:
  20.           bus: virtio
  21.         name: cloudinitdisk
  22.       rng: {}
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   livenessProbe:
  27.     initialDelaySeconds: 120
  28.     periodSeconds: 20
  29.     tcpSocket:
  30.       port: 1500
  31.     timeoutSeconds: 10
  32.   terminationGracePeriodSeconds: 0
  33.   volumes:
  34.   - name: containerdisk
  35.     containerDisk:
  36.       image: quay.io/containerdisks/fedora:latest
  37.   - cloudInitNoCloud:
  38.       userData: |-
  39.         #cloud-config
  40.         password: fedora
  41.         user: fedora
  42.         chpasswd: { expire: False }
  43.         bootcmd:
  44.           - ["sudo", "dnf", "install", "-y", "nmap-ncat"]
  45.           - ["sudo", "systemd-run", "--unit=httpserver", "nc", "-klp", "1500", "-e", '/usr/bin/echo -e HTTP/1.1 200 OK\\nContent-Length: 12\\n\\nHello World!']
  46.     name: cloudinitdisk

Define Readiness Probes

Readiness Probes are configured in a similar way like liveness probes. Instead of spec.livenessProbe, spec.readinessProbe needs to be filled:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     kubevirt.io/vm: vmi-fedora-vmi
  6.   name: vmi-fedora
  7. spec:
  8.   template:
  9.     metadata:
  10.       labels:
  11.         kubevirt.io/domain: vmi-fedora
  12.         kubevirt.io/vm: vmi-fedora
  13.   domain:
  14.     devices:
  15.       disks:
  16.       - disk:
  17.           bus: virtio
  18.         name: containerdisk
  19.       - disk:
  20.           bus: virtio
  21.         name: cloudinitdisk
  22.       rng: {}
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   readinessProbe:
  27.     initialDelaySeconds: 120
  28.     periodSeconds: 20
  29.     timeoutSeconds: 10
  30.     failureThreshold: 3
  31.     successThreshold: 3
  32.     httpGet:
  33.       port: 1500
  34.   terminationGracePeriodSeconds: 0
  35.   volumes:
  36.   - name: containerdisk
  37.     containerDisk:
  38.       image: quay.io/containerdisks/fedora:latest
  39.   - cloudInitNoCloud:
  40.       userData: |-
  41.         #cloud-config
  42.         password: fedora
  43.         user: fedora
  44.         chpasswd: { expire: False }
  45.         bootcmd:
  46.           - ["sudo", "dnf", "install", "-y", "nmap-ncat"]
  47.           - ["sudo", "systemd-run", "--unit=httpserver", "nc", "-klp", "1500", "-e", '/usr/bin/echo -e HTTP/1.1 200 OK\\nContent-Length: 12\\n\\nHello World!']
  48.     name: cloudinitdisk

Note that in the case of Readiness Probes, it is also possible to set a failureThreshold and a successThreashold to only flip between ready and non-ready state if the probe succeeded or failed multiple times.

Dual-stack considerations

Some context is needed to understand the limitations imposed by a dual-stack network configuration on readiness - or liveness - probes. Users must be fully aware that a dual-stack configuration is currently only available when using a masquerade binding type. Furthermore, it must be recalled that accessing a VM using masquerade binding type is performed via the pod IP address; in dual-stack mode, both IPv4 and IPv6 addresses can be used to reach the VM.

Dual-stack networking configurations have a limitation when using HTTP / TCP probes - you cannot probe the VMI by its IPv6 address. The reason for this is the host field for both the HTTP and TCP probe actions default to the pod’s IP address, which is currently always the IPv4 address.

Since the pod’s IP address is not known before creating the VMI, it is not possible to pre-provision the probe’s host field.

Defining a Watchdog

A watchdog is a more VM centric approach where the responsiveness of the Operating System is focused on. One can configure the i6300esb watchdog device:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     special: vmi-with-watchdog
  6.   name: vmi-with-watchdog
  7. spec:
  8.   domain:
  9.     devices:
  10.       watchdog:
  11.         name: mywatchdog
  12.         i6300esb:
  13.           action: "poweroff"
  14.       disks:
  15.       - disk:
  16.           bus: virtio
  17.         name: containerdisk
  18.       - disk:
  19.           bus: virtio
  20.         name: cloudinitdisk
  21.     machine:
  22.       type: ""
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   terminationGracePeriodSeconds: 0
  27.   volumes:
  28.   - containerDisk:
  29.       image: quay.io/containerdisks/fedora:latest
  30.     name: containerdisk
  31.   - cloudInitNoCloud:
  32.       userData: |-
  33.         #cloud-config
  34.         password: fedora
  35.         user: fedora
  36.         chpasswd: { expire: False }
  37.         bootcmd:
  38.           - ["sudo", "dnf", "install", "-y", "busybox"]
  39.     name: cloudinitdisk

The example above configures it with the poweroff action. It defines what will happen if the OS can’t respond anymore. Other possible actions are reset and shutdown. The VM in this example will have the device exposed as /dev/watchdog. This device can then be used by the watchdog binary. For example, if root executes this command inside the VM:

  1. sudo busybox watchdog -t 2000ms -T 4000ms /dev/watchdog

the watchdog will send a heartbeat every two seconds to /dev/watchdog and after four seconds without a heartbeat the defined action will be executed. In this case a hard poweroff.

Defining Guest-Agent Ping Probes

Guest-Agent probes are based on qemu-guest-agent guest-ping. This will ping the guest and return an error if the guest is not up and running. To easily define this on VM spec, specify guestAgentPing: {} in VM’s spec.template.spec.readinessProbe. virt-controller will translate this into a corresponding command wrapped by virt-probe.

Note: You can only define one of the type of probe, i.e. guest-agent exec or ping probes.

Important: If the qemu-guest-agent is not installed and enabled inside the VM, the probe will fail. Many images don’t enable the agent by default so make sure you either run one that does or enable it.

Make sure to provide enough delay and failureThreshold for the VM and the agent to be online.

In the following example the Fedora image does have qemu-guest-agent available by default. Nevertheless, in case qemu-guest-agent is not installed, it will be installed and enabled via cloud-init as shown in the example below. Also, cloud-init assigns the proper SELinux context, i.e. virt_qemu_ga_exec_t, to the /tmp/healthy.txt file. Otherwise, SELinux will deny the attempts to open the /tmp/healthy.txt file causing the probe to fail.

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   labels:
  5.     kubevirt.io/vm: vmi-guest-probe-vmi
  6.   name: vmi-fedora
  7. spec:
  8.   template:
  9.     metadata:
  10.       labels:
  11.         kubevirt.io/domain: vmi-guest-probe
  12.         kubevirt.io/vm: vmi-guest-probe
  13.   domain:
  14.     devices:
  15.       disks:
  16.       - disk:
  17.           bus: virtio
  18.         name: containerdisk
  19.       - disk:
  20.           bus: virtio
  21.         name: cloudinitdisk
  22.       rng: {}
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   readinessProbe:
  27.     exec:
  28.       command: ["cat", "/tmp/healthy.txt"]
  29.     failureThreshold: 10
  30.     initialDelaySeconds: 20
  31.     periodSeconds: 10
  32.     timeoutSeconds: 5
  33.   terminationGracePeriodSeconds: 180
  34.   volumes:
  35.   - name: containerdisk
  36.     containerDisk:
  37.       image: quay.io/containerdisks/fedora
  38.   - cloudInitNoCloud:
  39.       userData: |-
  40.         #cloud-config
  41.         password: fedora
  42.         user: fedora
  43.         chpasswd: { expire: False }
  44.         packages:
  45.           qemu-guest-agent
  46.         runcmd:
  47.           - ["touch", "/tmp/healthy.txt"]
  48.           - ["sudo", "chcon", "-t", "virt_qemu_ga_exec_t", "/tmp/healthy.txt"]
  49.           - ["sudo", "systemctl", "enable", "--now", "qemu-guest-agent"]
  50.     name: cloudinitdisk

Note that, in the above example if SELinux is not installed in your container disk image, the command chcon should be removed from the VM manifest shown below. Otherwise, the chcon command will fail.

The .status.ready field will switch to true indicating that probes are returning successfully:

  1. kubectl wait vmis/vmi-guest-probe --for=condition=Ready --timeout=5m

Additionally, the following command can be used inside the VM to watch the incoming qemu-ga commands:

  1. journalctl _COMM=qemu-ga --follow