Configuring a watchdog

Expose a watchdog by configuring the virtual machine (VM) for a watchdog device, installing the watchdog, and starting the watchdog service.

Prerequisites

  • The virtual machine must have kernel support for an i6300esb watchdog device. Fedora images support i6300esb.

Defining a watchdog device

Define how the watchdog proceeds when the operating system (OS) no longer responds.

Available actions

poweroff

The virtual machine (VM) powers down immediately. If spec.running is set to true, or spec.runStrategy is not set to manual, then the VM reboots.

reset

The VM reboots in place and the guest OS cannot react. Because the length of time required for the guest OS to reboot can cause liveness probes to timeout, use of this option is discouraged. This timeout can extend the time it takes the VM to reboot if cluster-level protections notice the liveness probe failed and forcibly reschedule it.

shutdown

The VM gracefully powers down by stopping all services.

Procedure

  1. Create a YAML file with the following contents:

    1. apiVersion: kubevirt.io/v1
    2. kind: VirtualMachine
    3. metadata:
    4.   labels:
    5.     kubevirt.io/vm: vm2-rhel84-watchdog
    6.   name: <vm-name>
    7. spec:
    8.   running: false
    9.   template:
    10.     metadata:
    11.      labels:
    12.         kubevirt.io/vm: vm2-rhel84-watchdog
    13.     spec:
    14.       domain:
    15.         devices:
    16.           watchdog:
    17.             name: <watchdog>
    18.             i6300esb:
    19.               action: "poweroff" (1)
    20. ...
    1Specify the watchdog action (poweroff, reset, or shutdown).

    The example above configures the i6300esb watchdog device on a RHEL8 VM with the poweroff action and exposes the device as /dev/watchdog.

    This device can now be used by the watchdog binary.

  2. Apply the YAML file to your cluster by running the following command:

    1. $ oc apply -f <file_name>.yaml

Verification

This procedure is provided for testing watchdog functionality only and must not be run on production machines.

  1. Run the following command to verify that the VM is connected to the watchdog device:

    1. $ lspci | grep watchdog -i

  2. Run one of the following commands to confirm the watchdog is active:

    • Trigger a kernel panic:

      1. # echo c > /proc/sysrq-trigger

    • Terminate the watchdog service:

      1. # pkill -9 watchdog

Installing a watchdog device

Install the watchdog package on your virtual machine and start the watchdog service.

Procedure

  1. As a root user, install the watchdog package and dependencies:

    1. # yum install watchdog

  2. Uncomment the following line in the /etc/watchdog.conf file, and save the changes:

    1. #watchdog-device = /dev/watchdog

  3. Enable the watchdog service to start on boot:

    1. # systemctl enable --now watchdog.service

Additional resources