Control libvirt logging for each component

Generally, cluster admins can control the log verbosity of each KubeVirt component in KubeVirt CR. For more details, please, check the KubeVirt documentation.

Nonetheless, regular users can also adjust the qemu component logging to have a finer control over it. The annotation kubevirt.io/libvirt-log-filters enables you to modify each component’s log level.

Example:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   annotations:
  5.     kubevirt.io/libvirt-log-filters: "2:qemu.qemu_monitor 3:*"
  6.   labels:
  7.     special: vmi-debug-tools
  8.   name: vmi-debug-tools
  9. spec:
  10.   domain:
  11.     devices:
  12.       disks:
  13.       - disk:
  14.           bus: virtio
  15.         name: containerdisk
  16.       - disk:
  17.           bus: virtio
  18.         name: cloudinitdisk
  19.       rng: {}
  20.     resources:
  21.       requests:
  22.         memory: 1024M
  23.   volumes:
  24.   - containerDisk:
  25.       image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel
  26.     name: containerdisk
  27.   - cloudInitNoCloud:
  28.       userData: |-
  29.         #cloud-config
  30.         password: fedora
  31.         chpasswd: { expire: False }
  32.     name: cloudinitdisk

Then, it is possible to obtain the logs from the virt-launcher output:

  1. $ kubectl get pods
  2. NAME                                  READY   STATUS    RESTARTS   AGE
  3. virt-launcher-vmi-debug-tools-fk64q   3/3     Running   0          64s
  4. $ kubectl  logs virt-launcher-vmi-debug-tools-fk64q
  5. [..]
  6. {"component":"virt-launcher","level":"info","msg":"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\"timestamp\": {\"seconds\": 1698324640, \"microseconds\": 523652}, \"event\": \"NIC_RX_FILTER_CHANGED\", \"data\": {\"name\": \"ua-default\", \"path\": \"/machine/peripheral/ua-default/virtio-backend\"}}","pos":"qemuMonitorJSONIOProcessLine:205","subcomponent":"libvirt","thread":"80","timestamp":"2023-10-26T12:50:40.523000Z"}
  7. {"component":"virt-launcher","level":"info","msg":"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\"timestamp\": {\"seconds\": 1698324644, \"microseconds\": 165626}, \"event\": \"VSERPORT_CHANGE\", \"data\": {\"open\": true, \"id\": \"channel0\"}}","pos":"qemuMonitorJSONIOProcessLine:205","subcomponent":"libvirt","thread":"80","timestamp":"2023-10-26T12:50:44.165000Z"}
  8. [..]
  9. {"component":"virt-launcher","level":"info","msg":"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\"timestamp\": {\"seconds\": 1698324646, \"microseconds\": 707666}, \"event\": \"RTC_CHANGE\", \"data\": {\"offset\": 0, \"qom-path\": \"/machine/unattached/device[8]\"}}","pos":"qemuMonitorJSONIOProcessLine:205","subcomponent":"libvirt","thread":"80","timestamp":"2023-10-26T12:50:46.708000Z"}
  10. [..]

The annotation enables the filter from the container creation. However, in certain cases you might desire to change the logging level dynamically once the container and libvirt have already been started. In this case, virt-admin comes to the rescue.

Example:

  1. $ kubectl get pods
  2. NAME                                READY   STATUS    RESTARTS   AGE
  3. virt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          26m
  4. $ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session  daemon-log-filters  "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util"
  5. $ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session  daemon-log-filters
  6.  Logging filters: 1:*libvirt* 1:*qemu* 1:*conf* 1:*security* 3:*event* 3:*json* 3:*file* 3:*object* 1:*util*

Otherwise, if you prefer to redirect the output to a file and fetch it later, you can rely on kubectl cp to retrieve the file. In this case, we are saving the file in the /var/run/libvirt directory because the compute container has the permissions to write there.

Example:

  1. $ kubectl get pods
  2. NAME                                READY   STATUS    RESTARTS   AGE
  3. virt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          26m
  4. $ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session daemon-log-outputs "1:file:/var/run/libvirt/libvirtd.log"
  5. $ kubectl cp virt-launcher-vmi-ephemeral-nqcld:/var/run/libvirt/libvirtd.log libvirt-kubevirt.log
  6. tar: Removing leading `/' from member names