Launch QEMU with strace

This guide explains how launch QEMU with a debugging tool in virt-launcher pod. This method can be useful to debug early failures or starting QEMU as a child of the debug tool relying on ptrace. The second point is particularly relevant when a process is operating in a non-privileged environment since otherwise, it would need root access to be able to ptrace the process.

Ephemeral containers are among the emerging techniques to overcome the lack of debugging tool inside the original image. This solution does, however, come with a number of limitations. For example, it is possible to spawn a new container inside the same pod of the application to debug and share the same PID namespace. Though they share the same PID namespace, KubeVirt’s usage of unprivileged containers makes it, for example, impossible to ptrace a running container. Therefore, this technique isn’t appropriate for our needs.

Due to its security and image size reduction, KubeVirt container images are based on distroless containers. These kinds of images are extremely beneficial for deployments, but they are challenging to troubleshoot because there is no package management, which prevents the installation of additional tools on the flight.

Wrapping the QEMU binary in a script is one practical method for debugging QEMU launched by Libvirt. This script launches the QEMU as a child of this process together with the debugging tool (such as strace or valgrind).

The final part that needs to be added is the configuration for Libvirt to use the wrapped script rather than calling the QEMU program directly.

It is possible to alter the generated XML with the help of KubeVirt sidecars. This allows us to use the wrapping script in place of the built-in emulator.

The primary concept behind this configuration is that all of the additional tools, scripts, and final output files will be stored in a PerstistentVolumeClaim (PVC) that this guide refers to as debug-tools. The virt-launcher pod that we wish to debug will have this PVC attached to it.

PVC:

  1. apiVersion: v1
  2. kind: PersistentVolumeClaim
  3. metadata:
  4.   name: debug-tools
  5. spec:
  6.   accessModes:
  7.     - ReadWriteOnce
  8.   volumeMode: Filesystem
  9.   resources:
  10.     requests:
  11.       storage: 1Gi

In this guide, we’ll apply the above concepts to debug QEMU inside virt-launcher using strace without the need of build a custom virt-launcher image.

You can see a full demo of this setup: asciicast

How to bring the debug tools and wrapping script into distroless containers

This section provides an example of how to provide extra tools into the distroless container that will be supplied as a PVC using a Dockerfile. Although there are several ways to accomplish this, this covers a relatively simple technique. Alternatively, you could run a pod and manually populate the PVC by execing into the pod.

Dockerfile:

  1. FROM quay.io/centos/centos:stream9 as build
  3. ENV DIR /debug-tools
  4. RUN mkdir -p ${DIR}/logs
  6. RUN  yum install  --installroot=${DIR} -y strace && yum clean all
  8. COPY ./wrap_qemu_strace.sh $DIR/wrap_qemu_strace.sh
  9. RUN chmod 0755 ${DIR}/wrap_qemu_strace.sh
  10. RUN chown 107:107 ${DIR}/wrap_qemu_strace.sh
  11. RUN chown 107:107 ${DIR}/logs

The directory debug-tools stores the content that will be later copied inside the debug-tools PVC. We are essentially adding the missing utilities in the custom directory with yum install --installroot=${DIR}}, and the parent image matches with the parent images of virt-launcher.

The wrap_qemu_strace.sh is the wrapping script that will be used to launch QEMU with strace similarly as the example with valgrind.

  1. #!/bin/bash
  3. LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/run/debug/usr/lib64 /var/run/debug/usr/bin/strace \
  4.         -o /var/run/debug/logs/strace.out \
  5.         /usr/libexec/qemu-kvm $@

It is important to set the dynamic library path LD_LIBRARY_PATH to the path where the PVC will be mounted in the virt-launcher container.

Then, you will simply need to build the image and your debug setup is ready. The Dockerfle and the script wrap_qemu_strace.sh need to be in the same directory where you run the command.

  1. $ podman build -t debug .

The second step is to populate the PVC. This can be easily achieved using a kubernetes Job like:

  1. apiVersion: batch/v1
  2. kind: Job
  3. metadata:
  4.   name: populate-pvc
  5. spec:
  6.   template:
  7.     spec:
  8.       volumes:
  9.         - name: populate
  10.           persistentVolumeClaim:
  11.             claimName: debug-tools
  12.       containers:
  13.         - name: populate
  14.           image: registry:5000/debug:latest
  15.           command: ["sh", "-c", "cp -r /debug-tools/* /vol"]
  16.           imagePullPolicy: Always
  17.           volumeMounts:
  18.             - mountPath: "/vol"
  19.               name: populate
  20.       restartPolicy: Never
  21.   backoffLimit: 4

The image referenced in the Job is the image we built in the previous step. Once applied this and the job completed, thedebug-tools PVC is ready to be used.

How to start qemu launched by a debugging tool (e.g strace)

This part is achieved by using ConfigMaps and a KubeVirt sidecar (more details in the section Using ConfigMap to run custom script).

Configmap:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: my-config-map
  5. data:
  6.   my_script.sh: |
  7.     #!/bin/sh
  8.     tempFile=`mktemp --dry-run`
  9.     echo $4 > $tempFile
  10.     sed -i "s|<emulator>/usr/libexec/qemu-kvm</emulator>|<emulator>/var/run/debug/wrap_qemu_strace.sh</emulator>|" $tempFile
  11.     cat $tempFile

The script that replaces the QEMU binary with the wrapping script in the XML is stored in the configmap my-config-map. This script will run as a hook, as explained in full in the documentation for the KubeVirt sidecar.

Once all the objects created, we can finally run the guest to debug.

VMI:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachineInstance
  3. metadata:
  4.   annotations:
  5.     hooks.kubevirt.io/hookSidecars: '[{"args": ["--version", "v1alpha2"],
  6.     "image":"registry:5000/kubevirt/sidecar-shim:devel",
  7.     "pvc": {"name": "debug-tools","volumePath": "/debug", "sharedComputePath": "/var/run/debug"},
  8.     "configMap": {"name": "my-config-map","key": "my_script.sh", "hookPath": "/usr/bin/onDefineDomain"}}]'
  9.   labels:
  10.     special: vmi-debug-tools
  11.   name: vmi-debug-tools
  12. spec:
  13.   domain:
  14.     devices:
  15.       disks:
  16.       - disk:
  17.           bus: virtio
  18.         name: containerdisk
  19.       - disk:
  20.           bus: virtio
  21.         name: cloudinitdisk
  22.       rng: {}
  23.     resources:
  24.       requests:
  25.         memory: 1024M
  26.   terminationGracePeriodSeconds: 0
  27.   volumes:
  28.   - containerDisk:
  29.       image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel
  30.     name: containerdisk
  31.   - cloudInitNoCloud:
  32.       userData: |-
  33.         #cloud-config
  34.         password: fedora
  35.         chpasswd: { expire: False }
  36.     name: cloudinitdisk

The VMI example is a simply VM instance declaration and the interesting parts are the annotations for the hook: * image refers to the sidecar-shim already built and shipped with KubeVirt * pvc refers to the PVC populated with the debug setup. The name refers to the claim name, the volumePath is the path inside the sidecar container where the volume is mounted while the sharedComputePath is the path of the same volume inside the compute container. * configMap refers to the confimap containing the script to modify the XML for the wrapping script

Once the VM is declared, the hook will modify the emulator section and Libvirt will call the wrapping script instead of QEMU directly.

How to fetch the output

The wrapping script configures strace to store the output in the PVC. In this way, it is possible to retrieve the output file in a later time, for example using an additional pod like:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: fetch-logs
  5. spec:
  6.   securityContext:
  7.     runAsUser: 107
  8.     fsGroup: 107
  9.   volumes:
  10.     - name: populate
  11.       persistentVolumeClaim:
  12.         claimName: debug-tools
  13.   containers:
  14.     - name: populate
  15.       image: busybox:latest
  16.       command: ["tail", "-f", "/dev/null"]
  17.       volumeMounts:
  18.         - mountPath: "/vol"
  19.           name: populate

It is then possible to copy the file locally with:

  1. $ kubectl cp fetch-logs:/vol/logs/strace.out strace.out