Usage of libguestfs-tools and virtctl guestfs

Libguestfs tools are a set of utilities for accessing and modifying VM disk images. The command virtctl guestfs helps to deploy an interactive container with the libguestfs-tools and the PVC attached to it. This command is particularly useful if the users need to modify, inspect or debug VM disks on a PVC.

  1. $ virtctl guestfs -h
  2. Create a pod with libguestfs-tools, mount the pvc and attach a shell to it. The pvc is mounted under the /disks directory inside the pod for filesystem-based pvcs, or as /dev/vda for block-based pvcs
  4. Usage:
  5.   virtctl guestfs [flags]
  7. Examples:
  8.   # Create a pod with libguestfs-tools, mount the pvc and attach a shell to it:
  9.   virtctl guestfs <pvc-name>
  11. Flags:
  12.   -h, --help                 help for guestfs
  13.       --image string         libguestfs-tools container image
  14.       --kvm                  Use kvm for the libguestfs-tools container (default true)
  15.       --pull-policy string   pull policy for the libguestfs image (default "IfNotPresent")
  17. Use "virtctl options" for a list of global command-line options (applies to all commands).

By default virtctl guestfs sets up kvm for the interactive container. This considerably speeds up the execution of the libguestfs-tools since they use QEMU. If the cluster doesn’t have any kvm supporting nodes, the user must disable kvm by setting the option --kvm=false. If not set, the libguestfs-tools pod will remain pending because it cannot be scheduled on any node.

The command automatically uses the image exposed by KubeVirt under the http endpoint /apis/subresources.kubevirt.io/<kubevirt-version>/guestfs, but it can be configured to use a custom image by using the option --image. Users can also overwrite the pull policy of the image by setting the option pull-policy.

The command checks if a PVC is used by another pod in which case it will fail. However, once libguestfs-tools has started, the setup doesn’t prevent a new pod starting and using the same PVC. The user needs to verify that there are no active virtctl guestfs pods before starting the VM which accesses the same PVC.

Currently, virtctl guestfs supports only a single PVC. Future versions might support multiple PVCs attached to the interactive pod.

Examples and use-cases

Generally, the user can take advantage of the virtctl guestfs command for all typical usage of libguestfs-tools. It is strongly recommended to consult the official documentation. This command simply aims to help in configuring the correct containerized environment in the Kubernetes cluster where KubeVirt is installed.

For all the examples, the user has to start the interactive container by referencing the PVC in the virtctl guestfs command. This will deploy the interactive pod and attach the stdin and stdout.

Example:

  1. $ virtctl guestfs pvc-test
  2. Use image: registry:5000/kubevirt/libguestfs-tools@sha256:6644792751b2ba9442e06475a809448b37d02d1937dbd15ad8da4d424b5c87dd 
  3. The PVC has been mounted at /disk 
  4. Waiting for container libguestfs still in pending, reason: ContainerCreating, message:  
  5. Waiting for container libguestfs still in pending, reason: ContainerCreating, message:  
  6. bash-5.0#

Once the libguestfs-tools pod has been deployed, the user can access the disk and execute the desired commands. Later, once the user has completed the operations on the disk, simply exit the container and the pod be will automatically terminated.

  1. Inspect the disk filesystem to retrive the version of the OS on the disk:

    1. bash-5.0# virt-cat -a disk.img /etc/os-release 
    2. NAME=Fedora
    3. VERSION="34 (Cloud Edition)"
    4. ID=fedora
    5. VERSION_ID=34
    6. VERSION_CODENAME=""
    7. PLATFORM_ID="platform:f34"
    8. PRETTY_NAME="Fedora 34 (Cloud Edition)"
    9. ANSI_COLOR="0;38;2;60;110;180"
    10. LOGO=fedora-logo-icon
    11. CPE_NAME="cpe:/o:fedoraproject:fedora:34"
    12. HOME_URL="https://fedoraproject.org/"
    13. DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/"
    14. SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
    15. BUG_REPORT_URL="https://bugzilla.redhat.com/"
    16. REDHAT_BUGZILLA_PRODUCT="Fedora"
    17. REDHAT_BUGZILLA_PRODUCT_VERSION=34
    18. REDHAT_SUPPORT_PRODUCT="Fedora"
    19. REDHAT_SUPPORT_PRODUCT_VERSION=34
    20. PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
    21. VARIANT="Cloud Edition"
    22. VARIANT_ID=cloud

  2. Add users (for example after the disk has been imported using CDI)

    1. bash-5.0# virt-customize -a disk.img --run-command 'useradd -m test-user -s /bin/bash' --password  'test-user:password:test-password'
    2. [   0.0] Examining the guest ...
    3. [   4.1] Setting a random seed
    4. [   4.2] Setting the machine ID in /etc/machine-id
    5. [   4.2] Running: useradd -m test-user -s /bin/bash
    6. [   4.3] Setting passwords
    7. [   5.3] Finishing off

  3. Run virt-rescue and repair a broken partition or initrd (for example by running dracut)

    ``` bash-5.0# virt-rescue -a disk.img […] The virt-rescue escape key is ‘^]’. Type ‘^] h’ for help.

    Welcome to virt-rescue, the libguestfs rescue shell.

    Note: The contents of / (root) are the rescue appliance. You have to mount the guest’s partitions under /sysroot before you can examine them.

    fdisk -l Disk /dev/sda: 6 GiB, 6442450944 bytes, 12582912 sectors Disk model: QEMU HARDDISK
    Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: F8DC0844-9194-4B34-B432-13FA4B70F278

    Device Start End Sectors Size Type /dev/sda1 2048 4095 2048 1M BIOS boot /dev/sda2 4096 2101247 2097152 1G Linux filesystem /dev/sda3 2101248 12580863 10479616 5G Linux filesystem

  1. Disk /dev/sdb: 4 GiB, 4294967296 bytes, 8388608 sectors
  2. Disk model: QEMU HARDDISK   
  3. Units: sectors of 1 * 512 = 512 bytes
  4. Sector size (logical/physical): 512 bytes / 512 bytes
  5. I/O size (minimum/optimal): 512 bytes / 512 bytes
  6. ><rescue> mount /dev/sda3 sysroot/
  7. ><rescue> mount /dev/sda2 sysroot/boot
  8. ><rescue> chroot sysroot/
  9. ><rescue> ls boot/
  10. System.map-5.11.12-300.fc34.x86_64
  11. config-5.11.12-300.fc34.x86_64
  12. efi
  13. grub2
  14. initramfs-0-rescue-8afb5b540fab48728e48e4196a3a48ee.img
  15. initramfs-5.11.12-300.fc34.x86_64.img
  16. loader
  17. vmlinuz-0-rescue-8afb5b540fab48728e48e4196a3a48ee
  18. ><rescue> dracut -f boot/initramfs-5.11.12-300.fc34.x86_64.img 5.11.12-300.fc34.x86_64
  19. [...]
  20. ><rescue> exit # <- exit from chroot
  21. ><rescue> umount sysroot/boot
  22. ><rescue> umount sysroot
  23. ><rescue> exit
  24. ```

  1. Install an OS from scratch

    1. bash-5.0# virt-builder centos-8.2 -o disk.img --root-password password:password-test
    2. [   1.5] Downloading: http://builder.libguestfs.org/centos-8.2.xz
    3. ######################################################################## 100.0%#=#=#                                                    ######################################################################## 100.0%
    4. [  58.3] Planning how to build this image
    5. [  58.3] Uncompressing
    6. [  65.7] Opening the new disk
    7. [  70.8] Setting a random seed
    8. [  70.8] Setting passwords
    9. [  72.0] Finishing off
    10.                    Output file: disk.img
    11.                    Output size: 6.0G
    12.                  Output format: raw
    13.             Total usable space: 5.3G
    14.                     Free space: 4.0G (74%)

  2. Identify the partition and filesystem on the disk

    1. bash-5.0# virt-filesystems -a disk.img --partitions --filesystem --long
    2. Name       Type        VFS   Label  MBR  Size        Parent
    3. /dev/sda2  filesystem  ext4  -      -    1023303680  -
    4. /dev/sda4  filesystem  xfs   -      -    4710203392  -
    5. /dev/sda1  partition   -     -      -    1048576     /dev/sda
    6. /dev/sda2  partition   -     -      -    1073741824  /dev/sda
    7. /dev/sda3  partition   -     -      -    644874240   /dev/sda
    8. /dev/sda4  partition   -     -      -    4720689152  /dev/sda

Limitations

Currently, it is not possible to resize the xfs filesystem.