Export API

It can be desirable to export a Virtual Machine and its related disks out of a cluster so you can import that Virtual Machine into another system or cluster. The Virtual Machine disks are the most prominent things you will want to export. The export API makes it possible to declaratively export Virtual Machine disks. It is also possible to export individual PVCs and their contents, for instance when you have created a memory dump from a VM or are using virtio-fs to have a Virtual Machine populate a PVC.

In order not to overload the kubernetes API server the data is transferred through a dedicated export proxy server. The proxy server can then be exposed to the outside world through a service associated with an Ingress/Route or NodePort. As an alternative, the port-forward flag can be used with the virtctl integration to bypass the need of an Ingress/Route.

Export Feature Gate

VMExport support must be enabled in the feature gates to be available. The feature gates field in the KubeVirt CR must be expanded by adding the VMExport to it.

Export token

In order to securely export a Virtual Machine Disk, you must create a token that is used to authorize users accessing the export endpoint. This token must be in the same namespace as the Virtual Machine. The contents of the secret can be passed as a token header or parameter to the export URL. The name of the header or argument is x-kubevirt-export-token with a value that matches the content of the secret. The secret can be named any valid secret in the namespace. We recommend you generate an alpha numeric token of at least 12 characters. The data key should be token. For example:

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: example-token
  5. stringData:
  6.   token: 1234567890ab

Export Virtual Machine volumes

After you have created the token you can now create a VMExport CR that identifies the Virtual Machine you want to export. You can create a VMExport that looks like this:

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5. spec:
  6.   tokenSecretRef: example-token
  7.   source:
  8.     apiGroup: "kubevirt.io"
  9.     kind: VirtualMachine
  10.     name: example-vm

The following volumes present in the VM will be exported:

  • PersistentVolumeClaims
  • DataVolumes
  • MemoryDump

All other volume types are not exported. To avoid the export of inconsistent data, a Virtual Machine can only be exported while it is powered off. Any active VM exports will be terminated if the Virtual Machine is started. To export data from a running Virtual Machine you must first create a Virtual Machine Snapshot (see below).

If the VM contains multiple volumes that can be exported, each volume will get its own URL links. If the VM contains no volumes that can be exported, the VMExport will go into a Skipped phase, and no export server is started.

Export Virtual Machine Snapshot volumes

You can create a VMExport CR that identifies the Virtual Machine Snapshot you want to export. You can create a VMExport that looks like this:

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5. spec:
  6.   tokenSecretRef: example-token
  7.   source:
  8.     apiGroup: "snapshot.kubevirt.io"
  9.     kind: VirtualMachineSnapshot
  10.     name: example-vmsnapshot

When you create a VMExport based on a Virtual Machine Snapshot, the controller will attempt to create PVCs from the volume snapshots contained in Virtual Machine Snapshot. Once all the PVCs are ready, the export server will start and you can begin the export. If the Virtual Machine Snapshot contains multiple volumes that can be exported, each volume will get its own URL links. If the Virtual Machine snapshot contains no volumes that can be exported, the VMExport will go into a skipped phase, and no export server is started.

Export Persistent Volume Claim

You can create a VMExport CR that identifies the Persistent Volume Claim (PVC) you want to export. You can create a VMExport that looks like this:

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5. spec:
  6.   tokenSecretRef: example-token
  7.   source:
  8.     apiGroup: ""
  9.     kind: PersistentVolumeClaim
  10.     name: example-pvc

In this example the PVC name is example-pvc. Note the PVC doesn’t need to contain a Virtual Machine Disk, it can contain any content, but the main use case is exporting Virtual Machine Disks. After you post this yaml to the cluster, a new export server is created in the same namespace as the PVC. If the source PVC is in use by another pod (such as the virt-launcher pod) then the export will remain pending until the PVC is no longer in use. If the exporter server is active and another pod starts using the PVC, the exporter server will be terminated until the PVC is not in use anymore.

The VirtualMachineExport CR will contain a status with internal and external links to the export service. The internal links are only valid inside the cluster, and the external links are valid for external access through an Ingress or Route. The cert field will contain the CA that signed the certificate of the export server for internal links, or the CA that signed the Route or Ingress.

KubeVirt content-type

The following is an example of exporting a PVC that contains a KubeVirt disk image. The controller determines if the PVC contains a kubevirt disk by checking if there is a special annotation on the PVC, or if there is a DataVolume ownerReference on the PVC, or if the PVC has a volumeMode of block.

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5.   namespace: example
  6. spec:
  7.   source:
  8.     apiGroup: ""
  9.     kind: PersistentVolumeClaim
  10.     name: example-pvc
  11.   tokenSecretRef: example-token
  12. status:
  13.   conditions:
  14.   - lastProbeTime: null
  15.     lastTransitionTime: "2022-06-21T14:10:09Z"
  16.     reason: podReady
  17.     status: "True"
  18.     type: Ready
  19.   - lastProbeTime: null
  20.     lastTransitionTime: "2022-06-21T14:09:02Z"
  21.     reason: pvcBound
  22.     status: "True"
  23.     type: PVCReady
  24.   links:
  25.     external:
  26.       cert: |-
  27.         -----BEGIN CERTIFICATE-----
  28.         ...
  29.         -----END CERTIFICATE-----
  30.       volumes:
  31.       - formats:
  32.         - format: raw
  33.           url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img
  34.         - format: gzip
  35.           url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img.gz
  36.         name: example-disk
  37.     internal:
  38.       cert: |-
  39.         -----BEGIN CERTIFICATE-----
  40.         ...
  41.         -----END CERTIFICATE-----
  42.       volumes:
  43.       - formats:
  44.         - format: raw
  45.           url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img
  46.         - format: gzip
  47.           url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img.gz
  48.         name: example-disk
  49.   phase: Ready
  50.   serviceName: virt-export-example-export

Archive content-type

Archive content-type is automatically selected if we are unable to determine the PVC contains a KubeVirt disk. The archive will contain all the files that are in the PVC.

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5.   namespace: example
  6. spec:
  7.   source:
  8.     apiGroup: ""
  9.     kind: PersistentVolumeClaim
  10.     name: example-pvc
  11.   tokenSecretRef: example-token
  12. status:
  13.   conditions:
  14.   - lastProbeTime: null
  15.     lastTransitionTime: "2022-06-21T14:10:09Z"
  16.     reason: podReady
  17.     status: "True"
  18.     type: Ready
  19.   - lastProbeTime: null
  20.     lastTransitionTime: "2022-06-21T14:09:02Z"
  21.     reason: pvcBound
  22.     status: "True"
  23.     type: PVCReady
  24.   links:
  25.     external:
  26.       cert: |-
  27.         -----BEGIN CERTIFICATE-----
  28.         ...
  29.         -----END CERTIFICATE-----
  30.       volumes:
  31.       - formats:
  32.         - format: dir
  33.           url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example/dir
  34.         - format: tar.gz
  35.           url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example/disk.tar.gz
  36.         name: example-disk
  37.     internal:
  38.       cert: |-
  39.         -----BEGIN CERTIFICATE-----
  40.         ...
  41.         -----END CERTIFICATE-----
  42.       volumes:
  43.       - formats:
  44.         - format: dir
  45.           url: https://virt-export-example-export.example.svc/volumes/example/dir
  46.         - format: tar.gz
  47.           url: https://virt-export-example-export.example.svc/volumes/example/disk.tar.gz
  48.         name: example-disk
  49.   phase: Ready
  50.   serviceName: virt-export-example-export

Manifests

The VirtualMachine manifests can be retrieved by accessing the manifests in the VirtualMachineExport status. The all type will return the VirtualMachine manifest, any DataVolumes, and a configMap that contains the public CA certificate of the Ingress/Route of the external URL, or the CA of the export server of the internal URL. The auth-header-secret will be a secret that contains a Containerized Data Importer (CDI) compatible header. This header contains a text version of the export token.

Both internal and external links will contain a manifests field. If there are no external links, then there will not be any external manifests either. The virtualMachine manifests field is only available if the source is a VirtualMachine or VirtualMachineSnapshot. Exporting a PersistentVolumeClaim will not generate a Virtual Machine manifest.

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5.   namespace: example
  6. spec:
  7.   source:
  8.     apiGroup: ""
  9.     kind: PersistentVolumeClaim
  10.     name: example-pvc
  11.   tokenSecretRef: example-token
  12. status:
  13.   conditions:
  14.   - lastProbeTime: null
  15.     lastTransitionTime: "2022-06-21T14:10:09Z"
  16.     reason: podReady
  17.     status: "True"
  18.     type: Ready
  19.   - lastProbeTime: null
  20.     lastTransitionTime: "2022-06-21T14:09:02Z"
  21.     reason: pvcBound
  22.     status: "True"
  23.     type: PVCReady
  24.   links:
  25.     external:
  26.       ...
  27.       manifests:
  28.       - type: all
  29.         url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/external/manifests/all
  30.       - type: auth-header-secret
  31.         url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/external/manifests/secret
  32.     internal:
  33.       ...
  34.       manifests:
  35.       - type: all
  36.         url: https://virt-export-export-pvc.default.svc/internal/manifests/all
  37.       - type: auth-header-secret
  38.         url: https://virt-export-export-pvc.default.svc/internal/manifests/secret
  39.   phase: Ready
  40.   serviceName: virt-export-example-export

Format types

There are 4 format types that are possible:

  • Raw. The unaltered raw KubeVirt disk image.
  • Gzip. The raw KubeVirt disk image but gzipped to help with transferring efficiency.
  • Dir. A directory listing, allowing you to find the files contained in the PVC.
  • Tar.gz The contents of the PVC tarred and gzipped in a single file.

Raw and Gzip will be selected if the PVC is determined to be a KubeVirt disk. KubeVirt disks contain a single disk.img file (or are a block device). Dir will return a list of the files in the PVC, to download a specific file you can replace /dir in the URL with the path and file name. For instance if the PVC contains the file /example/data.txt you can replace /dir with /example/data.txt to download just data.txt file. Or you can use the tar.gz URL to get all the contents of the PVC in a tar file.

The export server certificate is valid for 7 days after which it is rotated by deleting the export server pod and associated secret and generating a new one. If for whatever reason the export server pod dies, the associated secret is also automatically deleted and a new pod and secret are generated. The VirtualMachineExport object status will be automatically updated to reflect the new certificate.

The external link certificates are associated with the Ingress/Route that points to the service created by the KubeVirt operator. The CA that signed the Ingress/Route will part of the certificates.

TTL (Time to live) for an Export

For various reasons (security being one), users should be able to specify a TTL for the VMExport objects that limits the lifetime of an export.
This is done via the ttlDuration field which accepts a k8s duration,
which defaults to 2 hours when not specified.

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.     name: example-export
  5. spec:
  6.     source:
  7.         apiGroup: "kubevirt.io"
  8.         kind: VirtualMachine
  9.         name: example-vm
  10.     tokenSecretRef: example-token
  11.     ttlDuration: 1h

virtctl integration: vmexport

The virtctl vmexport command allows users to interact with the export API in an easy-to-use way.

vmexport uses two mandatory arguments:

  • The vmexport functions (create|delete|download).
  • The VirtualMachineExport name.

These three functions are:

Create

  1. # Creates a VMExport object according to the specified flag.
  3. # The flag should either be:
  5. # --pvc, to specify the name of the pvc to export.
  6. # --snapshot, to specify the name of the VM snapshot to export.
  7. # --vm, to specify the name of the Virtual Machine to export.
  9. $ virtctl vmexport create name [flags]

Delete

  1. # Deletes the specified VMExport object.
  3. $ virtctl vmexport delete name

Download

  1. # Downloads a volume from the defined VMExport object.
  3. # The main available flags are:
  5. # --output, mandatory flag to specify the output file.
  6. # --volume, optional flag to specify the name of the downloadable volume.
  7. # --vm|--snapshot|--pvc, if specified, are used to create the VMExport object assuming it doesn't exist. The name of the object to export has to be specified.
  8. # --format, optional flag to specify wether to download the file in compressed (default) or raw format.
  9. # --port-forward, optional flag to easily download the volume without the need of an ingress or route. Also, the local port can be optionally specified with the --local-port flag.
  11. $ virtctl vmexport download name [flags]

By default, the volume will be downloaded in compressed format. Users can specify the desired format (gzip or raw) by using the format flag, as shown below:

  1. # Downloads a volume from the defined VMExport object and, if necessary, decompresses it.
  2. $ virtctl vmexport download name --format=raw [flags]

TTL (Time to live)

TTL can also be added when creating a VMExport via virtctl

  1. $ virtctl vmexport create name --ttl=1h

For more information about usage and examples:

  1. $ virtctl vmexport --help
  3. Export a VM volume.
  5. Usage:
  6.   virtctl vmexport [flags]
  8. Examples:
  9.   # Create a VirtualMachineExport to export a volume from a virtual machine:
  10.     virtctl vmexport create vm1-export --vm=vm1
  12.     # Create a VirtualMachineExport to export a volume from a virtual machine snapshot
  13.     virtctl vmexport create snap1-export --snapshot=snap1
  15.     # Create a VirtualMachineExport to export a volume from a PVC
  16.     virtctl vmexport create pvc1-export --pvc=pvc1
  18.     # Delete a VirtualMachineExport resource
  19.     virtctl vmexport delete snap1-export
  21.     # Download a volume from an already existing VirtualMachineExport (--volume is optional when only one volume is available)
  22.     virtctl vmexport download vm1-export --volume=volume1 --output=disk.img.gz
  24.     # Create a VirtualMachineExport and download the requested volume from it
  25.     virtctl vmexport download vm1-export --vm=vm1 --volume=volume1 --output=disk.img.gz
  27. Flags:
  28.   -h, --help              help for vmexport
  29.       --insecure          When used with the 'download' option, specifies that the http request should be insecure.
  30.       --keep-vme          When used with the 'download' option, specifies that the vmexport object should not be deleted after the download finishes.
  31.       --output string     Specifies the output path of the volume to be downloaded.
  32.       --pvc string        Sets PersistentVolumeClaim as vmexport kind and specifies the PVC name.
  33.       --snapshot string   Sets VirtualMachineSnapshot as vmexport kind and specifies the snapshot name.
  34.       --vm string         Sets VirtualMachine as vmexport kind and specifies the vm name.
  35.       --volume string     Specifies the volume to be downloaded.
  37. Use "virtctl options" for a list of global command-line options (applies to all commands).

Use cases

Clone VM from one cluster to another cluster

If you want to transfer KubeVirt disk images from a source cluster to another target cluster, you can use the VMExport in the source to expose the disks and use Containerized Data Importer (CDI) in the target cluster to import the image into the target cluster. Let’s assume we have an Ingress or Route in the source cluster that exposes the export proxy with the following example domain virt-exportproxy-example.example.com and we have a Virtual Machine in the source cluster with one disk, which looks like this:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   labels:
  5.     kubevirt.io/vm: vm-example-datavolume
  6.   name: example-vm
  7. spec:
  8.   dataVolumeTemplates:
  9.   - metadata:
  10.       creationTimestamp: null
  11.       name: example-dv
  12.     spec:
  13.       storage:
  14.         accessModes:
  15.         - ReadWriteOnce
  16.         resources:
  17.           requests:
  18.             storage: 20Gi
  19.         storageClassName: local
  20.       source:
  21.         registry:
  22.           url: docker://quay.io/containerdisks/centos-stream:9
  23.   running: false
  24.   template:
  25.     metadata:
  26.       labels:
  27.         kubevirt.io/vm: vm-example-datavolume
  28.     spec:
  29.       domain:
  30.         devices:
  31.           disks:
  32.           - disk:
  33.               bus: virtio
  34.             name: datavolumedisk1
  35.         resources:
  36.           requests:
  37.             memory: 2Gi
  38.       terminationGracePeriodSeconds: 0
  39.       volumes:
  40.       - dataVolume:
  41.           name: example-dv
  42.         name: datavolumedisk1

This is a VM that has a DataVolume (DV) example-dv that is populated from a container disk and we want to export that disk to the target cluster. To export this VM we have to create a token that we can use in the target cluster to get access to the export, or we can let the export controller generate one for us. For example

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: example-token
  5. stringData:
  6.   token: 1234567890ab

The value of the token is 1234567890ab hardly a secure token, but it is an example. We can now create a VMExport that looks like this:

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5. spec:
  6.   tokenSecretRef: example-token #optional, if omitted the export controller will generate a token
  7.   source:
  8.     apiGroup: "kubevirt.io"
  9.     kind: VirtualMachine
  10.     name: example-vm

If the VM is not running the status of the VMExport object will get updated once the export-server pod is running to look something like this:

  1. apiVersion: export.kubevirt.io/v1alpha1
  2. kind: VirtualMachineExport
  3. metadata:
  4.   name: example-export
  5.   namespace: example
  6. spec:
  7.   tokenSecretRef: example-token
  8.   source:
  9.     apiGroup: "kubevirt.io"
  10.     kind: VirtualMachine
  11.     name: example-vm
  12. status:
  13.   conditions:
  14.   - lastProbeTime: null
  15.     reason: podReady
  16.     status: "True"
  17.     type: Ready
  18.   - lastProbeTime: null
  19.     reason: pvcBound
  20.     status: "True"
  21.     type: PVCReady
  22.   links:
  23.     external:
  24.       cert: |-
  25.         -----BEGIN CERTIFICATE-----
  26.         ...
  27.         -----END CERTIFICATE-----
  28.       volumes:
  29.       - formats:
  30.         - format: raw
  31.           url: https://virt-exportproxy-example.example.com/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example-dv/disk.img
  32.         - format: gzip
  33.           url: https://virt-exportproxy-example.example.com/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example-dv/disk.img.gz
  34.         name: example-disk
  35.     internal:
  36.       cert: |-
  37.         -----BEGIN CERTIFICATE-----
  38.         ...
  39.         -----END CERTIFICATE-----
  40.       volumes:
  41.       - formats:
  42.         - format: raw
  43.           url: https://virt-export-example-export.example.svc/volumes/example-dv/disk.img
  44.         - format: gzip
  45.           url: https://virt-export-example-export.example.svc/volumes/example-dv/disk.img.gz
  46.         name: example-disk
  47.   phase: Ready
  48.   serviceName: virt-export-example-export

Note in this example we are in the example namespace in the source cluster, which is why the internal links domain ends with .example.svc. The external links are what will be visible to outside of the source cluster, so we can use that for when we import into the target cluster.

Now we are ready to import this disk into the target cluster. In order for CDI to import, we will need to provide appropriate yaml that contains the following: - CA cert (as config map) - The token needed to access the disk images in a CDI compatible format - The VM yaml - DataVolume yaml (optional if not part of the VM definition)

virtctl provides an additional argument to the download command called --manifest that will retrieve the appropriate information from the export server, and either save it to a file with the --output argument or write to standard out. By default this output will not contain the header secret as it contains the token in plaintext. To get the header secret you specify the --include-secret argument. The default output format is yaml but it is possible to get json output as well.

Assuming there is a running VirtualMachineExport called example-export and the same namespace exists in the target cluster. The name of the kubeconfig of the target cluster is named kubeconfig-target, to clone the vm into the target cluster run the following commands:

  1. $ virtctl vmexport download example-export --manifest --include-secret --output=import.yaml
  2. $ kubectl apply -f import.yaml --kubeconfig=kubeconfig-target

The first command generates the yaml and writes it to import.yaml. The second command applies the generated yaml to the target cluster. It is possible to combine the two commands writing to standard out with the first command, and piping it into the second command. Use this option if the export token should not be written to a file anywhere. This will create the VM in the target cluster, and provides CDI in the target cluster with everything required to import the disk images.

After the import completes you should be able to start the VM in the target cluster.

Download a VM volume locally using virtctl vmexport

Several steps from the previous section can be simplified considerably by using the vmexport command.

Again, let’s assume we have an Ingress or Route in our cluster that exposes the export proxy, and that we have a Virtual Machine in the cluster with one disk like this:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   labels:
  5.     kubevirt.io/vm: vm-example-datavolume
  6.   name: example-vm
  7. spec:
  8.   dataVolumeTemplates:
  9.   - metadata:
  10.       creationTimestamp: null
  11.       name: example-dv
  12.     spec:
  13.       storage:
  14.         accessModes:
  15.         - ReadWriteOnce
  16.         resources:
  17.           requests:
  18.             storage: 20Gi
  19.         storageClassName: local
  20.       source:
  21.         registry:
  22.           url: docker://quay.io/containerdisks/centos-stream:9
  23.   running: false
  24.   template:
  25.     metadata:
  26.       labels:
  27.         kubevirt.io/vm: vm-example-datavolume
  28.     spec:
  29.       domain:
  30.         devices:
  31.           disks:
  32.           - disk:
  33.               bus: virtio
  34.             name: datavolumedisk1
  35.         resources:
  36.           requests:
  37.             memory: 2Gi
  38.       terminationGracePeriodSeconds: 0
  39.       volumes:
  40.       - dataVolume:
  41.           name: example-dv
  42.         name: datavolumedisk1

Once we meet these requirements, the process of downloading the volume locally can be accomplished by different means:

Performing each step separately

We can download the volume by performing every single step in a different command. We start by creating the export object:

  1. # We use an arbitrary name for the VMExport object, but specify our VM name in the flag.
  3. $ virtctl vmexport create vmexportname --vm=example-vm

Then, we download the volume in the specified output:

  1. # Since our virtual machine only has one volume, there's no need to specify the volume name with the --volume flag.
  3. # After the download, the VMExport object is deleted by default, so we are using the optional --keep-vme flag to delete it manually.
  5. $ virtctl vmexport download vmexportname --output=/tmp/disk.img --keep-vme

Lastly, we delete the VMExport object:

  1. $ virtctl vmexport delete vmexportname
Performing one single step

All the previous steps can be simplified in one, single command:

  1. # Since we are using a create flag (--vm) with download, the command creates the object assuming the VMExport doesn't exist.
  3. # Also, since we are not using --keep-vme, the VMExport object is deleted after the download.
  5. $ virtctl vmexport download vmexportname --vm=example-vm --output=/tmp/disk.img

After the download finishes, we can find our disk in /tmp/disk.img.