Live Migration

Live migration is a process during which a running Virtual Machine Instance moves to another compute node while the guest workload continues to run and remain accessible.

Enabling the live-migration support

Live migration must be enabled in the feature gates to be supported. The feature gates field in the KubeVirt CR must be expanded by adding the LiveMigration to it.

Limitations

  • Virtual machines using a PersistentVolumeClaim (PVC) must have a shared ReadWriteMany (RWX) access mode to be live migrated.

  • Live migration is not allowed with a pod network binding of bridge interface type ()

  • Live migration requires ports 49152, 49153 to be available in the virt-launcher pod. If these ports are explicitly specified in masquarade interface, live migration will not function.

Initiate live migration

Live migration is initiated by posting a VirtualMachineInstanceMigration (VMIM) object to the cluster. The example below starts a migration process for a virtual machine instance vmi-fedora

  1. apiVersion: kubevirt.io/v1alpha3
  2. kind: VirtualMachineInstanceMigration
  3. metadata:
  4.   name: migration-job
  5. spec:
  6.   vmiName: vmi-fedora

Migration Status Reporting

Condition and migration method

When starting a virtual machine instance, it has also been calculated whether the machine is live migratable. The result is being stored in the VMI VMI.status.conditions. The calculation can be based on multiple parameters of the VMI, however, at the moment, the calculation is largely based on the Access Mode of the VMI volumes. Live migration is only permitted when the volume access mode is set to ReadWriteMany. Requests to migrate a non-LiveMigratable VMI will be rejected.

The reported Migration Method is also being calculated during VMI start. BlockMigration indicates that some of the VMI disks require copying from the source to the destination. LiveMigration means that only the instance memory will be copied.

  1. Status:
  2.   Conditions:
  3.     Status: True
  4.     Type: LiveMigratable
  5.   Migration Method: BlockMigration

Migration Status

The migration progress status is being reported in the VMI VMI.status. Most importantly, it indicates whether the migration has been Completed or if it Failed.

Below is an example of a successful migration.

  1. Migration State:
  2.     Completed:        true
  3.     End Timestamp:    2019-03-29T03:37:52Z
  4.     Migration Config:
  5.       Completion Timeout Per GiB:  800
  6.       Progress Timeout:             150
  7.     Migration UID:                  c64d4898-51d3-11e9-b370-525500d15501
  8.     Source Node:                    node02
  9.     Start Timestamp:                2019-03-29T04:02:47Z
  10.     Target Direct Migration Node Ports:
  11.       35001:                      0
  12.       41068:                      49152
  13.       38284:                      49153
  14.     Target Node:                  node01
  15.     Target Node Address:          10.128.0.46
  16.     Target Node Domain Detected:  true
  17.     Target Pod:                   virt-launcher-testvmimcbjgw6zrzcmp8wpddvztvzm7x2k6cjbdgktwv8tkq

Cancel live migration

Live migration can also be canceled by simply deleting the migration object. A successfully aborted migration will indicate that the abort has been requested Abort Requested, and that it succeeded: Abort Status: Succeeded. The migration in this case will be Completed and Failed.

  1. Migration State:
  2.     Abort Requested:  true
  3.     Abort Status:     Succeeded
  4.     Completed:        true
  5.     End Timestamp:    2019-03-29T04:02:49Z
  6.     Failed:           true
  7.     Migration Config:
  8.       Completion Timeout Per GiB:  800
  9.       Progress Timeout:             150
  10.     Migration UID:                  57a693d6-51d7-11e9-b370-525500d15501
  11.     Source Node:                    node02
  12.     Start Timestamp:                2019-03-29T04:02:47Z
  13.     Target Direct Migration Node Ports:
  14.       39445:                      0
  15.       43345:                      49152
  16.       44222:                      49153
  17.     Target Node:                  node01
  18.     Target Node Address:          10.128.0.46
  19.     Target Node Domain Detected:  true
  20.     Target Pod:                   virt-launcher-testvmimcbjgw6zrzcmp8wpddvztvzm7x2k6cjbdgktwv8tkq

Changing Cluster Wide Migration Limits

KubeVirt puts some limits in place, so that migrations don’t overwhelm the cluster. By default, it is configured to only run 5 migrations in parallel with an additional limit of a maximum of 2 outbound migrations per node. Finally, every migration is limited to a bandwidth of 64MiB/s.

These values can be change in the kubevirt CR:

  1.     apiVersion: kubevirt.io/v1alpha3
  2.     kind: Kubevirt
  3.     metadata:
  4.       name: kubevirt
  5.       namespace: kubevirt
  6.     spec:
  7.       configuration:
  8.         developerConfiguration:
  9.           featureGates:
  10.             - "LiveMigration"
  11.         migrationConfiguration:
  12.           parallelMigrationsPerCluster: 5
  13.           parallelOutboundMigrationsPerNode: 2
  14.           bandwidthPerMigration: 64Mi
  15.           completionTimeoutPerGiB: 800
  16.           progressTimeout: 150
  17.           disableTLS: false

Migration timeouts

Depending on the type, the live migration process will copy virtual machine memory pages and disk blocks to the destination. During this process non-locked pages and blocks are being copied and become free for the instance to use again. To achieve a successful migration, it is assumed that the instance will write to the free pages and blocks (pollute the pages) at a lower rate than these are being copied.

Completion time

In some cases the virtual machine can write to different memory pages / disk blocks at a higher rate than these can be copied, which will prevent the migration process from completing in a reasonable amount of time. In this case, live migration will be aborted if it is running for a long period of time. The timeout is calculated base on the size of the VMI, it’s memory and the ephemeral disks that are needed to be copied. The configurable parameter completionTimeoutPerGiB, which defaults to 800s is the time for GiB of data to wait for the migration to be completed before aborting it. A VMI with 8Gib of memory will time out after 6400 seconds.

Progress timeout

Live migration will also be aborted when it will be noticed that copying memory doesn’t make any progress. The time to wait for live migration to make progress in transferring data is configurable by progressTimeout parameter, which defaults to 150s

Disabling secure migrations

FEATURE STATE: KubeVirt v0.43

Sometimes it may be desirable to disable TLS encryption of migrations to improve performance. Use disableTLS to do that:

  1.     apiVersion: kubevirt.io/v1
  2.     kind: Kubevirt
  3.     metadata:
  4.       name: kubevirt
  5.       namespace: kubevirt
  6.     spec:
  7.       configuration:
  8.         developerConfiguration:
  9.           featureGates:
  10.             - "LiveMigration"
  11.         migrationConfiguration:
  12.           disableTLS: true

Note: While this increases perfomance it may allow MITM attacks. Be careful.