About Working with Virtual Machines

CloudStack provides administrators with complete control over the lifecycle of all guest VMs executing in the cloud. CloudStack provides several guest management operations for end users and administrators. VMs may be stopped, started, rebooted, and destroyed.

Guest VMs have a name and group. VM names and groups are opaque to CloudStack and are available for end users to organize their VMs. Each VM can have three names for use in different contexts. Only two of these names can be controlled by the user:

• Instance name – a unique, immutable ID that is generated by CloudStack and can not be modified by the user. This name conforms to the requirements in IETF RFC 1123.
• Display name – the name displayed in the CloudStack web UI. Can be set by the user. Defaults to instance name.
• Name – host name that the DHCP server assigns to the VM. Can be set by the user. Defaults to instance name

Note

You can append the display name of a guest VM to its internal name. For more information, see “Appending a Display Name to the Guest VM’s Internal Name”.

Guest VMs can be configured to be Highly Available (HA). An HA-enabled VM is monitored by the system. If the system detects that the VM is down, it will attempt to restart the VM, possibly on a different host. For more information, see HA-Enabled Virtual Machines on

Each new VM is allocated one public IP address. When the VM is started, CloudStack automatically creates a static NAT between this public IP address and the private IP address of the VM.

If elastic IP is in use (with the NetScaler load balancer), the IP address initially allocated to the new VM is not marked as elastic. The user must replace the automatically configured IP with a specifically acquired elastic IP, and set up the static NAT mapping between this new IP and the guest VM’s private IP. The VM’s original IP address is then released and returned to the pool of available public IPs. Optionally, you can also decide not to allocate a public IP to a VM in an EIP-enabled Basic zone. For more information on Elastic IP, see “About Elastic IP”.

CloudStack cannot distinguish a guest VM that was shut down by the user (such as with the “shutdown” command in Linux) from a VM that shut down unexpectedly. If an HA-enabled VM is shut down from inside the VM, CloudStack will restart it. To shut down an HA-enabled VM, you must go through the CloudStack UI or API.

Note

Monitor VMs for Max Capacity

The CloudStack administrator should monitor the total number of VM instances in each cluster, and disable allocation to the cluster if the total is approaching the maximum that the hypervisor can handle. Be sure to leave a safety margin to allow for the possibility of one or more hosts failing, which would increase the VM load on the other hosts as the VMs are automatically redeployed. Consult the documentation for your chosen hypervisor to find the maximum permitted number of VMs per host, then use CloudStack global configuration settings to set this as the default limit. Monitor the VM activity in each cluster at all times. Keep the total number of VMs below a safe level that allows for the occasional host failure. For example, if there are N hosts in the cluster, and you want to allow for one host in the cluster to be down at any given time, the total number of VM instances you can permit in the cluster is at most (N-1) * (per-host-limit). Once a cluster reaches this number of VMs, use the CloudStack UI to disable allocation of more VMs to the cluster.

VM Lifecycle

Virtual machines can be in the following states:

• Created
• Running
• Stopped
• Destroyed
• Expunged

With the intermediate states of

• Creating
• Starting
• Stopping
• Expunging

Creating VMs

Virtual machines are usually created from a template. Users can also create blank virtual machines. A blank virtual machine is a virtual machine without an OS template. Users can attach an ISO file and install the OS from the CD/DVD-ROM.

Note

You can create a VM without starting it. You can determine whether the VM needs to be started as part of the VM deployment. A request parameter, startVM, in the deployVm API provides this feature. For more information, see the Developer’s Guide.

To create a VM from a template:

1. Log in to the CloudStack UI as an administrator or user.

2. In the left navigation bar, click Instances.

3. Click Add Instance.

4. Select a zone. Admin users will have the option to select a pod, cluster or host.

5. Select a template, then follow the steps in the wizard. For more information about how the templates came to be in this list, see *Working with Templates*.

6. Be sure that the hardware you have allows starting the selected service offering.

   Note

   VMware only: If the selected template contains OVF properties, different deployment options or configurations, multiple NICs or end-user license agreements, then the wizard will display these properties.

   See “Support for Virtual Appliances”.

7. Click Submit and your VM will be created and started.

   Note

   For security reason, the internal name of the VM is visible only to the root admin.

To create a VM from an ISO:

Note

XenServer

Windows VMs running on XenServer require PV drivers, which may be provided in the template or added after the VM is created. The PV drivers are necessary for essential management functions such as mounting additional volumes and ISO images, live migration, and graceful shutdown.

1. Log in to the CloudStack UI as an administrator or user.
2. In the left navigation bar, click Instances.
3. Click Add Instance.
4. Select a zone. Admin users will have the option to select a pod, cluster or host.
5. Select ISO Boot, and follow the steps in the wizard.
6. Click Submit and your VM will be created and started.

Install Required Tools and Drivers

Be sure the following are installed on each VM:

• For XenServer, install PV drivers and Xen tools on each VM. This will enable live migration and clean guest shutdown. Xen tools are required in order for dynamic CPU and RAM scaling to work.
• For vSphere, install VMware Tools on each VM. This will enable console view to work properly. VMware Tools are required in order for dynamic CPU and RAM scaling to work.

To be sure that Xen tools or VMware Tools is installed, use one of the following techniques:

• Create each VM from a template that already has the tools installed; or,
• When registering a new template, the administrator or user can indicate whether tools are installed on the template. This can be done through the UI or using the updateTemplate API; or,
• If a user deploys a virtual machine with a template that does not have Xen tools or VMware Tools, and later installs the tools on the VM, then the user can inform CloudStack using the updateVirtualMachine API. After installing the tools and updating the virtual machine, stop and start the VM.

Accessing VMs

Any user can access their own virtual machines. The administrator can access all VMs running in the cloud.

To access a VM through the CloudStack UI:

1. Log in to the CloudStack UI as a user or admin.
2. Click Instances, then click the name of a running VM.
3. Click the View Console button .

To access a VM directly over the network:

1. The VM must have some port open to incoming traffic. For example, in a basic zone, a new VM might be assigned to a security group which allows incoming traffic. This depends on what security group you picked when creating the VM. In other cases, you can open a port by setting up a port forwarding policy. See “IP Forwarding and Firewalling”.
2. If a port is open but you can not access the VM using ssh, it’s possible that ssh is not already enabled on the VM. This will depend on whether ssh is enabled in the template you picked when creating the VM. Access the VM through the CloudStack UI and enable ssh on the machine using the commands for the VM’s operating system.
3. If the network has an external firewall device, you will need to create a firewall rule to allow access. See “IP Forwarding and Firewalling”.

Stopping and Starting VMs

Once a VM instance is created, you can stop, restart, or delete it as needed. In the CloudStack UI, click Instances, select the VM, and use the Stop, Start, Reboot, and Destroy buttons.

A stop will attempt to gracefully shut down the operating system, via an ACPI ‘stop’ command which is similar to pressing the soft power switch on a physical server. If the operating system cannot be stopped, it will be forcefully terminated. This has the same effect as pulling out the power cord from a physical machine.

A reboot should not be considered as a stop followed by a start. In CloudStack, a start command reconfigures the virtual machine to the stored parameters in CloudStack’s database. The reboot process does not do this.

When starting a VM, admin users have the option to specify a pod, cluster, or host.

Deleting VMs

Users can delete their own virtual machines. A running virtual machine will be abruptly stopped before it is deleted. Administrators can delete any virtual machines.

To delete a virtual machine:

1. Log in to the CloudStack UI as a user or admin.
2. In the left navigation, click Instances.
3. Choose the VM that you want to delete.
4. Click the Destroy Instance button.
5. Optionally both expunging and the deletion of any attached volumes can be enabled.

When a virtual machine is destroyed, it can no longer be seen by the end user, however, it can be seen (and recovered) by a root admin. In this state it still consumes logical resources. Global settings control the maximum time from a VM being destroyed, to the physical disks being removed. When the VM and its rooot disk have been deleted, the VM is said to have been expunged.

Once a virtual machine is expunged, it cannot be recovered. All the resources used by the virtual machine will be reclaimed by the system, This includes the virtual machine’s IP address.

Managing Virtual Machines

Changing the VM Name, OS, or Group

After a VM is created, you can modify the display name, operating system, and the group it belongs to.

To access a VM through the CloudStack UI:

1. Log in to the CloudStack UI as a user or admin.
2. In the left navigation, click Instances.
3. Select the VM that you want to modify.
4. Click the Stop button to stop the VM.
5. Click Edit.
6. Make the desired changes to the following:
7. Display name: Enter a new display name if you want to change the name of the VM.
8. OS Type: Select the desired operating system.
9. Group: Enter the group name for the VM.
10. Click Apply.

Appending a Display Name to the Guest VM’s Internal Name

Every guest VM has an internal name. The host uses the internal name to identify the guest VMs. CloudStack gives you an option to provide a guest VM with a display name. You can set this display name as the internal name so that the vCenter can use it to identify the guest VM. A new global parameter, vm.instancename.flag, has now been added to achieve this functionality.

The default format of the internal name is i-<user_id>-<vm_id>-<instance.name>, where instance.name is a global parameter. However, If vm.instancename.flag is set to true, and if a display name is provided during the creation of a guest VM, the display name is appended to the internal name of the guest VM on the host. This makes the internal name format as i-<user_id>-<vm_id>-<displayName>. The default value of vm.instancename.flag is set to false. This feature is intended to make the correlation between instance names and internal names easier in large data center deployments.

The following table explains how a VM name is displayed in different scenarios.

Changing the Service Offering for a VM

To upgrade or downgrade the level of compute resources available to a virtual machine, you can change the VM’s compute offering.

1. Log in to the CloudStack UI as a user or admin.

2. In the left navigation, click Instances.

3. Choose the VM that you want to work with.

4. (Skip this step if you have enabled dynamic VM scaling; see CPU and Memory Scaling for Running VMs.)

   Click the Stop button to stop the VM.

5. Click the Change Service button.

   The Change service dialog box is displayed.

6. Select the offering you want to apply to the selected VM.

7. Click OK.

CPU and Memory Scaling for Running VMs

(Supported on VMware and XenServer)

It is not always possible to accurately predict the CPU and RAM requirements when you first deploy a VM. You might need to increase these resources at any time during the life of a VM. You can dynamically modify CPU and RAM levels to scale up these resources for a running VM without incurring any downtime.

Dynamic CPU and RAM scaling can be used in the following cases:

• User VMs on hosts running VMware and XenServer.
• System VMs on VMware.
• VMware Tools or XenServer Tools must be installed on the virtual machine.
• The new requested CPU and RAM values must be within the constraints allowed by the hypervisor and the VM operating system.
• New VMs that are created after the installation of CloudStack 4.2 can use the dynamic scaling feature. If you are upgrading from a previous version of CloudStack, your existing VMs created with previous versions will not have the dynamic scaling capability unless you update them using the following procedure.

Updating Existing VMs

If you are upgrading from a previous version of CloudStack, and you want your existing VMs created with previous versions to have the dynamic scaling capability, update the VMs using the following steps:

1. Make sure the zone-level setting enable.dynamic.scale.vm is set to true. In the left navigation bar of the CloudStack UI, click Infrastructure, then click Zones, click the zone you want, and click the Settings tab.
2. Install Xen tools (for XenServer hosts) or VMware Tools (for VMware hosts) on each VM if they are not already installed.
3. Stop the VM.
4. Click the Edit button.
5. Click the Dynamically Scalable checkbox.
6. Click Apply.
7. Restart the VM.

Configuring Dynamic CPU and RAM Scaling

To configure this feature, use the following new global configuration variables:

• enable.dynamic.scale.vm: Set to True to enable the feature. By default, the feature is turned off.
• scale.retry: How many times to attempt the scaling operation. Default = 2.

How to Dynamically Scale CPU and RAM

To modify the CPU and/or RAM capacity of a virtual machine, you need to change the compute offering of the VM to a new compute offering that has the desired CPU and RAM values. You can use the same steps described above in “Changing the Service Offering for a VM”, but skip the step where you stop the virtual machine. Of course, you might have to create a new compute offering first.

When you submit a dynamic scaling request, the resources will be scaled up on the current host if possible. If the host does not have enough resources, the VM will be live migrated to another host in the same cluster. If there is no host in the cluster that can fulfill the requested level of CPU and RAM, the scaling operation will fail. The VM will continue to run as it was before.

Limitations

• You can not do dynamic scaling for system VMs on XenServer.
• CloudStack will not check to be sure that the new CPU and RAM levels are compatible with the OS running on the VM.
• When scaling memory or CPU for a Linux VM on VMware, you might need to run scripts in addition to the other steps mentioned above. For more information, see Hot adding memory in Linux (1012764) in the VMware Knowledge Base.
• (VMware) If resources are not available on the current host, scaling up will fail on VMware because of a known issue where CloudStack and vCenter calculate the available capacity differently. For more information, see https://issues.apache.org/jira/browse/CLOUDSTACK-1809.
• On VMs running Linux 64-bit and Windows 7 32-bit operating systems, if the VM is initially assigned a RAM of less than 3 GB, it can be dynamically scaled up to 3 GB, but not more. This is due to a known issue with these operating systems, which will freeze if an attempt is made to dynamically scale from less than 3 GB to more than 3 GB.

Resetting the Virtual Machine Root Volume on Reboot

For secure environments, and to ensure that VM state is not persisted across reboots, you can reset the root disk. For more information, see “Reset VM to New Root Disk on Reboot”.

Moving VMs Between Hosts (Manual Live Migration)

The CloudStack administrator can move a running VM from one host to another without interrupting service to users or going into maintenance mode. This is called manual live migration, and can be done under the following conditions:

• The root administrator is logged in. Domain admins and users can not perform manual live migration of VMs.
• The VM is running. Stopped VMs can not be live migrated.
• The destination host must have enough available capacity. If not, the VM will remain in the “migrating” state until memory becomes available.
• (KVM) The VM must not be using local disk storage. (On XenServer and VMware, VM live migration with local disk is enabled by CloudStack support for XenMotion and vMotion.)
• (KVM) The destination host must be in the same cluster as the original host. (On XenServer and VMware, VM live migration from one cluster to another is enabled by CloudStack support for XenMotion and vMotion.)

To manually live migrate a virtual machine

1. Log in to the CloudStack UI as a user or admin.

2. In the left navigation, click Instances.

3. Choose the VM that you want to migrate.

4. Click the Migrate Instance button.

5. From the list of suitable hosts, choose the one to which you want to move the VM.

   Note

   If the VM’s storage has to be migrated along with the VM, this will be noted in the host list. CloudStack will take care of the storage migration for you.

6. Click OK.

Note

(KVM) If the VM’s storage has to be migrated along with the VM, from a mounted NFS storage pool to a cluster-wide mounted NFS storage pool, then the ‘migrateVirtualMachineWithVolume’ API has to be used. There is no UI integration for this feature.

(CloudMonkey) > migrate virtualmachinewithvolume virtualmachineid=<virtual machine uuid> hostid=<destination host uuid> migrateto[i].volume=<virtual machine volume number i uuid> migrateto[i].pool=<destination storage pool uuid for volume number i>

where i in [0,..,N] and N = number of volumes of the virtual machine

Assigning VMs to Hosts

At any point in time, each virtual machine instance is running on a single host. How does CloudStack determine which host to place a VM on? There are several ways:

• Automatic default host allocation. CloudStack can automatically pick the most appropriate host to run each virtual machine.
• Instance type preferences. CloudStack administrators can specify that certain hosts should have a preference for particular types of guest instances. For example, an administrator could state that a host should have a preference to run Windows guests. The default host allocator will attempt to place guests of that OS type on such hosts first. If no such host is available, the allocator will place the instance wherever there is sufficient physical capacity.
• Vertical and horizontal allocation. Vertical allocation consumes all the resources of a given host before allocating any guests on a second host. This reduces power consumption in the cloud. Horizontal allocation places a guest on each host in a round-robin fashion. This may yield better performance to the guests in some cases.
• Admin users preferences. Administrators have the option to specify a pod, cluster, or host to run the VM in. CloudStack will then select a host within the given infrastructure.
• End user preferences. Users can not control exactly which host will run a given VM instance, but they can specify a zone for the VM. CloudStack is then restricted to allocating the VM only to one of the hosts in that zone.
• Host tags. The administrator can assign tags to hosts. These tags can be used to specify which host a VM should use. The CloudStack administrator decides whether to define host tags, then create a service offering using those tags and offer it to the user.
• Affinity groups. By defining affinity groups and assigning VMs to them, the user or administrator can influence (but not dictate) which VMs should run on separate hosts. This feature is to let users specify that certain VMs won’t be on the same host.
• CloudStack also provides a pluggable interface for adding new allocators. These custom allocators can provide any policy the administrator desires.

Affinity Groups

By defining affinity groups and assigning VMs to them, the user or administrator can influence (but not dictate) which VMs should run on separate hosts. This feature is to let users specify that VMs with the same “host anti-affinity” type won’t be on the same host. This serves to increase fault tolerance. If a host fails, another VM offering the same service (for example, hosting the user’s website) is still up and running on another host.

The scope of an affinity group is per user account.

Creating a New Affinity Group

To add an affinity group:

1. Log in to the CloudStack UI as an administrator or user.
2. In the left navigation bar, click Affinity Groups.
3. Click Add affinity group. In the dialog box, fill in the following fields:
   • Name. Give the group a name.
   • Description. Any desired text to tell more about the purpose of the group.
   • Type. The only supported type shipped with CloudStack is Host Anti-Affinity. This indicates that the VMs in this group should avoid being placed on the same host with each other. If you see other types in this list, it means that your installation of CloudStack has been extended with customized affinity group plugins.

Assign a New VM to an Affinity Group

To assign a new VM to an affinity group:

• Create the VM as usual, as described in “Creating VMs”. In the Add Instance wizard, there is a new Affinity tab where you can select the affinity group.

Change Affinity Group for an Existing VM

To assign an existing VM to an affinity group:

1. Log in to the CloudStack UI as an administrator or user.
2. In the left navigation bar, click Instances.
3. Click the name of the VM you want to work with.
4. Stop the VM by clicking the Stop button.
5. Click the Change Affinity button.

View Members of an Affinity Group

To see which VMs are currently assigned to a particular affinity group:

1. In the left navigation bar, click Affinity Groups.

2. Click the name of the group you are interested in.

3. Click View Instances. The members of the group are listed.

   From here, you can click the name of any VM in the list to access all its details and controls.

Delete an Affinity Group

To delete an affinity group:

1. In the left navigation bar, click Affinity Groups.

2. Click the name of the group you are interested in.

3. Click Delete.

   Any VM that is a member of the affinity group will be disassociated from the group. The former group members will continue to run normally on the current hosts, but if the VM is restarted, it will no longer follow the host allocation rules from its former affinity group.

Changing a VM’s Base Image

Every VM is created from a base image, which is a template or ISO which has been created and stored in CloudStack. Both cloud administrators and end users can create and modify templates, ISOs, and VMs.

In CloudStack, you can change an existing VM’s base image from one template to another, or from one ISO to another. (You can not change from an ISO to a template, or from a template to an ISO).

For example, suppose there is a template based on a particular operating system, and the OS vendor releases a software patch. The administrator or user naturally wants to apply the patch and then make sure existing VMs start using it. Whether a software update is involved or not, it’s also possible to simply switch a VM from its current template to any other desired template.

To change a VM’s base image, call the restoreVirtualMachine API command and pass in the virtual machine ID and a new template ID. The template ID parameter may refer to either a template or an ISO, depending on which type of base image the VM was already using (it must match the previous type of image). When this call occurs, the VM’s root disk is first destroyed, then a new root disk is created from the source designated in the template ID parameter. The new root disk is attached to the VM, and now the VM is based on the new template.

You can also omit the template ID parameter from the restoreVirtualMachine call. In this case, the VM’s root disk is destroyed and recreated, but from the same template or ISO that was already in use by the VM.

Advanced VM Instance Settings

Each user VM has a set of “details” associated with it (as visible via listVirtualMachine API call) - those “details” are shown on the “Settings” tab of the VM in the GUI (words “setting(s)” and “detail(s)” are here used interchangeably).

The Settings tab is always present/visible, but settings can be changed only when the VM is in a Stopped state. Some VM details/settings can be hidden via “user.vm.blacklisted.details” global setting (you can find below the list of those hidden by default).

Note

Since version 4.15, VMware VM settings for the ROOT disk controller, NIC adapter type and data disk controller are populated automatically with the values inherited from the template.

When adding a new setting or modifying the existing ones, setting names are shown/offered in a drop-down list, as well as their possible values (with the exception of boolean or numerical values).

Read-only details/settings that are hidden by default:

• rootdisksize
• cpuOvercommitRatio
• memoryOvercommitRatio
• Message.ReservedCapacityFreed.Flag

An example list of settings as well as their possible values are shown on the images below:

(VMware hypervisor)

(VMware disk controllers)

(VMware NIC models)

(KVM disk controllers)

Virtual Machine Snapshots

(Supported on VMware, XenServer and KVM (NFS only))

In addition to the existing CloudStack ability to snapshot individual VM volumes, you can take a VM snapshot to preserve all the VM’s data volumes as well as (optionally) its CPU/memory state. This is useful for quick restore of a VM. For example, you can snapshot a VM, then make changes such as software upgrades. If anything goes wrong, simply restore the VM to its previous state using the previously saved VM snapshot.

The snapshot is created using the hypervisor’s native snapshot facility. The VM snapshot includes not only the data volumes, but optionally also whether the VM is running or turned off (CPU state) and the memory contents. The snapshot is stored in CloudStack’s primary storage.

VM snapshots can have a parent/child relationship. Each successive snapshot of the same VM is the child of the snapshot that came before it. Each time you take an additional snapshot of the same VM, it saves only the differences between the current state of the VM and the state stored in the most recent previous snapshot. The previous snapshot becomes a parent, and the new snapshot is its child. It is possible to create a long chain of these parent/child snapshots, which amount to a “redo” record leading from the current state of the VM back to the original.

After VM snapshots are created, they can be tagged with a key/value pair, like many other resources in CloudStack.

KVM supports VM snapshots when using NFS shared storage. If raw block storage is used (i.e. Ceph), then VM snapshots are not possible, since there is no possibility to write RAM memory content anywhere.

If you need more information about VM snapshots on VMware, check out the VMware documentation and the VMware Knowledge Base, especially Understanding virtual machine snapshots.

Limitations on VM Snapshots

• If a VM has some stored snapshots, you can’t attach new volume to the VM or delete any existing volumes. If you change the volumes on the VM, it would become impossible to restore the VM snapshot which was created with the previous volume structure. If you want to attach a volume to such a VM, first delete its snapshots.
• VM snapshots which include both data volumes and memory can’t be kept if you change the VM’s service offering. Any existing VM snapshots of this type will be discarded.
• You can’t make a VM snapshot at the same time as you are taking a volume snapshot.
• You should use only CloudStack to create VM snapshots on hosts managed by CloudStack. Any snapshots that you make directly on the hypervisor will not be tracked in CloudStack.

Configuring VM Snapshots

The cloud administrator can use global configuration variables to control the behavior of VM snapshots. To set these variables, go through the Global Settings area of the CloudStack UI.

Using VM Snapshots

To create a VM snapshot using the CloudStack UI:

1. Log in to the CloudStack UI as a user or administrator.

2. Click Instances.

3. Click the name of the VM you want to snapshot.

4. Click the Take VM Snapshot button.

   Note

   If a snapshot is already in progress, then clicking this button will have no effect.

5. Provide a name and description. These will be displayed in the VM Snapshots list.

6. (For running VMs only) If you want to include the VM’s memory in the snapshot, click the Memory checkbox. This saves the CPU and memory state of the virtual machine. If you don’t check this box, then only the current state of the VM disk is saved. Checking this box makes the snapshot take longer.

7. Quiesce VM: check this box if you want to quiesce the file system on the VM before taking the snapshot. Not supported on XenServer when used with CloudStack-provided primary storage.

   When this option is used with CloudStack-provided primary storage, the quiesce operation is performed by the underlying hypervisor (VMware is supported). When used with another primary storage vendor’s plugin, the quiesce operation is provided according to the vendor’s implementation.

8. Click OK.

To delete a snapshot or restore a VM to the state saved in a particular snapshot:

1. Navigate to the VM as described in the earlier steps.

2. Click View VM Snapshots.

3. In the list of snapshots, click the name of the snapshot you want to work with.

4. Depending on what you want to do:

   To delete the snapshot, click the Delete button.

   To revert to the snapshot, click the Revert button.

Note

VM snapshots are deleted automatically when a VM is destroyed. You don’t have to manually delete the snapshots in this case.

Support for Virtual Appliances

About Virtual Appliances

CloudStack allows users to deploy virtual appliances on VMware such as its been made directly though vCenter. Vendors of virtual appliances for VMware often produce ‘templates’ of their appliances in an OVA format. An OVA file contain disc images, as well as the configuration data of the virtual appliance and also at times a EULA which must be acknowledged.

Virtual Appliances are supported only on VMware.

Note

Since version 4.15, all the new templates registered are treated as virtual appliance templates.

To keep the existing functionality, all the templates registered before version 4.15 are not affected by this changes.

Deployment options (configurations)

VMware templates can provide different deployment options in their OVF descriptor file. CloudStack obtains the different deployment options when the template is registered and it displays them to the users in the virtual machine deployment wizard, under the ‘Compute Offering’ section.

After the user selects a deployment option, CloudStack lists the compute offerings which match or exceed the deployment options hardware requirements for CPU and memory.

Note

All the custom unconstrained compute offerings are displayed, but only those constrained custom offerings in which the maximum or minimum requirements for CPU and memory are supported by the selected deployment option.

The ‘Compute Offering’ section will be similar to this:

Network interfaces

In case the template requires the virtual appliance to connect different network interfaces, these are displayed in the ‘Networks’ section, similar to this:

Properties

If the template contains properties that require the user input, those are being displayed on the ‘Properties’ section, similar to this:

End-user license agreements

If the template contains one or more end-user license agreements, the user must accept them prior to deploy their virtual appliance. If the license agreements are not accepted, then it is not possible to deploy a virtual appliance.

Advanced deployment settings

It is not possible to choose the boot type (BIOS, UEFI) and boot mode for virtual appliances. The boot mode and type used by the virtual appliances is defined in the template.

Unmanaging Virtual Machines

About Unmanaged Virtual Machines

As of ACS 4.14, CloudStack has the concept of unmanaged virtual machines. These are virtual machines that are on CloudStack managed hosts, but that are not in CloudStack’s database and therefore CloudStack cannot control (manage) then in any way. Previously, such VMs could exist, but CloudStack did not ‘see’ them (their existence would be reported in logs as unrecognised VMs).

From ACS 4.14 onwards, CloudStack is able to list these VMs via the listUnmanagedInstances API command and then import (also known as ingest) those unmanaged VMs via the importUnmanagedInstance API so that they become CloudStack managed guest instances

From ACS 4.15 onwards, administrators are able to unmanage guest virtual machines.

Note

This is currently only available for vSphere clusters.

Unmanaging Virtual Machines via API

Administrators are able to unmanage guest virtual machines from CloudStack. Once unmanaged, CloudStack can no longer monitor, control or administer the provisioning and orchestration related operations on a virtual machine.

To unmanage a guest virtual machine, an administrator must invoke the unmanageVirtualMachine API passing the ID of the virtual machine to unmanage. The API has the following preconditions:

• The virtual machine must not be destroyed
• The virtual machine state must be ‘Running’ or ‘Stopped’
• The virtual machine must be a VMware virtual machine

The API execution will perform the following pre-checks, failing if they are not met:

• There are no volume snapshots associated with any of the virtual machine volumes
• There is no ISO attached to the virtual machine

Note

This is currently only available for vSphere clusters.

Preserving unmanaged virtual machine NICs

The zone setting: unmanage.vm.preserve.nics can be used to preserve virtual machine NICs and its MAC addresses after unmanaging them. If set to true, the virtual machine NICs (and their MAC addresses) are preserved when unmanaging it. Otherwise, NICs are removed and MAC addresses can be reassigned.

Unmanaging virtual machine actions

• Clean up virtual machine NICs and deallocate network resources used such as IP addresses and DHCP entries on virtual routers.

  • If ‘unmanage.vm.preserve.nics’ = ‘false’ then the NICs are deallocated and removed from CloudStack
  • If ‘unmanage.vm.preserve.nics’ = ‘true’ then the NICs remain allocated and are not removed from the database. The NIC’s MAC addresses remain preserved and therefore cannot be assigned to any new NIC.

• Clean up virtual machine volumes in the CloudStack database

• Clean up virtual machine snapshots in the CloudStack database (if any)

• Revoke host access to any managed volumes attached to the VM (applicable to managed storage only)

• Clean up the virtual machine from the following:

  • Remove the virtual machine from security groups (if any)
  • Remove the virtual machine from instance groups (if any)
  • Remove firewall rules for the virtual machine (if any)
  • Remove port forwarding rules for the virtual machine (if any)
  • Remove load balancing rules for the virtual machine (if any)
  • Disable static NAT (if the virtual machine is assigned to it)
  • Remove the virtual machine from affinity groups (if any)

• Remove VM details from the CloudStack database

• Decrement the account resources count for volumes and virtual machines

• Generate usage events:

  • For volumes destroyed, with type: ‘VOLUME.DELETE’
  • For virtual machine snapshots destroyed (if any), with type: ‘VMSNAPSHOT.DELETE’ and ‘VMSNAPSHOT.OFF_PRIMARY’
  • For virtual machine NICs destroyed: with type: ‘NETWORK.OFFERING.REMOVE’
  • For the virtual machine being unmanaged: stopped and destroyed usage events (similar to the generated usage events when expunging a virtual machine), with types: ‘VM.STOP’ and ‘VM.DESTROY’, unless the VM has been already stopped before being unmanaged and in this case only ‘VM.DESTROY’ is generated.

Importing Virtual Machines

The ability to import VMs allows Cloud operators (both public and private) to onboard new tenants simply and quickly, with the minimum amount disk IO. But also can be used in disaster recovery scenarios at remote sites (if storage is replicated) and in the recreation of VMs which have been backed up (part of the code is indeed used in CloudStack’s Backup and Recovery feature).

The most complex part of importing VMs is the mapping of an unmanaged VM’s networks to CloudStack networks. As an operator could be importing tens or even hundreds of VMs, a UI for this feature has not been created as yet.

If the ‘destination’ network VLAN(s) and the requested service offerings match the existing VM, then the instance can be imported whilst it is running. If the VLANs or service offerings do not match, then the instance to be imported must be stopped. Once the instance has been added to CloudStack, starting it through CloudStack will alter the instances settings in line with those set in the CloudStack DB.

To import instances, it is imagined that a Cloud Provider will:

1. List all of the existing networks which the instances to be imported are on.
2. Create corresponding networks in CloudStack
3. Use the listUnmanagedInstances API to create a CSV of instances to be imported.
4. Where required, add metadata to the CSV such as the Account into which each VM is to associated, the network which each VM is to be attached to, the Compute Offering required for each instance, and the Disk Offering for each disk
5. Create a script that will loop through the CSV, sending the importUnmanagedInstance API command with the corresponding parameters for each instance being read from the CSV

listUnmanagedInstances API

Prerequisites to list unmanaged instances (vSphere)

In order for CloudStack to list the instances that are not managed by CloudStack on a host/cluster, the host(s) in the vSphere cluster must have been added to CloudStack. The standard prerequisites for adding a host to CloudStack apply.

API

This API will list all unmanaged VMs for a given cluster. Optionally, the vSphere name for an existing unmanaged VM can be given to retrieve VM details. The API will filter all CloudStack managed VMs, and will also filter templates that show up as VMs on vCenter.

Request parameters:

- clusterid (CloudStack UUID of cluster)
- name (vSphere instance name)

Response:

- clusterid
- hostid
- name
- osdisplayname
- memory
- powerstate
- cpuCoresPerSocket
- cpunumber
- cpuspeed
- disk
   - id
   - capacity (in bytes)
   - controller
   - controllerunit
   - imagepath
   - position
- nic
   - id
   - macaddress
   - networkname
   - vlanid
   - pcislot
   - adaptertype (when available)
   - ipaddress (Only returned when VMware tools are running on instance)

Importing Unmanaged Instances

importUnmanagedInstance API

Request parameters:

- clusterid (CloudStack UUID of cluster)
- name (vSphere instance name)
- displayname
- hostname
- account (An optional account name for the virtual machine. Must be used with domainid parameter)
- domainid (An optional domain ID for the virtual machine. Must be used with account parameter)
- projectid
- templateid
- serviceofferingid
- nicnetworklist (Map for NIC ID and corresponding Network UUID)
- nicipaddresslist (Map for NIC ID and corresponding IP address)
- datadiskofferinglist (Map for data disk ID and corresponding disk offering UUID)
- details (Map for VM details)
- migrateallowed (VM and its volumes are allowed to migrate to different host/storage pool when offering tags conflict with host/storage pool)
- forced (If true, a VM is imported despite some of its NIC's MAC addresses being already present)

Note

The forced parameter is false by default and prevents importing a VM which has a NIC containing a MAC address that has been previously assigned by CloudStack. If it is set to true, the NICs with MAC addresses which already exist in the CloudStack database have the existing MAC addresses reassigned to its NICs.

Response:

Same response as that of deployVirtualMachine API.

Prerequisites to Importing Unmanaged Instances (vSphere)

There are a few prerequisites to importing unmanaged instances into CloudStack. Largely these are simply that the networks which you are going to attach the instance in CloudStack need to already exist in CloudStack also the storage which an unmanaged instance is on (before importing) and also the storage which you wish the instance to be on after importing must already have been added to CloudStack.

VMs can be imported to isolated, shared or L2 networks. VMs can also be imported and then automatically migrated to storage in accordance with service offerings using the migrateallowed API parameter.

Dummy Template

The assumption that all guest instances in CloudStack are created from a template or ISO is hardcoded into CloudStack. This source template will not exist for instances which have been imported into CloudStack, there for a dummy template has been created in the CloudStack database. When a template ID is not supplied when importing the instance, the built-in dummy template ID will be used. As this template is only a dummy one, it will not be possible to ‘revert’ to the original template unless you specify a real template ID.

Offerings and Automatic Mapping

When importing an instance, CloudStack needs to attach the virtual network interfaces (vNICs) to CloudStack networks. vNICs are associated with a network in one of two ways.

1. Automatically (available for L2 and shared networks)
2. Manual assignment of vNIC to network (ID) as a map if a VM has more that one NIC

In an enterprise, the vast majority of networks will operate as Layer 2 networks with IP addressing handled by an IPAM system such as Active Directory or InfoBlox. This makes CloudStack’s L2 networks the natural choice for a like-for-like migration/on-boarding of VMs.

When importing an instance to a shared or L2 network, CloudStack will automatically look for a CloudStack network that has the same VLAN(s) as the instance’s NIC(s) is already on. This can be overridden by providing a network_id for the ‘nicnetworklist’ parameter

Note

this includes PVLANs on L2 networks.

To assigning a specific IP address to a NIC, the ‘nicipaddresslist’ parameter is used. This parameter should not be used for L2 networks, and is optional for shared networks. To ask CloudStack to assign an instance’s existing IP when importing, a value of auto can be used.

nicipaddresslist[0].nic=NIC_ID nicipaddresslist[0].ip4Address=auto

Auto-assigning IP addresses requires VMware tools to be on the guest instance (for the IP to be reported to vCenter) and is not supported if an unmanaged VM reports more than one IP address associated with its NIC (CloudStack cannot tell which is the primary address). For instances with more than 1 IP addresses per NIC, pass the first IP address via the import API and then add secondary addresses via the ‘addIpToNic’ API

Custom vs Fixed Offerings

All guest instances in CloudStack must have an associated compute offering. The import API supports using ‘fixed’ (ie 2 vCPUs with 2GB RAM hardcoded into the offering) and ‘custom’ (user can choose the number of vCPUs and memory) offerings. When a custom offering is chosen, then the CloudStack will automatically set the number vCPUs, CPU speed and amount of RAM, to be the same as the VM before importing it. When using custom offerings, the instance to be imported can remain running. If the compute offering is ‘fixed’ and it matches the vCPU and RAM of the existing instance, the instance can remain running while being imported, otherwise the instance must be stopped first and it will be reconfigured with the new values when it is started.

For maximum compatibility when importing a VM, the Custom Constrained type of compute offerings in CloudStack are the recommended type of offerings. The amount of memory and number of CPUs assigned to the imported VM will automatically be matched to the existing VM, while the CPU speed will have been set to a sensible value by the admin when creating the offering.

Note

To use Custom Unconstrained type of compute offering, CPU speed will need to be passed using details parameter when the CPU reservation is not set for the unmanaged VM in vSphere. CPU speed in the latter case can be passed as, details[0].cpuSpeed=SOME_VALUE.

Disk Offerings

To import a VM which has data disks attached, a map of the disk ID and corresponding disk offering ID must be passed via the datadiskofferinglist parameter.

For example:

datadiskofferinglist[0].disk=<DISK_ID> datadiskofferinglist[0].diskOffering=<DISK_OFFERING_ID>

Note

If the selected disk offering is greater in size than the actual disk size, CloudStack will not perform resize of the disk when importing. The disk will remain with its original size, but CloudStack will have a record as per the offering.

Host and Storage Tags

When the migrateallowed parameter is set to true, if the host or storage tags in the compute/disk offerings are incompatible with the current host and/or storage pool(s), CloudStack will migrate the VM and its volumes to a suitable host and storage pool.

When migrateallowed is false and there is a conflict, an appropriate error will be returned.

Migration is supported for both running and stopped VMs. Live-migration is supported for running imported VM. When a stopped VM is imported, CloudStack will migrate VM to a suitable host when it is restarted.

For volumes, live-migration will be carried out for the volumes of a running VM. As per existing CloudStack behaviour, a stopped imported VM may not appear in vCenter when its root volume is migrated until the VM is restarted.

Registered Operating System

Import API will try to recognize and map the operating system type for the unmanaged VM to the one from the list of the guest operating systems available in CloudStack. If the operating system type can not be mapped, the API will return an error, and the templateid parameter (value = ID of a template with the appropriate operating system) will be needed for a successful import. When templateid is defined in the import API call, the guest operating system details of the imported VM will be set to the operating system details of the specified template after VM restart.

Other notes for the importUnmanagedInstance API

• The API will use name for the hostname of the VM when hostname parameter is not explicitly passed. The hostname cannot be longer than 63 characters. Only ASCII letters a-z, A-Z, digits 0-9, hyphen are allowed. Must start with a letter and end with a letter or a digit.
• NIC adapters and disk controllers of the VM will remain same as they were before the import, irrespective of the template configurations.
• When the VM operating system is automatically recognized during the import (i.e. templateid parameter is not specified), and the operating system of the VM (as reported by the hypervisor) can be matched to multiple operating systems in the CloudStack, the first match will be used as the operating system for the imported VM in CloudStack. An example of this is i.e. “CentOS 7 (64-bit)” operating system type, as visible in vSphere, since this one can be matched against “CentOS 7” or “CentOS 7.1” or “CentOS 7.2” in CloudStack (based on the existing guest OS mappings), and here the first one (“CentOS 7”) will be used as the operating system for the imported VM.
• Importing VMs with different types of disk controllers for data disks and multiple NICs of different types is not supported and will result in an error response. Root disk and other (data disks) disks can have different type of controller.
• After import, once the VM is started from CloudStack its CPU and RAM configuration, including CPU limits, CPU reservations, memory reservation, etc. may change from the original configuration, since all those properties are now controlled by CloudStack (i.e. by cluster-level settings and Compute Offering settings).
• After importing a running VM, the VM will need to be stopped and started (not restarted) via CloudStack to be able to access the console of a VM.

Discovery of Existing Networks (for vSphere)

To import existing VMs, the networks that they are attached to need to already exist as CloudStack networks. As an existing environment can have a great many networks which need creating, A Python 3 script has been created to enumerate the existing networks.

The script (discover_networks.py) can be found in the vm/hypervisor/vmware directory in the CloudStack scripts install location. For most operating systems, CloudStack installs scripts in /usr/share/cloudstack-common/. The script leverages VMware’s pyvmomi library (https://github.com/vmware/pyvmomi). The script lists all networks for a vCenter host or cluster which have at least one virtual machine attached to them. The script will iterate through these networks and will report the following parameters for them:

• cluster (vCenter Cluster belongs to)
• host (vCenter Host belongs to)
• portgroup (Portgroup of the network)
• switch (Switch to which network is connected)
• virtualmachines (Virtual machines that are currently connected to the network along with their NIC device details)
• vlanid (VLAN ID of the network)

The script can take the following arguments:

-h, --help show this help message and exit
-s HOST, --host HOST vSphere service to connect to
-o PORT, --port PORT Port to connect on
-u USER, --user USER User name to use
-p PASSWORD, --password PASSWORD Password to use
-c CLUSTER, --cluster CLUSTER Cluster for listing network
-S, --disable_ssl_verification Disable ssl host certificate verification
-d, --debug Debug log messages

Note

To run this script host machine should have Python 3 and module pyvmomi installed.

Python binaries can be found here: https://www.python.org/downloads/

Install instructions for pyvmomi are here: https://github.com/vmware/pyvmomi#installing

The output of this script can then be used in conjunction with the ‘createNetwork’ API to add all of the networks to CloudStack that will be required for a successful import.

Virtual Machine Backups (Backup and Recovery Feature)

About Backup And Recovery

CloudStack version 4.14 introduces a new Backup and Recovery (B&R) framework that provides CloudStack with users the ability to back up their guest VMs for recovery purposes via 3rd party backup solutions. The framework abstracts the API commands required for common backup and recovery operations, from the vendor specific commmands needed to perform those actions and provides a plugin model to enable any solution which provides backup and recovery ‘like’ features to be integrated.

The following providers are currently supported:

• VMware with Veeam Backup and Recovery

See the Veeam Backup and Recovery plugin documentation for plugin specific information. Veeam Backup and Recovery Plugin

Backup and Recovery Concepts

Backup and recovery has been designed to support two modes:

• ‘SLA’ based backups
• Adhoc and user scheduled backups

‘SLA’ based backups are ones where the Cloud provider (ie the root admin) controls the time, and frequency of a backup scheme. A user signs up for a ‘Gold’ offering, which might give them a RPO of 12 hours and the last 14 backups kept; however the user would not be allowed to perform additional backups nor set the exact time that these backups took place. The user might be charged a fix rate for these backups regardless of the size of the backups.

To use an SLA based backup policy the user adds their VMs to the offering/policy. The job then runs at its predetermined times and ‘includes’ the VM when it runs. A user can remove the VM from the offering/policy and it will no longer be included in the job when it runs.

Adhoc and user scheduled backups follow the same idea as volume snapshots, however they leverage the backup solution rather than secondary storage. These could likely be billed on backup storage consumed or protected capacity (the full virtual size of the VM(s) being backed up.

Adhoc and user scheduled backups are created and managed in the same fashion as volume snapshots are.

Configuring Backup and Recovery

The cloud administrator can use global configuration variables to control the behavior of B&R feature. To set these variables, go through the Global Settings area of the CloudStack UI.

Plugin specific settings

Each backup and recovery plugin is likely to have settings specific to that plugin. Refer to the CloudStack documentation for your plugin for details on how to configure those settings.

Backup Offerings

Admins can import an external provider’s backup offerings using UI or API for a particular zone, as well as manage a backup offering’s lifecyle. Admins can also specify if a backup offering allows user-defined backup schedules and ad-hoc backups. Users can list and consume the imported backup offerings, only root admins can import or delete offerings.

Supported APIs:

• listBackupProviders: lists available backup provider plugins
• listBackupProviderOfferings: lists external backup policy/offering from a provider
• importBackupProviderOfferings: allows importing of an external backup policy/offering to CloudStack as a backup offering
• listBackupOfferings: lists CloudStack’s backup offerings (searching via keyword, and pagination supported)
• deleteBackupOffering: deletes a backup offering by its ID

Importing Backup Offerings

See plugin specific documentation to create ‘Backup provider offerings’

To import a backup provider offering;

1. (As root) navigate to Service Offerings, click on the ‘select offering’ dropdown box and select ‘Backup Offerings’

2. Click on Import Backup Offering

3. Enter your user-friendly name and description and select the applicable zone. The External ID will then be populated with the template jobs which CloudStack retrieves from the connected provider.

   

Creating VM Backups

SLA/Policy Based backups

With the backup and recovery feature enabled for a zone, users simply add and remove a VM from a backup offering.

Adhoc and Scheduled Backups

For backup offerings that allow ad-hoc user backups and user-defined backup schedules, user will be allowed to define a backup schedule for a VM that is assigned to a backup offering using UI and API. A VM with backup will not be allowed to add/remove volumes similar to VM snapshots.

To trigger an adhoc backup of a VM, navigate to the instance and click on the ‘Create Backup’ icon.

To setup a recurring backup schedule, navigate to the instance and click on the ‘Backup Schedule’ icon.

Then set the time and frequency of the backups, click ‘Configure’ and then ‘Close’

Restoring VM Backups

Users will need to stop a VM to restore to any existing VM backup, restoration of an expunged VM will not restore nics and recovery any network which may/may not exist. User may however restore a specific volume from a VM backup and attach that volume to a specified VM.

Supported APIs:

• assignVirtualMachineToBackupOffering: adds a VM to a backup offering.
• removeVirtualMachineFromBackupOffering: removes a VM from a backup offering, if forced true parameter is passed this may also remove any and all the backups of a VM associated with a backup offering.
• createBackupSchedule: creates a backup schedule for a VM.
• updateBackupSchedule: updates backup schedule.
• listBackupSchedule: returns backup schedule of a VM if defined.
• deleteBackupSchedule: deletes backup schedule of a VM.
• createBackup: creates an adhoc backup for a VM.
• deleteVMBackup: deletes a VM backup (not support for per restore point for Veeam).
• listBackups: lists backups.
• restoreBackup: restore a previous VM backup in-place of a stopped or destroyed VM.
• restoreVolumeFromBackup: restore and attach a backed-up volume (of a VM backup) to a specified VM.

Using SSH Keys for Authentication

In addition to the username and password authentication, CloudStack supports using SSH keys to log in to the cloud infrastructure for additional security. You can use the createSSHKeyPair API to generate the SSH keys.

Because each cloud user has their own SSH key, one cloud user cannot log in to another cloud user’s instances unless they share their SSH key files. Using a single SSH key pair, you can manage multiple instances.

Creating an Instance Template that Supports SSH Keys

Create an instance template that supports SSH Keys.

1. Create a new instance by using the template provided by cloudstack.

   For more information on creating a new instance, see

2. Download the cloudstack script from The SSH Key Gen Script to the instance you have created.

   wget http://downloads.sourceforge.net/project/cloudstack/SSH%20Key%20Gen%20Script/cloud-set-guest-sshkey.in?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcloudstack%2Ffiles%2FSSH%2520Key%2520Gen%2520Script%2F&ts=1331225219&use_mirror=iweb

3. Copy the file to /etc/init.d.

   cp cloud-set-guest-sshkey.in /etc/init.d/

4. Give the necessary permissions on the script:

   chmod +x /etc/init.d/cloud-set-guest-sshkey.in

5. Run the script while starting up the operating system:

   chkconfig --add cloud-set-guest-sshkey.in

6. Stop the instance.

Creating the SSH Keypair

You must make a call to the createSSHKeyPair api method. You can either use the CloudStack Python API library or the curl commands to make the call to the cloudstack api.

For example, make a call from the cloudstack server to create a SSH keypair called “keypair-doc” for the admin account in the root domain:

Note

Ensure that you adjust these values to meet your needs. If you are making the API call from a different server, your URL/PORT will be different, and you will need to use the API keys.

1. Run the following curl command:

   curl --globoff "http://localhost:8096/?command=createSSHKeyPair&name=keypair-doc&account=admin&domainid=5163440e-c44b-42b5-9109-ad75cae8e8a2"

   The output is something similar to what is given below:

   <?xml version="1.0" encoding="ISO-8859-1"?><createsshkeypairresponse cloud-stack-version="3.0.0.20120228045507"><keypair><name>keypair-doc</name><fingerprint>f6:77:39:d5:5e:77:02:22:6a:d8:7f:ce:ab:cd:b3:56</fingerprint><privatekey>-----BEGIN RSA PRIVATE KEY-----
   MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci
   dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB
   AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu
   mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy
   QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7
   VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK
   lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm
   nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14
   4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd
   KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3
   -----END RSA PRIVATE KEY-----
   </privatekey></keypair></createsshkeypairresponse>

2. Copy the key data into a file. The file looks like this:

   -----BEGIN RSA PRIVATE KEY-----
   MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci
   dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB
   AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu
   mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy
   QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7
   VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK
   lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm
   nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14
   4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd
   KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3
   -----END RSA PRIVATE KEY-----

3. Save the file.

Creating an Instance

After you save the SSH keypair file, you must create an instance by using the template that you created at Section 5.2.1, “ Creating an Instance Template that Supports SSH Keys”. Ensure that you use the same SSH key name that you created at Section 5.2.2, “Creating the SSH Keypair”.

Note

You cannot create the instance by using the GUI at this time and associate the instance with the newly created SSH keypair.

A sample curl command to create a new instance is:

curl --globoff http://localhost:<port number>/?command=deployVirtualMachine&zoneId=1&serviceOfferingId=18727021-7556-4110-9322-d625b52e0813&templateId=e899c18a-ce13-4bbf-98a9-625c5026e0b5&securitygroupids=ff03f02f-9e3b-48f8-834d-91b822da40c5&account=admin&domainid=1&keypair=keypair-doc

Substitute the template, service offering and security group IDs (if you are using the security group feature) that are in your cloud environment.

Logging In Using the SSH Keypair

To test your SSH key generation is successful, check whether you can log in to the cloud setup.

For example, from a Linux OS, run:

ssh -i ~/.ssh/keypair-doc <ip address>

The -i parameter tells the ssh client to use a ssh key found at ~/.ssh/keypair-doc.

Resetting SSH Keys

With the API command resetSSHKeyForVirtualMachine, a user can set or reset the SSH keypair assigned to a virtual machine. A lost or compromised SSH keypair can be changed, and the user can access the VM by using the new keypair. Just create or register a new keypair, then call resetSSHKeyForVirtualMachine.

User-Data and Meta-Data

CloudStack provides APIs to attach up to 32KB of user-data to a deployed VM.

There are two CloudStack APIs that can be used to store user-data: deployVirtualMachine and updateVirtualMachine They both support the parameter userdata=. The value for this parameter must be a base64-encoded multi-part MIME message. See further below for an example of what this should look like.

HTTP GET parameters are limited to a length of 2048 bytes, but it is possible to store larger user-data blobs by sending them in the body via HTTP POST instead of GET.

From inside the VM, the user-data is accessible via the virtual router, if the UserData service is enabled on the network offering.

If you are using the DNS service of the virtual router, a special hostname called data-server. is provided, that will point to a valid user-data server.

Otherwise you have to determine the virtual router address via other means, such as DHCP leases. Be careful to scan all routers if you have multiple networks attached to a VM, in case not all of them have the UserData service enabled.

User-data is available from the URL http://data-server./latest/user-data and can be fetched via curl or other HTTP client.

It is also possible to fetch VM metadata from the same service, via the URL http://data-server./latest/{metadata type}. For backwards compatibility, the previous URL http://data-server./latest/{metadata type} is also supported.

For metadata type, use one of the following:

• service-offering. A description of the VMs service offering
• availability-zone. The Zone name
• local-ipv4. The guest IP of the VM
• local-hostname. The hostname of the VM
• public-ipv4. The first public IP for the router.
• public-hostname. This is the same as public-ipv4
• instance-id. The instance name of the VM

Determining the virtual router address without DNS

If can’t or don’t want to use the virtual router’s DNS service, it’s also possible to determine the user-data server from a DHCP lease.

1. Run the following command to find the virtual router.

   # cat /var/lib/dhcp/dhclient.eth0.leases | grep dhcp-server-identifier | tail -1

2. Access the data-server via its IP

   # curl http://10.1.1.1/latest/user-data

Fetching user-data via the API

User-data is not included with the normal VM state for historic reasons. To read out the base64-encoded user-data via the API, use the getVirtualMachineUserData API call:

cmk get virtualmachineuserdata virtualmachineid=8fd996b6-a102-11ea-ba47-23394b299ae9

Using cloud-init

cloud-init can be used to access and interpret user-data inside virtual machines. If you install cloud-init into your VM templates, it will allow you to store SSH keys and user passwords on each new VM deployment automatically (Adding Password Management to Your Templates and using ssh keys).

1. Install cloud-init package into a VM template:

   # yum install cloud-init
     or
   $ sudo apt-get install cloud-init

2. Create a datasource configuration file in the VM template: /etc/cloud/cloud.cfg.d/99_cloudstack.cfg

   datasource:
     CloudStack: {}
     None: {}
   datasource_list:
     - CloudStack

Custom user-data example

This example uses cloud-init to automatically update all OS packages on the first launch.

1. Create user-data, wrapped into a multi-part MIME message and encoded in base64:

base64 <<EOF
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
# Upgrade the instance on first boot
# (ie run apt-get upgrade)
#
# Default: false
# Aliases: apt_upgrade
package_upgrade: true
EOF

1. Deploy a VM with this user-data:

cmk deploy virtualmachine name=..... userdata=Q29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSIvLyIKTUlNRS1WZXJzaW9uOiAxLjAKCi0tLy8KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCk1JTUUtVmVyc2lvbjogMS4wCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKQ29udGVudC1EaXNwb3NpdGlvbjogYXR0YWNobWVudDsgZmlsZW5hbWU9ImNsb3VkLWNvbmZpZy50eHQiCgojY2xvdWQtY29uZmlnCgojIFVwZ3JhZGUgdGhlIGluc3RhbmNlIG9uIGZpcnN0IGJvb3QKIyAoaWUgcnVuIGFwdC1nZXQgdXBncmFkZSkKIwojIERlZmF1bHQ6IGZhbHNlCiMgQWxpYXNlczogYXB0X3VwZ3JhZGUKcGFja2FnZV91cGdyYWRlOiB0cnVlCg==

Disclaimer

Refer to the cloud-init CloudStack datasource documentation for latest capabilities. cloud-init and the cloud-init CloudStack datasource are not supported by Apache CloudStack community.

Assigning GPU/vGPU to Guest VMs

CloudStack can deploy guest VMs with Graphics Processing Unit (GPU) or Virtual Graphics Processing Unit (vGPU) capabilities on XenServer hosts. At the time of VM deployment or at a later stage, you can assign a physical GPU ( known as GPU-passthrough) or a portion of a physical GPU card (vGPU) to a guest VM by changing the Service Offering. With this capability, the VMs running on CloudStack meet the intensive graphical processing requirement by means of the high computation power of GPU/vGPU, and CloudStack users can run multimedia rich applications, such as Auto-CAD, that they otherwise enjoy at their desk on a virtualized environment. CloudStack leverages the XenServer support for NVIDIA GRID Kepler 1 and 2 series to run GPU/vGPU enabled VMs. NVIDIA GRID cards allows sharing a single GPU cards among multiple VMs by creating vGPUs for each VM. With vGPU technology, the graphics commands from each VM are passed directly to the underlying dedicated GPU, without the intervention of the hypervisor. This allows the GPU hardware to be time-sliced and shared across multiple VMs. XenServer hosts use the GPU cards in following ways:

GPU passthrough: GPU passthrough represents a physical GPU which can be directly assigned to a VM. GPU passthrough can be used on a hypervisor alongside GRID vGPU, with some restrictions: A GRID physical GPU can either host GRID vGPUs or be used as passthrough, but not both at the same time.

GRID vGPU: GRID vGPU enables multiple VMs to share a single physical GPU. The VMs run an NVIDIA driver stack and get direct access to the GPU. GRID physical GPUs are capable of supporting multiple virtual GPU devices (vGPUs) that can be assigned directly to guest VMs. Guest VMs use GRID virtual GPUs in the same manner as a physical GPU that has been passed through by the hypervisor: an NVIDIA driver loaded in the guest VM provides direct access to the GPU for performance-critical fast paths, and a paravirtualized interface to the GRID Virtual GPU Manager, which is used for nonperformant management operations. NVIDIA GRID Virtual GPU Manager for XenServer runs in dom0. CloudStack provides you with the following capabilities:

• Adding XenServer hosts with GPU/vGPU capability provisioned by the administrator.
• Creating a Compute Offering with GPU/vGPU capability.
• Deploying a VM with GPU/vGPU capability.
• Destroying a VM with GPU/vGPU capability.
• Allowing an user to add GPU/vGPU support to a VM without GPU/vGPU support by changing the Service Offering and vice-versa.
• Migrating VMs (cold migration) with GPU/vGPU capability.
• Managing GPU cards capacity.
• Querying hosts to obtain information about the GPU cards, supported vGPU types in case of GRID cards, and capacity of the cards.

Prerequisites and System Requirements

Before proceeding, ensure that you have these prerequisites:

• The vGPU-enabled XenServer 6.2 and later versions. For more information, see Citrix 3D Graphics Pack.
• GPU/vPGU functionality is supported for following HVM guest operating systems: For more information, see Citrix 3D Graphics Pack.
• Windows 7 (x86 and x64)
• Windows Server 2008 R2
• Windows Server 2012
• Windows 8 (x86 and x64)
• Windows 8.1 (“Blue”) (x86 and x64)
• Windows Server 2012 R2 (server equivalent of “Blue”)
• CloudStack does not restrict the deployment of GPU-enabled VMs with guest OS types that are not supported by XenServer for GPU/vGPU functionality. The deployment would be successful and a GPU/vGPU will also get allocated for VMs; however, due to missing guest OS drivers, VM would not be able to leverage GPU resources. Therefore, it is recommended to use GPU-enabled service offering only with supported guest OS.
• NVIDIA GRID K1 (16 GiB video RAM) AND K2 (8 GiB of video RAM) cards supports homogeneous virtual GPUs, implies that at any given time, the vGPUs resident on a single physical GPU must be all of the same type. However, this restriction doesn’t extend across physical GPUs on the same card. Each physical GPU on a K1 or K2 may host different types of virtual GPU at the same time. For example, a GRID K2 card has two physical GPUs, and supports four types of virtual GPU; GRID K200, GRID K220Q, GRID K240Q, AND GRID K260Q.
• NVIDIA driver must be installed to enable vGPU operation as for a physical NVIDIA GPU.
• XenServer tools are installed in the VM to get maximum performance on XenServer, regardless of type of vGPU you are using. Without the optimized networking and storage drivers that the XenServer tools provide, remote graphics applications running on GRID vGPU will not deliver maximum performance.
• To deliver high frames from multiple heads on vGPU, install XenDesktop with HDX 3D Pro remote graphics.

Before continuing with configuration, consider the following:

• Deploying VMs GPU/vGPU capability is not supported if hosts are not available with enough GPU capacity.
• A Service Offering cannot be created with the GPU values that are not supported by CloudStack UI. However, you can make an API call to achieve this.
• Dynamic scaling is not supported. However, you can choose to deploy a VM without GPU support, and at a later point, you can change the system offering to upgrade to the one with vGPU. You can achieve this by offline upgrade: stop the VM, upgrade the Service Offering to the one with vGPU, then start the VM.
• Live migration of GPU/vGPU enabled VM is not supported.
• Limiting GPU resources per Account/Domain is not supported.
• Disabling GPU at Cluster level is not supported.
• Notification thresholds for GPU resource is not supported.

Supported GPU Devices

GPU/vGPU Assignment Workflow

CloudStack follows the below sequence of operations to provide GPU/vGPU support for VMs:

1. Ensure that XenServer host is ready with GPU installed and configured. For more information, see Citrix 3D Graphics Pack.

2. Add the host to CloudStack. CloudStack checks if the host is GPU-enabled or not. CloudStack queries the host and detect if it’s GPU enabled.

3. Create a compute offering with GPU/vGPU support: For more information, see Creating a New Compute Offering..

4. Continue with any of the following operations:

   • Deploy a VM.

     Deploy a VM with GPU/vGPU support by selecting appropriate Service Offering. CloudStack decide which host to choose for VM deployment based on following criteria:

     • Host has GPU cards in it. In case of vGPU, CloudStack checks if cards have the required vGPU type support and enough capacity available. Having no appropriate hosts results in an InsufficientServerCapacity exception.
     • Alternately, you can choose to deploy a VM without GPU support, and at a later point, you can change the system offering. You can achieve this by offline upgrade: stop the VM, upgrade the Service Offering to the one with vGPU, then start the VM. In this case, CloudStack gets a list of hosts which have enough capacity to host the VM. If there is a GPU-enabled host, CloudStack reorders this host list and place the GPU-enabled hosts at the bottom of the list.

   • Migrate a VM.

     CloudStack searches for hosts available for VM migration, which satisfies GPU requirement. If the host is available, stop the VM in the current host and perform the VM migration task. If the VM migration is successful, the remaining GPU capacity is updated for both the hosts accordingly.

   • Destroy a VM.

     GPU resources are released automatically when you stop a VM. Once the destroy VM is successful, CloudStack will make a resource call to the host to get the remaining GPU capacity in the card and update the database accordingly.