About Working with Virtual Machines

CloudStack provides administrators with complete control over the lifecycle of all guest VMs executing in the cloud. CloudStack provides several guest management operations for end users and administrators. VMs may be stopped, started, rebooted, and destroyed.

Guest VMs have a name and group. VM names and groups are opaque to CloudStack and are available for end users to organize their VMs. Each VM can have three names for use in different contexts. Only two of these names can be controlled by the user:

• Instance name – a unique, immutable ID that is generated by CloudStack and can not be modified by the user. This name conforms to the requirements in IETF RFC 1123.
• Display name – the name displayed in the CloudStack web UI. Can be set by the user. Defaults to instance name.
• Name – host name that the DHCP server assigns to the VM. Can be set by the user. Defaults to instance name

Note

You can append the display name of a guest VM to its internal name. For more information, see “Appending a Display Name to the Guest VM’s Internal Name”.

Guest VMs can be configured to be Highly Available (HA). An HA-enabled VM is monitored by the system. If the system detects that the VM is down, it will attempt to restart the VM, possibly on a different host. For more information, see HA-Enabled Virtual Machines on

Each new VM is allocated one public IP address. When the VM is started, CloudStack automatically creates a static NAT between this public IP address and the private IP address of the VM.

If elastic IP is in use (with the NetScaler load balancer), the IP address initially allocated to the new VM is not marked as elastic. The user must replace the automatically configured IP with a specifically acquired elastic IP, and set up the static NAT mapping between this new IP and the guest VM’s private IP. The VM’s original IP address is then released and returned to the pool of available public IPs. Optionally, you can also decide not to allocate a public IP to a VM in an EIP-enabled Basic zone. For more information on Elastic IP, see “About Elastic IP”.

CloudStack cannot distinguish a guest VM that was shut down by the user (such as with the “shutdown” command in Linux) from a VM that shut down unexpectedly. If an HA-enabled VM is shut down from inside the VM, CloudStack will restart it. To shut down an HA-enabled VM, you must go through the CloudStack UI or API.

Best Practices for Virtual Machines

For VMs to work as expected and provide excellent service, follow these guidelines.

Monitor VMs for Max Capacity

The CloudStack administrator should monitor the total number of VM instances in each cluster, and disable allocation to the cluster if the total is approaching the maximum that the hypervisor can handle. Be sure to leave a safety margin to allow for the possibility of one or more hosts failing, which would increase the VM load on the other hosts as the VMs are automatically redeployed. Consult the documentation for your chosen hypervisor to find the maximum permitted number of VMs per host, then use CloudStack global configuration settings to set this as the default limit. Monitor the VM activity in each cluster at all times. Keep the total number of VMs below a safe level that allows for the occasional host failure. For example, if there are N hosts in the cluster, and you want to allow for one host in the cluster to be down at any given time, the total number of VM instances you can permit in the cluster is at most (N-1) * (per-host-limit). Once a cluster reaches this number of VMs, use the CloudStack UI to disable allocation of more VMs to the cluster.

Install Required Tools and Drivers

Be sure the following are installed on each VM:

• For XenServer, install PV drivers and Xen tools on each VM. This will enable live migration and clean guest shutdown. Xen tools are required in order for dynamic CPU and RAM scaling to work.
• For vSphere, install VMware Tools on each VM. This will enable console view to work properly. VMware Tools are required in order for dynamic CPU and RAM scaling to work.

To be sure that Xen tools or VMware Tools is installed, use one of the following techniques:

• Create each VM from a template that already has the tools installed; or,
• When registering a new template, the administrator or user can indicate whether tools are installed on the template. This can be done through the UI or using the updateTemplate API; or,
• If a user deploys a virtual machine with a template that does not have Xen tools or VMware Tools, and later installs the tools on the VM, then the user can inform CloudStack using the updateVirtualMachine API. After installing the tools and updating the virtual machine, stop and start the VM.

VM Lifecycle

Virtual machines can be in the following states:

Once a virtual machine is destroyed, it cannot be recovered. All the resources used by the virtual machine will be reclaimed by the system. This includes the virtual machine’s IP address.

A stop will attempt to gracefully shut down the operating system, which typically involves terminating all the running applications. If the operation system cannot be stopped, it will be forcefully terminated. This has the same effect as pulling the power cord to a physical machine.

A reboot is a stop followed by a start.

CloudStack preserves the state of the virtual machine hard disk until the machine is destroyed.

A running virtual machine may fail because of hardware or network issues. A failed virtual machine is in the down state.

The system places the virtual machine into the down state if it does not receive the heartbeat from the hypervisor for three minutes.

The user can manually restart the virtual machine from the down state.

The system will start the virtual machine from the down state automatically if the virtual machine is marked as HA-enabled.

Creating VMs

Virtual machines are usually created from a template. Users can also create blank virtual machines. A blank virtual machine is a virtual machine without an OS template. Users can attach an ISO file and install the OS from the CD/DVD-ROM.

Note

You can create a VM without starting it. You can determine whether the VM needs to be started as part of the VM deployment. A request parameter, startVM, in the deployVm API provides this feature. For more information, see the Developer’s Guide.

To create a VM from a template:

1. Log in to the CloudStack UI as an administrator or user.

2. In the left navigation bar, click Instances.

3. Click Add Instance.

4. Select a zone. Admin users will have the option to select a pod, cluster or host.

5. Select a template, then follow the steps in the wizard. For more information about how the templates came to be in this list, see *Working with Templates*.

6. Be sure that the hardware you have allows starting the selected service offering.

   Note

   VMware only: If the selected template OVF contains static properties, the wizard will display these properties. Static properties do not have the concept of required and optional fields . Therefore CloudStack is not able to enforce the population of ‘important fields’.

7. Click Submit and your VM will be created and started.

   Note

   For security reason, the internal name of the VM is visible only to the root admin.

To create a VM from an ISO:

Note

(XenServer) Windows VMs running on XenServer require PV drivers, which may be provided in the template or added after the VM is created. The PV drivers are necessary for essential management functions such as mounting additional volumes and ISO images, live migration, and graceful shutdown.

1. Log in to the CloudStack UI as an administrator or user.
2. In the left navigation bar, click Instances.
3. Click Add Instance.
4. Select a zone. Admin users will have the option to select a pod, cluster or host.
5. Select ISO Boot, and follow the steps in the wizard.
6. Click Submit and your VM will be created and started.

Accessing VMs

Any user can access their own virtual machines. The administrator can access all VMs running in the cloud.

To access a VM through the CloudStack UI:

1. Log in to the CloudStack UI as a user or admin.
2. Click Instances, then click the name of a running VM.
3. Click the View Console button .

To access a VM directly over the network:

1. The VM must have some port open to incoming traffic. For example, in a basic zone, a new VM might be assigned to a security group which allows incoming traffic. This depends on what security group you picked when creating the VM. In other cases, you can open a port by setting up a port forwarding policy. See “IP Forwarding and Firewalling”.
2. If a port is open but you can not access the VM using ssh, it’s possible that ssh is not already enabled on the VM. This will depend on whether ssh is enabled in the template you picked when creating the VM. Access the VM through the CloudStack UI and enable ssh on the machine using the commands for the VM’s operating system.
3. If the network has an external firewall device, you will need to create a firewall rule to allow access. See “IP Forwarding and Firewalling”.

Stopping and Starting VMs

Once a VM instance is created, you can stop, restart, or delete it as needed. In the CloudStack UI, click Instances, select the VM, and use the Stop, Start, Reboot, and Destroy buttons.

When starting a VM, admin users have the option to specify a pod, cluster, or host.

Assigning VMs to Hosts

At any point in time, each virtual machine instance is running on a single host. How does CloudStack determine which host to place a VM on? There are several ways:

• Automatic default host allocation. CloudStack can automatically pick the most appropriate host to run each virtual machine.
• Instance type preferences. CloudStack administrators can specify that certain hosts should have a preference for particular types of guest instances. For example, an administrator could state that a host should have a preference to run Windows guests. The default host allocator will attempt to place guests of that OS type on such hosts first. If no such host is available, the allocator will place the instance wherever there is sufficient physical capacity.
• Vertical and horizontal allocation. Vertical allocation consumes all the resources of a given host before allocating any guests on a second host. This reduces power consumption in the cloud. Horizontal allocation places a guest on each host in a round-robin fashion. This may yield better performance to the guests in some cases.
• Admin users preferences. Administrators have the option to specify a pod, cluster, or host to run the VM in. CloudStack will then select a host within the given infrastructure.
• End user preferences. Users can not control exactly which host will run a given VM instance, but they can specify a zone for the VM. CloudStack is then restricted to allocating the VM only to one of the hosts in that zone.
• Host tags. The administrator can assign tags to hosts. These tags can be used to specify which host a VM should use. The CloudStack administrator decides whether to define host tags, then create a service offering using those tags and offer it to the user.
• Affinity groups. By defining affinity groups and assigning VMs to them, the user or administrator can influence (but not dictate) which VMs should run on separate hosts. This feature is to let users specify that certain VMs won’t be on the same host.
• CloudStack also provides a pluggable interface for adding new allocators. These custom allocators can provide any policy the administrator desires.

Affinity Groups

By defining affinity groups and assigning VMs to them, the user or administrator can influence (but not dictate) which VMs should run on separate hosts. This feature is to let users specify that VMs with the same “host anti-affinity” type won’t be on the same host. This serves to increase fault tolerance. If a host fails, another VM offering the same service (for example, hosting the user’s website) is still up and running on another host.

The scope of an affinity group is per user account.

Creating a New Affinity Group

To add an affinity group:

1. Log in to the CloudStack UI as an administrator or user.
2. In the left navigation bar, click Affinity Groups.
3. Click Add affinity group. In the dialog box, fill in the following fields:
   • Name. Give the group a name.
   • Description. Any desired text to tell more about the purpose of the group.
   • Type. The only supported type shipped with CloudStack is Host Anti-Affinity. This indicates that the VMs in this group should avoid being placed on the same host with each other. If you see other types in this list, it means that your installation of CloudStack has been extended with customized affinity group plugins.

Assign a New VM to an Affinity Group

To assign a new VM to an affinity group:

• Create the VM as usual, as described in “Creating VMs”. In the Add Instance wizard, there is a new Affinity tab where you can select the affinity group.

Change Affinity Group for an Existing VM

To assign an existing VM to an affinity group:

1. Log in to the CloudStack UI as an administrator or user.
2. In the left navigation bar, click Instances.
3. Click the name of the VM you want to work with.
4. Stop the VM by clicking the Stop button.
5. Click the Change Affinity button.

View Members of an Affinity Group

To see which VMs are currently assigned to a particular affinity group:

1. In the left navigation bar, click Affinity Groups.

2. Click the name of the group you are interested in.

3. Click View Instances. The members of the group are listed.

   From here, you can click the name of any VM in the list to access all its details and controls.

Delete an Affinity Group

To delete an affinity group:

1. In the left navigation bar, click Affinity Groups.

2. Click the name of the group you are interested in.

3. Click Delete.

   Any VM that is a member of the affinity group will be disassociated from the group. The former group members will continue to run normally on the current hosts, but if the VM is restarted, it will no longer follow the host allocation rules from its former affinity group.

Virtual Machine Snapshots

(Supported on VMware, XenServer and KVM (NFS only))

In addition to the existing CloudStack ability to snapshot individual VM volumes, you can take a VM snapshot to preserve all the VM’s data volumes as well as (optionally) its CPU/memory state. This is useful for quick restore of a VM. For example, you can snapshot a VM, then make changes such as software upgrades. If anything goes wrong, simply restore the VM to its previous state using the previously saved VM snapshot.

The snapshot is created using the hypervisor’s native snapshot facility. The VM snapshot includes not only the data volumes, but optionally also whether the VM is running or turned off (CPU state) and the memory contents. The snapshot is stored in CloudStack’s primary storage.

VM snapshots can have a parent/child relationship. Each successive snapshot of the same VM is the child of the snapshot that came before it. Each time you take an additional snapshot of the same VM, it saves only the differences between the current state of the VM and the state stored in the most recent previous snapshot. The previous snapshot becomes a parent, and the new snapshot is its child. It is possible to create a long chain of these parent/child snapshots, which amount to a “redo” record leading from the current state of the VM back to the original.

After VM snapshots are created, they can be tagged with a key/value pair, like many other resources in CloudStack.

KVM supports VM snapshots when using NFS shared storage. If raw block storage is used (i.e. Ceph), then VM snapshots are not possible, since there is no possibility to write RAM memory content anywhere.

If you need more information about VM snapshots on VMware, check out the VMware documentation and the VMware Knowledge Base, especially Understanding virtual machine snapshots.

Limitations on VM Snapshots

• If a VM has some stored snapshots, you can’t attach new volume to the VM or delete any existing volumes. If you change the volumes on the VM, it would become impossible to restore the VM snapshot which was created with the previous volume structure. If you want to attach a volume to such a VM, first delete its snapshots.
• VM snapshots which include both data volumes and memory can’t be kept if you change the VM’s service offering. Any existing VM snapshots of this type will be discarded.
• You can’t make a VM snapshot at the same time as you are taking a volume snapshot.
• You should use only CloudStack to create VM snapshots on hosts managed by CloudStack. Any snapshots that you make directly on the hypervisor will not be tracked in CloudStack.

Configuring VM Snapshots

The cloud administrator can use global configuration variables to control the behavior of VM snapshots. To set these variables, go through the Global Settings area of the CloudStack UI.

Using VM Snapshots

To create a VM snapshot using the CloudStack UI:

1. Log in to the CloudStack UI as a user or administrator.

2. Click Instances.

3. Click the name of the VM you want to snapshot.

4. Click the Take VM Snapshot button.

   Note

   If a snapshot is already in progress, then clicking this button will have no effect.

5. Provide a name and description. These will be displayed in the VM Snapshots list.

6. (For running VMs only) If you want to include the VM’s memory in the snapshot, click the Memory checkbox. This saves the CPU and memory state of the virtual machine. If you don’t check this box, then only the current state of the VM disk is saved. Checking this box makes the snapshot take longer.

7. Quiesce VM: check this box if you want to quiesce the file system on the VM before taking the snapshot. Not supported on XenServer when used with CloudStack-provided primary storage.

   When this option is used with CloudStack-provided primary storage, the quiesce operation is performed by the underlying hypervisor (VMware is supported). When used with another primary storage vendor’s plugin, the quiesce operation is provided according to the vendor’s implementation.

8. Click OK.

To delete a snapshot or restore a VM to the state saved in a particular snapshot:

1. Navigate to the VM as described in the earlier steps.

2. Click View VM Snapshots.

3. In the list of snapshots, click the name of the snapshot you want to work with.

4. Depending on what you want to do:

   To delete the snapshot, click the Delete button.

   To revert to the snapshot, click the Revert button.

Note

VM snapshots are deleted automatically when a VM is destroyed. You don’t have to manually delete the snapshots in this case.

Changing the VM Name, OS, or Group

After a VM is created, you can modify the display name, operating system, and the group it belongs to.

To access a VM through the CloudStack UI:

1. Log in to the CloudStack UI as a user or admin.
2. In the left navigation, click Instances.
3. Select the VM that you want to modify.
4. Click the Stop button to stop the VM.
5. Click Edit.
6. Make the desired changes to the following:
7. Display name: Enter a new display name if you want to change the name of the VM.
8. OS Type: Select the desired operating system.
9. Group: Enter the group name for the VM.
10. Click Apply.

Appending a Display Name to the Guest VM’s Internal Name

Every guest VM has an internal name. The host uses the internal name to identify the guest VMs. CloudStack gives you an option to provide a guest VM with a display name. You can set this display name as the internal name so that the vCenter can use it to identify the guest VM. A new global parameter, vm.instancename.flag, has now been added to achieve this functionality.

The default format of the internal name is i-<user_id>-<vm_id>-<instance.name>, where instance.name is a global parameter. However, If vm.instancename.flag is set to true, and if a display name is provided during the creation of a guest VM, the display name is appended to the internal name of the guest VM on the host. This makes the internal name format as i-<user_id>-<vm_id>-<displayName>. The default value of vm.instancename.flag is set to false. This feature is intended to make the correlation between instance names and internal names easier in large data center deployments.

The following table explains how a VM name is displayed in different scenarios.

Changing the Service Offering for a VM

To upgrade or downgrade the level of compute resources available to a virtual machine, you can change the VM’s compute offering.

1. Log in to the CloudStack UI as a user or admin.

2. In the left navigation, click Instances.

3. Choose the VM that you want to work with.

4. (Skip this step if you have enabled dynamic VM scaling; see CPU and Memory Scaling for Running VMs.)

   Click the Stop button to stop the VM.

5. Click the Change Service button.

   The Change service dialog box is displayed.

6. Select the offering you want to apply to the selected VM.

7. Click OK.

CPU and Memory Scaling for Running VMs

(Supported on VMware and XenServer)

It is not always possible to accurately predict the CPU and RAM requirements when you first deploy a VM. You might need to increase these resources at any time during the life of a VM. You can dynamically modify CPU and RAM levels to scale up these resources for a running VM without incurring any downtime.

Dynamic CPU and RAM scaling can be used in the following cases:

• User VMs on hosts running VMware and XenServer.
• System VMs on VMware.
• VMware Tools or XenServer Tools must be installed on the virtual machine.
• The new requested CPU and RAM values must be within the constraints allowed by the hypervisor and the VM operating system.
• New VMs that are created after the installation of CloudStack 4.2 can use the dynamic scaling feature. If you are upgrading from a previous version of CloudStack, your existing VMs created with previous versions will not have the dynamic scaling capability unless you update them using the following procedure.

Updating Existing VMs

If you are upgrading from a previous version of CloudStack, and you want your existing VMs created with previous versions to have the dynamic scaling capability, update the VMs using the following steps:

1. Make sure the zone-level setting enable.dynamic.scale.vm is set to true. In the left navigation bar of the CloudStack UI, click Infrastructure, then click Zones, click the zone you want, and click the Settings tab.
2. Install Xen tools (for XenServer hosts) or VMware Tools (for VMware hosts) on each VM if they are not already installed.
3. Stop the VM.
4. Click the Edit button.
5. Click the Dynamically Scalable checkbox.
6. Click Apply.
7. Restart the VM.

Configuring Dynamic CPU and RAM Scaling

To configure this feature, use the following new global configuration variables:

• enable.dynamic.scale.vm: Set to True to enable the feature. By default, the feature is turned off.
• scale.retry: How many times to attempt the scaling operation. Default = 2.

How to Dynamically Scale CPU and RAM

To modify the CPU and/or RAM capacity of a virtual machine, you need to change the compute offering of the VM to a new compute offering that has the desired CPU and RAM values. You can use the same steps described above in “Changing the Service Offering for a VM”, but skip the step where you stop the virtual machine. Of course, you might have to create a new compute offering first.

When you submit a dynamic scaling request, the resources will be scaled up on the current host if possible. If the host does not have enough resources, the VM will be live migrated to another host in the same cluster. If there is no host in the cluster that can fulfill the requested level of CPU and RAM, the scaling operation will fail. The VM will continue to run as it was before.

Limitations

• You can not do dynamic scaling for system VMs on XenServer.
• CloudStack will not check to be sure that the new CPU and RAM levels are compatible with the OS running on the VM.
• When scaling memory or CPU for a Linux VM on VMware, you might need to run scripts in addition to the other steps mentioned above. For more information, see Hot adding memory in Linux (1012764) in the VMware Knowledge Base.
• (VMware) If resources are not available on the current host, scaling up will fail on VMware because of a known issue where CloudStack and vCenter calculate the available capacity differently. For more information, see https://issues.apache.org/jira/browse/CLOUDSTACK-1809.
• On VMs running Linux 64-bit and Windows 7 32-bit operating systems, if the VM is initially assigned a RAM of less than 3 GB, it can be dynamically scaled up to 3 GB, but not more. This is due to a known issue with these operating systems, which will freeze if an attempt is made to dynamically scale from less than 3 GB to more than 3 GB.

Resetting the Virtual Machine Root Volume on Reboot

For secure environments, and to ensure that VM state is not persisted across reboots, you can reset the root disk. For more information, see “Reset VM to New Root Disk on Reboot”.

Moving VMs Between Hosts (Manual Live Migration)

The CloudStack administrator can move a running VM from one host to another without interrupting service to users or going into maintenance mode. This is called manual live migration, and can be done under the following conditions:

• The root administrator is logged in. Domain admins and users can not perform manual live migration of VMs.
• The VM is running. Stopped VMs can not be live migrated.
• The destination host must have enough available capacity. If not, the VM will remain in the “migrating” state until memory becomes available.
• (KVM) The VM must not be using local disk storage. (On XenServer and VMware, VM live migration with local disk is enabled by CloudStack support for XenMotion and vMotion.)
• (KVM) The destination host must be in the same cluster as the original host. (On XenServer and VMware, VM live migration from one cluster to another is enabled by CloudStack support for XenMotion and vMotion.)

To manually live migrate a virtual machine

1. Log in to the CloudStack UI as a user or admin.

2. In the left navigation, click Instances.

3. Choose the VM that you want to migrate.

4. Click the Migrate Instance button.

5. From the list of suitable hosts, choose the one to which you want to move the VM.

   Note

   If the VM’s storage has to be migrated along with the VM, this will be noted in the host list. CloudStack will take care of the storage migration for you.

6. Click OK.

Note

(KVM) If the VM’s storage has to be migrated along with the VM, from a mounted NFS storage pool to a cluster-wide mounted NFS storage pool, then the ‘migrateVirtualMachineWithVolume’ API has to be used. There is no UI integration for this feature.

(CloudMonkey) > migrate virtualmachinewithvolume virtualmachineid=<virtual machine uuid> hostid=<destination host uuid> migrateto[i].volume=<virtual machine volume number i uuid> migrateto[i].pool=<destination storage pool uuid for volume number i>

where i in [0,..,N] and N = number of volumes of the virtual machine

Deleting VMs

Users can delete their own virtual machines. A running virtual machine will be abruptly stopped before it is deleted. Administrators can delete any virtual machines.

To delete a virtual machine:

1. Log in to the CloudStack UI as a user or admin.
2. In the left navigation, click Instances.
3. Choose the VM that you want to delete.
4. Click the Destroy Instance button.
5. Optionally both expunging and the deletion of any attached volumes can be enabled.

Working with ISOs

CloudStack supports ISOs and their attachment to guest VMs. An ISO is a read-only file that has an ISO/CD-ROM style file system. Users can upload their own ISOs and mount them on their guest VMs.

ISOs are uploaded based on a URL. HTTP is the supported protocol. Once the ISO is available via HTTP specify an upload URL such as http://my.web.server/filename.iso.

ISOs may be public or private, like templates.ISOs are not hypervisor-specific. That is, a guest on vSphere can mount the exact same image that a guest on KVM can mount.

ISO images may be stored in the system and made available with a privacy level similar to templates. ISO images are classified as either bootable or not bootable. A bootable ISO image is one that contains an OS image. CloudStack allows a user to boot a guest VM off of an ISO image. Users can also attach ISO images to guest VMs. For example, this enables installing PV drivers into Windows. ISO images are not hypervisor-specific.

Adding an ISO

To make additional operating system or other software available for use with guest VMs, you can add an ISO. The ISO is typically thought of as an operating system image, but you can also add ISOs for other types of software, such as desktop applications that you want to be installed as part of a template.

1. Log in to the CloudStack UI as an administrator or end user.

2. In the left navigation bar, click Templates.

3. In Select View, choose ISOs.

4. Click Add ISO.

5. In the Add ISO screen, provide the following:

   • Name: Short name for the ISO image. For example, CentOS 6.2 64-bit.

   • Description: Display test for the ISO image. For example, CentOS 6.2 64-bit.

   • URL: The URL that hosts the ISO image. The Management Server must be able to access this location via HTTP. If needed you can place the ISO image directly on the Management Server

   • Zone: Choose the zone where you want the ISO to be available, or All Zones to make it available throughout CloudStack.

   • Bootable: Whether or not a guest could boot off this ISO image. For example, a CentOS ISO is bootable, a Microsoft Office ISO is not bootable.

   • OS Type: This helps CloudStack and the hypervisor perform certain operations and make assumptions that improve the performance of the guest. Select one of the following.

     • If the operating system of your desired ISO image is listed, choose it.
     • If the OS Type of the ISO is not listed or if the ISO is not bootable, choose Other.
     • (XenServer only) If you want to boot from this ISO in PV mode, choose Other PV (32-bit) or Other PV (64-bit)
     • (KVM only) If you choose an OS that is PV-enabled, the VMs created from this ISO will have a SCSI (virtio) root disk. If the OS is not PV-enabled, the VMs will have an IDE root disk. The PV-enabled types are:
       • Fedora 13
       • Fedora 12
       • Fedora 11
       • Fedora 10
       • Fedora 9
       • Other PV
       • Debian GNU/Linux
       • CentOS 5.3
       • CentOS 5.4
       • CentOS 5.5
       • Red Hat Enterprise Linux 5.3
       • Red Hat Enterprise Linux 5.4
       • Red Hat Enterprise Linux 5.5
       • Red Hat Enterprise Linux 6

     Note

     It is not recommended to choose an older version of the OS than the version in the image. For example, choosing CentOS 5.4 to support a CentOS 6.2 image will usually not work. In these cases, choose Other.

   • Extractable: Choose Yes if the ISO should be available for extraction.

   • Public: Choose Yes if this ISO should be available to other users.

   • Featured: Choose Yes if you would like this ISO to be more prominent for users to select. The ISO will appear in the Featured ISOs list. Only an administrator can make an ISO Featured.

6. Click OK.

   The Management Server will download the ISO. Depending on the size of the ISO, this may take a long time. The ISO status column will display Ready once it has been successfully downloaded into secondary storage. Clicking Refresh updates the download percentage.

7. Important: Wait for the ISO to finish downloading. If you move on to the next task and try to use the ISO right away, it will appear to fail. The entire ISO must be available before CloudStack can work with it.

Attaching an ISO to a VM

1. In the left navigation, click Instances.
2. Choose the virtual machine you want to work with.
3. Click the Attach ISO button.
4. In the Attach ISO dialog box, select the desired ISO.
5. Click OK.

Changing a VM’s Base Image

Every VM is created from a base image, which is a template or ISO which has been created and stored in CloudStack. Both cloud administrators and end users can create and modify templates, ISOs, and VMs.

In CloudStack, you can change an existing VM’s base image from one template to another, or from one ISO to another. (You can not change from an ISO to a template, or from a template to an ISO).

For example, suppose there is a template based on a particular operating system, and the OS vendor releases a software patch. The administrator or user naturally wants to apply the patch and then make sure existing VMs start using it. Whether a software update is involved or not, it’s also possible to simply switch a VM from its current template to any other desired template.

To change a VM’s base image, call the restoreVirtualMachine API command and pass in the virtual machine ID and a new template ID. The template ID parameter may refer to either a template or an ISO, depending on which type of base image the VM was already using (it must match the previous type of image). When this call occurs, the VM’s root disk is first destroyed, then a new root disk is created from the source designated in the template ID parameter. The new root disk is attached to the VM, and now the VM is based on the new template.

You can also omit the template ID parameter from the restoreVirtualMachine call. In this case, the VM’s root disk is destroyed and recreated, but from the same template or ISO that was already in use by the VM.

Using SSH Keys for Authentication

In addition to the username and password authentication, CloudStack supports using SSH keys to log in to the cloud infrastructure for additional security. You can use the createSSHKeyPair API to generate the SSH keys.

Because each cloud user has their own SSH key, one cloud user cannot log in to another cloud user’s instances unless they share their SSH key files. Using a single SSH key pair, you can manage multiple instances.

Creating an Instance Template that Supports SSH Keys

Create an instance template that supports SSH Keys.

1. Create a new instance by using the template provided by cloudstack.

   For more information on creating a new instance, see

2. Download the cloudstack script from The SSH Key Gen Script to the instance you have created.

   wget http://downloads.sourceforge.net/project/cloudstack/SSH%20Key%20Gen%20Script/cloud-set-guest-sshkey.in?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcloudstack%2Ffiles%2FSSH%2520Key%2520Gen%2520Script%2F&ts=1331225219&use_mirror=iweb

3. Copy the file to /etc/init.d.

   cp cloud-set-guest-sshkey.in /etc/init.d/

4. Give the necessary permissions on the script:

   chmod +x /etc/init.d/cloud-set-guest-sshkey.in

5. Run the script while starting up the operating system:

   chkconfig --add cloud-set-guest-sshkey.in

6. Stop the instance.

Creating the SSH Keypair

You must make a call to the createSSHKeyPair api method. You can either use the CloudStack Python API library or the curl commands to make the call to the cloudstack api.

For example, make a call from the cloudstack server to create a SSH keypair called “keypair-doc” for the admin account in the root domain:

Note

Ensure that you adjust these values to meet your needs. If you are making the API call from a different server, your URL/PORT will be different, and you will need to use the API keys.

1. Run the following curl command:

   curl --globoff "http://localhost:8096/?command=createSSHKeyPair&name=keypair-doc&account=admin&domainid=5163440e-c44b-42b5-9109-ad75cae8e8a2"

   The output is something similar to what is given below:

   <?xml version="1.0" encoding="ISO-8859-1"?><createsshkeypairresponse cloud-stack-version="3.0.0.20120228045507"><keypair><name>keypair-doc</name><fingerprint>f6:77:39:d5:5e:77:02:22:6a:d8:7f:ce:ab:cd:b3:56</fingerprint><privatekey>-----BEGIN RSA PRIVATE KEY-----
   MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci
   dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB
   AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu
   mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy
   QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7
   VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK
   lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm
   nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14
   4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd
   KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3
   -----END RSA PRIVATE KEY-----
   </privatekey></keypair></createsshkeypairresponse>

2. Copy the key data into a file. The file looks like this:

   -----BEGIN RSA PRIVATE KEY-----
   MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci
   dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB
   AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu
   mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy
   QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7
   VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK
   lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm
   nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14
   4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd
   KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3
   -----END RSA PRIVATE KEY-----

3. Save the file.

Creating an Instance

After you save the SSH keypair file, you must create an instance by using the template that you created at Section 5.2.1, “ Creating an Instance Template that Supports SSH Keys”. Ensure that you use the same SSH key name that you created at Section 5.2.2, “Creating the SSH Keypair”.

Note

You cannot create the instance by using the GUI at this time and associate the instance with the newly created SSH keypair.

A sample curl command to create a new instance is:

curl --globoff http://localhost:<port number>/?command=deployVirtualMachine&zoneId=1&serviceOfferingId=18727021-7556-4110-9322-d625b52e0813&templateId=e899c18a-ce13-4bbf-98a9-625c5026e0b5&securitygroupids=ff03f02f-9e3b-48f8-834d-91b822da40c5&account=admin&domainid=1&keypair=keypair-doc

Substitute the template, service offering and security group IDs (if you are using the security group feature) that are in your cloud environment.

Logging In Using the SSH Keypair

To test your SSH key generation is successful, check whether you can log in to the cloud setup.

For example, from a Linux OS, run:

ssh -i ~/.ssh/keypair-doc <ip address>

The -i parameter tells the ssh client to use a ssh key found at ~/.ssh/keypair-doc.

Resetting SSH Keys

With the API command resetSSHKeyForVirtualMachine, a user can set or reset the SSH keypair assigned to a virtual machine. A lost or compromised SSH keypair can be changed, and the user can access the VM by using the new keypair. Just create or register a new keypair, then call resetSSHKeyForVirtualMachine.

User-Data and Meta-Data

CloudStack provides API access to attach up to 2KB of data after base64 encoding to a deployed VM. Using HTTP POST(via POST body), you can send up to 32K of data after base64 encoding. Deployed VMs also have access to instance metadata via the virtual router.

Create virtual machine thru the API: deployVirtualMachine using the parameter userdata= to include user-data formated in base64.

Accessed user-data from VM. Once the IP address of the virtual router is known, use the following steps to retrieve user-data:

1. Run the following command to find the virtual router.

   # cat /var/lib/dhclient/dhclient-eth0.leases | grep dhcp-server-identifier | tail -1

2. Access user-data by running the following command using the result of the above command

   # curl http://10.1.1.1/latest/user-data

Meta Data can be accessed similarly, using a URL of the form http://10.1.1.1/latest/meta-data/{metadata type}. (For backwards compatibility, the previous URL http://10.1.1.1/latest/{metadata type} is also supported.) For metadata type, use one of the following:

• service-offering. A description of the VMs service offering
• availability-zone. The Zone name
• local-ipv4. The guest IP of the VM
• local-hostname. The hostname of the VM
• public-ipv4. The first public IP for the router. (E.g. the first IP of eth2)
• public-hostname. This is the same as public-ipv4
• instance-id. The instance name of the VM

Using Cloud-Init

Cloud-Init can be use to access an interpret user-data from virtual machines. Cloud-Init be installed into templates and also require CloudStack password and sshkey scripts (Adding Password Management to Your Templates and using ssh keys). User password management and resetSSHKeyForVirtualMachine API are not yet supported by cloud-init.

1. Install cloud-init package into a template:

   # yum install cloud-init
     or
   $ sudo apt-get install cloud-init

2. Create datasource configuration file: /etc/cloud/cloud.cfg.d/99_cloudstack.cfg

   datasource:
     CloudStack: {}
     None: {}
   datasource_list:
     - CloudStack

user-data example

This example use cloud-init to Upgrade Operating-System of the newly created VM:

#cloud-config
# Upgrade the instance on first boot
# (ie run apt-get upgrade)
#
# Default: false
# Aliases: apt_upgrade
package_upgrade: true

base64 formated:

I2Nsb3VkLWNvbmZpZw0KDQojIFVwZ3JhZGUgdGhlIGluc3RhbmNlIG9uIGZpcnN0IGJvb3QNCiMgKGllIHJ1biBhcHQtZ2V0IHVwZ3JhZGUpDQojDQojIERlZmF1bHQ6IGZhbHNlDQojIEFsaWFzZXM6IGFwdF91cGdyYWRlDQpwYWNrYWdlX3VwZ3JhZGU6IHRydWUNCg==

Refer to Cloud-Init CloudStack datasource documentation for latest capabilities. Cloud-Init and Cloud-Init CloudStack datasource are not supported by Apache CloudStack community.

Assigning GPU/vGPU to Guest VMs

CloudStack can deploy guest VMs with Graphics Processing Unit (GPU) or Virtual Graphics Processing Unit (vGPU) capabilities on XenServer hosts. At the time of VM deployment or at a later stage, you can assign a physical GPU ( known as GPU-passthrough) or a portion of a physical GPU card (vGPU) to a guest VM by changing the Service Offering. With this capability, the VMs running on CloudStack meet the intensive graphical processing requirement by means of the high computation power of GPU/vGPU, and CloudStack users can run multimedia rich applications, such as Auto-CAD, that they otherwise enjoy at their desk on a virtualized environment. CloudStack leverages the XenServer support for NVIDIA GRID Kepler 1 and 2 series to run GPU/vGPU enabled VMs. NVIDIA GRID cards allows sharing a single GPU cards among multiple VMs by creating vGPUs for each VM. With vGPU technology, the graphics commands from each VM are passed directly to the underlying dedicated GPU, without the intervention of the hypervisor. This allows the GPU hardware to be time-sliced and shared across multiple VMs. XenServer hosts use the GPU cards in following ways:

GPU passthrough: GPU passthrough represents a physical GPU which can be directly assigned to a VM. GPU passthrough can be used on a hypervisor alongside GRID vGPU, with some restrictions: A GRID physical GPU can either host GRID vGPUs or be used as passthrough, but not both at the same time.

GRID vGPU: GRID vGPU enables multiple VMs to share a single physical GPU. The VMs run an NVIDIA driver stack and get direct access to the GPU. GRID physical GPUs are capable of supporting multiple virtual GPU devices (vGPUs) that can be assigned directly to guest VMs. Guest VMs use GRID virtual GPUs in the same manner as a physical GPU that has been passed through by the hypervisor: an NVIDIA driver loaded in the guest VM provides direct access to the GPU for performance-critical fast paths, and a paravirtualized interface to the GRID Virtual GPU Manager, which is used for nonperformant management operations. NVIDIA GRID Virtual GPU Manager for XenServer runs in dom0. CloudStack provides you with the following capabilities:

• Adding XenServer hosts with GPU/vGPU capability provisioned by the administrator.
• Creating a Compute Offering with GPU/vGPU capability.
• Deploying a VM with GPU/vGPU capability.
• Destroying a VM with GPU/vGPU capability.
• Allowing an user to add GPU/vGPU support to a VM without GPU/vGPU support by changing the Service Offering and vice-versa.
• Migrating VMs (cold migration) with GPU/vGPU capability.
• Managing GPU cards capacity.
• Querying hosts to obtain information about the GPU cards, supported vGPU types in case of GRID cards, and capacity of the cards.

Prerequisites and System Requirements

Before proceeding, ensure that you have these prerequisites:

• The vGPU-enabled XenServer 6.2 and later versions. For more information, see Citrix 3D Graphics Pack.
• GPU/vPGU functionality is supported for following HVM guest operating systems: For more information, see Citrix 3D Graphics Pack.
• Windows 7 (x86 and x64)
• Windows Server 2008 R2
• Windows Server 2012
• Windows 8 (x86 and x64)
• Windows 8.1 (“Blue”) (x86 and x64)
• Windows Server 2012 R2 (server equivalent of “Blue”)
• CloudStack does not restrict the deployment of GPU-enabled VMs with guest OS types that are not supported by XenServer for GPU/vGPU functionality. The deployment would be successful and a GPU/vGPU will also get allocated for VMs; however, due to missing guest OS drivers, VM would not be able to leverage GPU resources. Therefore, it is recommended to use GPU-enabled service offering only with supported guest OS.
• NVIDIA GRID K1 (16 GiB video RAM) AND K2 (8 GiB of video RAM) cards supports homogeneous virtual GPUs, implies that at any given time, the vGPUs resident on a single physical GPU must be all of the same type. However, this restriction doesn’t extend across physical GPUs on the same card. Each physical GPU on a K1 or K2 may host different types of virtual GPU at the same time. For example, a GRID K2 card has two physical GPUs, and supports four types of virtual GPU; GRID K200, GRID K220Q, GRID K240Q, AND GRID K260Q.
• NVIDIA driver must be installed to enable vGPU operation as for a physical NVIDIA GPU.
• XenServer tools are installed in the VM to get maximum performance on XenServer, regardless of type of vGPU you are using. Without the optimized networking and storage drivers that the XenServer tools provide, remote graphics applications running on GRID vGPU will not deliver maximum performance.
• To deliver high frames from multiple heads on vGPU, install XenDesktop with HDX 3D Pro remote graphics.

Before continuing with configuration, consider the following:

• Deploying VMs GPU/vGPU capability is not supported if hosts are not available with enough GPU capacity.
• A Service Offering cannot be created with the GPU values that are not supported by CloudStack UI. However, you can make an API call to achieve this.
• Dynamic scaling is not supported. However, you can choose to deploy a VM without GPU support, and at a later point, you can change the system offering to upgrade to the one with vGPU. You can achieve this by offline upgrade: stop the VM, upgrade the Service Offering to the one with vGPU, then start the VM.
• Live migration of GPU/vGPU enabled VM is not supported.
• Limiting GPU resources per Account/Domain is not supported.
• Disabling GPU at Cluster level is not supported.
• Notification thresholds for GPU resource is not supported.

Supported GPU Devices

GPU/vGPU Assignment Workflow

CloudStack follows the below sequence of operations to provide GPU/vGPU support for VMs:

1. Ensure that XenServer host is ready with GPU installed and configured. For more information, see Citrix 3D Graphics Pack.

2. Add the host to CloudStack. CloudStack checks if the host is GPU-enabled or not. CloudStack queries the host and detect if it’s GPU enabled.

3. Create a compute offering with GPU/vGPU support: For more information, see Creating a New Compute Offering..

4. Continue with any of the following operations:

   • Deploy a VM.

     Deploy a VM with GPU/vGPU support by selecting appropriate Service Offering. CloudStack decide which host to choose for VM deployment based on following criteria:

     • Host has GPU cards in it. In case of vGPU, CloudStack checks if cards have the required vGPU type support and enough capacity available. Having no appropriate hosts results in an InsufficientServerCapacity exception.
     • Alternately, you can choose to deploy a VM without GPU support, and at a later point, you can change the system offering. You can achieve this by offline upgrade: stop the VM, upgrade the Service Offering to the one with vGPU, then start the VM. In this case, CloudStack gets a list of hosts which have enough capacity to host the VM. If there is a GPU-enabled host, CloudStack reorders this host list and place the GPU-enabled hosts at the bottom of the list.

   • Migrate a VM.

     CloudStack searches for hosts available for VM migration, which satisfies GPU requirement. If the host is available, stop the VM in the current host and perform the VM migration task. If the VM migration is successful, the remaining GPU capacity is updated for both the hosts accordingly.

   • Destroy a VM.

     GPU resources are released automatically when you stop a VM. Once the destroy VM is successful, CloudStack will make a resource call to the host to get the remaining GPU capacity in the card and update the database accordingly.

VM Instance Settings

Each user VM has a set of “details” associated with it (as visible via listVirtualMachine API call) - those “details” are shown on the “Settings” tab of the VM in the GUI (words “setting(s)” and “detail(s)” are here used interchangeably).

The Settings tab is always present/visible, but settings can be changed only when the VM is in a Stopped state. Some VM details/settings can be hidden via “user.vm.blacklisted.details” global setting (you can find below the list of those hidden by default).

When adding a new setting or modifying the existing ones, setting names are shown/offered in a drop-down list, as well as their possible values (with the exception of boolean or numerical values).

Read-only details/settings that are hidden by default:

• rootdisksize
• cpuOvercommitRatio
• memoryOvercommitRatio
• Message.ReservedCapacityFreed.Flag

An example list of settings as well as their possible values are shown on the images below:

(VMware hypervisor)

(VMware disk controllers)

(VMware NIC models)

(KVM disk controllers)