Startup Scripts

KubeVirt supports the ability to assign a startup script to a VirtualMachineInstance instance which is executed automatically when the VM initializes.

These scripts are commonly used to automate injection of users and SSH keys into VMs in order to provide remote access to the machine. For example, a startup script can be used to inject credentials into a VM that allows an Ansible job running on a remote host to access and provision the VM.

Startup scripts are not limited to any specific use case though. They can be used to run any arbitrary script in a VM on boot.

Cloud-init

cloud-init is a widely adopted project used for early initialization of a VM. Used by cloud providers such as AWS and GCP, cloud-init has established itself as the defacto method of providing startup scripts to VMs.

Cloud-init documentation can be found here: Cloud-init Documentation.

KubeVirt supports cloud-init’s “NoCloud” and “ConfigDrive” datasources which involve injecting startup scripts into a VM instance through the use of an ephemeral disk. VMs with the cloud-init package installed will detect the ephemeral disk and execute custom userdata scripts at boot.

Sysprep

Sysprep is an automation tool for Windows that automates Windows installation, setup, and custom software provisioning.

The general flow is:

  1. Seal the vm image with the Sysprep tool, for example by running:

    1. %WINDIR%\system32\sysprep\sysprep.exe /generalize /shutdown /oobe /mode:vm

    Note

    We need to make sure the base vm does not restart, which can be done by setting the vm run strategy as RerunOnFailure.

    VM runStrategy:

    1. spec:
    2.   runStrategy: RerunOnFailure

    More information can be found here:

    Note

    It is important that there is no answer file detected when the Sysprep Tool is triggered, because Windows Setup searches for answer files at the beginning of each configuration pass and caches it. If that happens, when the OS will start - it will just use the cached answer file, ignoring the one we provide through the Sysprep API. More information can be found here.

  2. Providing an Answer file named autounattend.xml in an attached media. The answer file can be provided in a ConfigMap or a Secret with the key autounattend.xml

    More information can be found here: Answer files (unattend.xml)

    Note

    There are also many easy to find online tools available for creating an answer file.

Cloud-init Examples

User Data

KubeVirt supports the cloud-init NoCloud and ConfigDrive data sources which involve injecting startup scripts through the use of a disk attached to the VM.

In order to assign a custom userdata script to a VirtualMachineInstance using this method, users must define a disk and a volume for the NoCloud or ConfigDrive datasource in the VirtualMachineInstance’s spec.

Data Sources

Under most circumstances users should stick to the NoCloud data source as it is the simplest cloud-init data source. Only if NoCloud is not supported by the cloud-init implementation (e.g. coreos-cloudinit) users should switch the data source to ConfigDrive.

Switching the cloud-init data source to ConfigDrive is as easy as changing the volume type in the VirtualMachineInstance’s spec from cloudInitNoCloud to cloudInitConfigDrive.

NoCloud data source:

  1. volumes:
  2.   - name: cloudinitvolume
  3.     cloudInitNoCloud:
  4.       userData: "#cloud-config"

ConfigDrive data source:

  1. volumes:
  2.   - name: cloudinitvolume
  3.     cloudInitConfigDrive:
  4.       userData: "#cloud-config"

See the examples below for more complete cloud-init examples.

Cloud-init user-data as clear text

In the example below, a SSH key is stored in the cloudInitNoCloud Volume’s userData field as clean text. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM’s device.

  1. # Create a VM manifest with the startup script
  2. # a cloudInitNoCloud volume's userData field.
  4. cat << END > my-vmi.yaml
  5. apiVersion: kubevirt.io/v1alpha3
  6. kind: VirtualMachineInstance
  7. metadata:
  8.   name: myvmi
  9. spec:
  10.   terminationGracePeriodSeconds: 5
  11.   domain:
  12.     resources:
  13.       requests:
  14.         memory: 64M
  15.     devices:
  16.       disks:
  17.       - name: containerdisk
  18.         disk:
  19.           bus: virtio
  20.       - name: cloudinitdisk
  21.         disk:
  22.           bus: virtio
  23.   volumes:
  24.     - name: containerdisk
  25.       containerDisk:
  26.         image: kubevirt/cirros-container-disk-demo:latest
  27.     - name: cloudinitdisk
  28.       cloudInitNoCloud:
  29.         userData: |
  30.           ssh_authorized_keys:
  31.             - ssh-rsa AAAAB3NzaK8L93bWxnyp test@test.com
  33. END
  35. # Post the Virtual Machine spec to KubeVirt.
  37. kubectl create -f my-vmi.yaml

Cloud-init user-data as base64 string

In the example below, a simple bash script is base64 encoded and stored in the cloudInitNoCloud Volume’s userDataBase64 field. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM’s device.

Users also have the option of storing the startup script in a Kubernetes Secret and referencing the Secret in the VM’s spec. Examples further down in the document illustrate how that is done.

  1. # Create a simple startup script
  3. cat << END > startup-script.sh
  4. #!/bin/bash
  5. echo "Hi from startup script!"
  6. END
  8. # Create a VM manifest with the startup script base64 encoded into
  9. # a cloudInitNoCloud volume's userDataBase64 field.
  11. cat << END > my-vmi.yaml
  12. apiVersion: kubevirt.io/v1alpha3
  13. kind: VirtualMachineInstance
  14. metadata:
  15.   name: myvmi
  16. spec:
  17.   terminationGracePeriodSeconds: 5
  18.   domain:
  19.     resources:
  20.       requests:
  21.         memory: 64M
  22.     devices:
  23.       disks:
  24.       - name: containerdisk
  25.         disk:
  26.           bus: virtio
  27.       - name: cloudinitdisk
  28.         disk:
  29.           bus: virtio
  30.   volumes:
  31.     - name: containerdisk
  32.       containerDisk:
  33.         image: kubevirt/cirros-container-disk-demo:latest
  34.     - name: cloudinitdisk
  35.       cloudInitNoCloud:
  36.         userDataBase64: $(cat startup-script.sh | base64 -w0)
  37. END
  39. # Post the Virtual Machine spec to KubeVirt.
  41. kubectl create -f my-vmi.yaml

Cloud-init UserData as k8s Secret

Users who wish to not store the cloud-init userdata directly in the VirtualMachineInstance spec have the option to store the userdata into a Kubernetes Secret and reference that Secret in the spec.

Multiple VirtualMachineInstance specs can reference the same Kubernetes Secret containing cloud-init userdata.

Below is an example of how to create a Kubernetes Secret containing a startup script and reference that Secret in the VM’s spec.

  1. # Create a simple startup script
  3. cat << END > startup-script.sh
  4. #!/bin/bash
  5. echo "Hi from startup script!"
  6. END
  8. # Store the startup script in a Kubernetes Secret
  9. kubectl create secret generic my-vmi-secret --from-file=userdata=startup-script.sh
  11. # Create a VM manifest and reference the Secret's name in the cloudInitNoCloud
  12. # Volume's secretRef field
  14. cat << END > my-vmi.yaml
  15. apiVersion: kubevirt.io/v1alpha3
  16. kind: VirtualMachineInstance
  17. metadata:
  18.   name: myvmi
  19. spec:
  20.   terminationGracePeriodSeconds: 5
  21.   domain:
  22.     resources:
  23.       requests:
  24.         memory: 64M
  25.     devices:
  26.       disks:
  27.       - name: containerdisk
  28.         disk:
  29.           bus: virtio
  30.       - name: cloudinitdisk
  31.         disk:
  32.           bus: virtio
  33.   volumes:
  34.     - name: containerdisk
  35.       containerDisk:
  36.         image: kubevirt/cirros-registry-disk-demo:latest
  37.     - name: cloudinitdisk
  38.       cloudInitNoCloud:
  39.         secretRef:
  40.           name: my-vmi-secret
  41. END
  43. # Post the VM
  44. kubectl create -f my-vmi.yaml

Injecting SSH keys with Cloud-init’s Cloud-config

In the examples so far, the cloud-init userdata script has been a bash script. Cloud-init has it’s own configuration that can handle some common tasks such as user creation and SSH key injection.

More cloud-config examples can be found here: Cloud-init Examples

Below is an example of using cloud-config to inject an SSH key for the default user (fedora in this case) of a Fedora Atomic disk image.

  1. # Create the cloud-init cloud-config userdata.
  2. cat << END > startup-script
  3. #cloud-config
  4. password: atomic
  5. chpasswd: { expire: False }
  6. ssh_pwauth: False
  7. ssh_authorized_keys:
  8.     - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J fedora@localhost.localdomain
  9. END
  11. # Create the VM spec
  12. cat << END > my-vmi.yaml
  13. apiVersion: kubevirt.io/v1alpha3
  14. kind: VirtualMachineInstance
  15. metadata:
  16.   name: sshvmi
  17. spec:
  18.   terminationGracePeriodSeconds: 0
  19.   domain:
  20.     resources:
  21.       requests:
  22.         memory: 1024M
  23.     devices:
  24.       disks:
  25.       - name: containerdisk
  26.         disk:
  27.           dev: vda
  28.       - name: cloudinitdisk
  29.         disk:
  30.           dev: vdb
  31.   volumes:
  32.     - name: containerdisk
  33.       containerDisk:
  34.         image: kubevirt/fedora-atomic-registry-disk-demo:latest
  35.     - name: cloudinitdisk
  36.       cloudInitNoCloud:
  37.         userDataBase64: $(cat startup-script | base64 -w0)
  38. END
  40. # Post the VirtualMachineInstance spec to KubeVirt.
  41. kubectl create -f my-vmi.yaml
  43. # Connect to VM with passwordless SSH key
  44. ssh -i <insert private key here> fedora@<insert ip here>

Inject SSH key using a Custom Shell Script

Depending on the boot image in use, users may have a mixed experience using cloud-init’s cloud-config to create users and inject SSH keys.

Below is an example of creating a user and injecting SSH keys for that user using a script instead of cloud-config.

  1. cat << END > startup-script.sh
  2. #!/bin/bash
  3. export NEW_USER="foo"
  4. export SSH_PUB_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J $NEW_USER@localhost.localdomain"
  6. sudo adduser -U -m $NEW_USER
  7. echo "$NEW_USER:atomic" | chpasswd
  8. sudo mkdir /home/$NEW_USER/.ssh
  9. sudo echo "$SSH_PUB_KEY" > /home/$NEW_USER/.ssh/authorized_keys
  10. sudo chown -R ${NEW_USER}: /home/$NEW_USER/.ssh
  11. END
  13. # Create the VM spec
  14. cat << END > my-vmi.yaml
  15. apiVersion: kubevirt.io/v1alpha3
  16. kind: VirtualMachineInstance
  17. metadata:
  18.   name: sshvmi
  19. spec:
  20.   terminationGracePeriodSeconds: 0
  21.   domain:
  22.     resources:
  23.       requests:
  24.         memory: 1024M
  25.     devices:
  26.       disks:
  27.       - name: containerdisk
  28.         disk:
  29.           dev: vda
  30.       - name: cloudinitdisk
  31.         disk:
  32.           dev: vdb
  33.   volumes:
  34.     - name: containerdisk
  35.       containerDisk:
  36.         image: kubevirt/fedora-atomic-registry-disk-demo:latest
  37.     - name: cloudinitdisk
  38.       cloudInitNoCloud:
  39.         userDataBase64: $(cat startup-script.sh | base64 -w0)
  40. END
  42. # Post the VirtualMachineInstance spec to KubeVirt.
  43. kubectl create -f my-vmi.yaml
  45. # Connect to VM with passwordless SSH key
  46. ssh -i <insert private key here> foo@<insert ip here>

Network Config

A cloud-init network version 1 configuration can be set to configure the network at boot.

Cloud-init user-data must be set for cloud-init to parse network-config even if it is just the user-data config header:

  1. #cloud-config

Cloud-init network-config as clear text

In the example below, a simple cloud-init network-config is stored in the cloudInitNoCloud Volume’s networkData field as clean text. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM’s device.

  1. # Create a VM manifest with the network-config in
  2. # a cloudInitNoCloud volume's networkData field.
  4. cat << END > my-vmi.yaml
  5. apiVersion: kubevirt.io/v1alpha2
  6. kind: VirtualMachineInstance
  7. metadata:
  8.   name: myvmi
  9. spec:
  10.   terminationGracePeriodSeconds: 5
  11.   domain:
  12.     resources:
  13.       requests:
  14.         memory: 64M
  15.     devices:
  16.       disks:
  17.       - name: containerdisk
  18.         volumeName: registryvolume
  19.         disk:
  20.           bus: virtio
  21.       - name: cloudinitdisk
  22.         volumeName: cloudinitvolume
  23.         disk:
  24.           bus: virtio
  25.   volumes:
  26.     - name: registryvolume
  27.       containerDisk:
  28.         image: kubevirt/cirros-container-disk-demo:latest
  29.     - name: cloudinitvolume
  30.       cloudInitNoCloud:
  31.         userData: "#cloud-config"
  32.         networkData: |
  33.           network:
  34.             version: 1
  35.             config:
  36.             - type: physical
  37.             name: eth0
  38.             subnets:
  39.               - type: dhcp
  41. END
  43. # Post the Virtual Machine spec to KubeVirt.
  45. kubectl create -f my-vmi.yaml

Cloud-init network-config as base64 string

In the example below, a simple network-config is base64 encoded and stored in the cloudInitNoCloud Volume’s networkDataBase64 field. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM’s device.

Users also have the option of storing the network-config in a Kubernetes Secret and referencing the Secret in the VM’s spec. Examples further down in the document illustrate how that is done.

  1. # Create a simple network-config
  3. cat << END > network-config
  4. network:
  5.   version: 1
  6.   config:
  7.   - type: physical
  8.   name: eth0
  9.   subnets:
  10.     - type: dhcp
  11. END
  13. # Create a VM manifest with the networkData base64 encoded into
  14. # a cloudInitNoCloud volume's networkDataBase64 field.
  16. cat << END > my-vmi.yaml
  17. apiVersion: kubevirt.io/v1alpha2
  18. kind: VirtualMachineInstance
  19. metadata:
  20.   name: myvmi
  21. spec:
  22.   terminationGracePeriodSeconds: 5
  23.   domain:
  24.     resources:
  25.       requests:
  26.         memory: 64M
  27.     devices:
  28.       disks:
  29.       - name: containerdisk
  30.         volumeName: registryvolume
  31.         disk:
  32.           bus: virtio
  33.       - name: cloudinitdisk
  34.         volumeName: cloudinitvolume
  35.         disk:
  36.           bus: virtio
  37.   volumes:
  38.     - name: registryvolume
  39.       containerDisk:
  40.         image: kubevirt/cirros-container-disk-demo:latest
  41.     - name: cloudinitvolume
  42.       cloudInitNoCloud:
  43.         userData: "#cloud-config"
  44.         networkDataBase64: $(cat network-config | base64 -w0)
  45. END
  47. # Post the Virtual Machine spec to KubeVirt.
  49. kubectl create -f my-vmi.yaml

Cloud-init network-config as k8s Secret

Users who wish to not store the cloud-init network-config directly in the VirtualMachineInstance spec have the option to store the network-config into a Kubernetes Secret and reference that Secret in the spec.

Multiple VirtualMachineInstance specs can reference the same Kubernetes Secret containing cloud-init network-config.

Below is an example of how to create a Kubernetes Secret containing a network-config and reference that Secret in the VM’s spec.

  1. # Create a simple network-config
  3. cat << END > network-config
  4. network:
  5.   version: 1
  6.   config:
  7.   - type: physical
  8.   name: eth0
  9.   subnets:
  10.     - type: dhcp
  11. END
  13. # Store the network-config in a Kubernetes Secret
  14. kubectl create secret generic my-vmi-secret --from-file=networkdata=network-config
  16. # Create a VM manifest and reference the Secret's name in the cloudInitNoCloud
  17. # Volume's secretRef field
  19. cat << END > my-vmi.yaml
  20. apiVersion: kubevirt.io/v1alpha2
  21. kind: VirtualMachineInstance
  22. metadata:
  23.   name: myvmi
  24. spec:
  25.   terminationGracePeriodSeconds: 5
  26.   domain:
  27.     resources:
  28.       requests:
  29.         memory: 64M
  30.     devices:
  31.       disks:
  32.       - name: containerdisk
  33.         volumeName: registryvolume
  34.         disk:
  35.           bus: virtio
  36.       - name: cloudinitdisk
  37.         volumeName: cloudinitvolume
  38.         disk:
  39.           bus: virtio
  40.   volumes:
  41.     - name: registryvolume
  42.       containerDisk:
  43.         image: kubevirt/cirros-registry-disk-demo:latest
  44.     - name: cloudinitvolume
  45.       cloudInitNoCloud:
  46.         userData: "#cloud-config"
  47.         networkDataSecretRef:
  48.           name: my-vmi-secret
  49. END
  51. # Post the VM
  52. kubectl create -f my-vmi.yaml

Debugging

Depending on the operating system distribution in use, cloud-init output is often printed to the console output on boot up. When developing userdata scripts, users can connect to the VM’s console during boot up to debug.

Example of connecting to console using virtctl:

  1. virtctl console <name of vmi>

Device Role Tagging

KubeVirt provides a mechanism for users to tag devices such as Network Interfaces with a specific role. The tag will be matched to the hardware address of the device and this mapping exposed to the guest OS via cloud-init.

This additional metadata will help the guest OS users with multiple networks interfaces to identify the devices that may have a specific role, such as a network device dedicated to a specific service or a disk intended to be used by a specific application (database, webcache, etc.)

This functionality already exists in platforms such as OpenStack. KubeVirt will provide the data in a similar format, known to users and services like cloud-init.

For example:

  1. kind: VirtualMachineInstance
  2. spec:
  3.   domain:
  4.     devices:
  5.       interfaces:
  6.       - masquerade: {}
  7.         name: default
  8.       - bridge: {}
  9.         name: ptp
  10.     tag: ptp
  11.       - name: sriov-net
  12.         sriov: {}
  13.         tag: nfvfunc
  14.   networks:
  15.   - name: default
  16.     pod: {}
  17.   - multus:
  18.       networkName: ptp-conf
  19.     name: ptp
  20.       networkName: sriov/sriov-network
  21.     name: sriov-net
  23. The metadata will be available in the guests config drive `openstack/latest/meta_data.json`
  25. {
  26.   "devices": [
  27.     {
  28.         "type": "nic",
  29.         "bus": "pci",
  30.         "address": "0000:00:02.0",
  31.         "mac": "01:22:22:42:22:21",
  32.         "tags": ["ptp"]
  33.     },
  34.     {
  35.         "type": "nic",
  36.         "bus": "pci",
  37.         "address": "0000:81:10.1",
  38.         "mac": "01:22:22:42:22:22",
  39.         "tags": ["nfvfunc"]
  40.     },
  41.   ]
  42. }

Sysprep Examples

Sysprep in a ConfigMap

The answer file can be provided in a ConfigMap:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: sysprep-config
  5. data:
  6.   autounattend.xml: |
  7.     <?xml version="1.0" encoding="utf-8"?>
  8.     <unattend xmlns="urn:schemas-microsoft-com:unattend">
  9.     ...
  10.     </unattend>

And attached to the VM like so:

  1. kind: VirtualMachine
  2. metadata:
  3.   name: windows-with-sysprep
  4. spec:
  5.   running: false
  6.   template:
  7.     metadata:
  8.       labels:
  9.         kubevirt.io/domain: windows-with-sysprep
  10.     spec:
  11.       domain:
  12.         cpu:
  13.           cores: 3
  14.         devices:
  15.           disks:
  16.           - bootOrder: 1
  17.             disk:
  18.               bus: virtio
  19.             name: harddrive
  20.           - name: sysprep
  21.             cdrom:
  22.               bus: sata
  23.         machine:
  24.           type: q35
  25.         resources:
  26.           requests:
  27.             memory: 6G
  28.       volumes:
  29.       - name: harddrive
  30.         persistentVolumeClaim:
  31.           claimName: windows_pvc
  32.       - name: sysprep
  33.         sysprep:
  34.           configMap:
  35.             name: sysprep-config

Sysprep in a Secret

The answer file can be provided in a Secret:

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: sysprep-config
  5. stringData:
  6. data:
  7.   autounattend.xml: |
  8.     <?xml version="1.0" encoding="utf-8"?>
  9.     <unattend xmlns="urn:schemas-microsoft-com:unattend">
  10.     ...
  11.     </unattend>

And attached to the VM like so:

  1. kind: VirtualMachine
  2. metadata:
  3.   name: windows-with-sysprep
  4. spec:
  5.   running: false
  6.   template:
  7.     metadata:
  8.       labels:
  9.         kubevirt.io/domain: windows-with-sysprep
  10.     spec:
  11.       domain:
  12.         cpu:
  13.           cores: 3
  14.         devices:
  15.           disks:
  16.           - bootOrder: 1
  17.             disk:
  18.               bus: virtio
  19.             name: harddrive
  20.           - name: sysprep
  21.             cdrom:
  22.               bus: sata
  23.         machine:
  24.           type: q35
  25.         resources:
  26.           requests:
  27.             memory: 6G
  28.       volumes:
  29.       - name: harddrive
  30.         persistentVolumeClaim:
  31.           claimName: windows_pvc
  32.       - name: sysprep
  33.         sysprep:
  34.           secret:
  35.             name: sysprep-secret

Base Sysprep VM

In the example below, a configMap with autounattend.xml file is used to modify the Windows iso image which is downloaded from Microsoft and creates a base installed Windows machine with virtio drivers installed and all the commands executed in post-install.ps1 For the below manifests to work it needs to have win10-iso DataVolume.

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: win10-template-configmap
  5. data:
  6.   autounattend.xml: |-
  7.     <?xml version="1.0" encoding="utf-8"?>
  8.     <unattend xmlns="urn:schemas-microsoft-com:unattend">
  9.       <settings pass="windowsPE">
  10.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-International-Core-WinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  11.           <SetupUILanguage>
  12.             <UILanguage>en-US</UILanguage>
  13.           </SetupUILanguage>
  14.           <InputLocale>0409:00000409</InputLocale>
  15.           <SystemLocale>en-US</SystemLocale>
  16.           <UILanguage>en-US</UILanguage>
  17.           <UILanguageFallback>en-US</UILanguageFallback>
  18.           <UserLocale>en-US</UserLocale>
  19.         </component>
  20.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-PnpCustomizationsWinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  21.           <DriverPaths>
  22.             <PathAndCredentials wcm:keyValue="4b29ba63" wcm:action="add">
  23.               <Path>E:\amd64\2k19</Path>
  24.             </PathAndCredentials>
  25.             <PathAndCredentials wcm:keyValue="25fe51ea" wcm:action="add">
  26.               <Path>E:\NetKVM\2k19\amd64</Path>
  27.             </PathAndCredentials>
  28.           </DriverPaths>
  29.         </component>
  30.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  31.           <DiskConfiguration>
  32.             <Disk wcm:action="add">
  33.               <CreatePartitions>
  34.                 <CreatePartition wcm:action="add">
  35.                   <Order>1</Order>
  36.                   <Type>Primary</Type>
  37.                   <Size>100</Size>
  38.                 </CreatePartition>
  39.                 <CreatePartition wcm:action="add">
  40.                   <Extend>true</Extend>
  41.                   <Order>2</Order>
  42.                   <Type>Primary</Type>
  43.                 </CreatePartition>
  44.               </CreatePartitions>
  45.               <ModifyPartitions>
  46.                 <ModifyPartition wcm:action="add">
  47.                   <Format>NTFS</Format>
  48.                   <Label>System Reserved</Label>
  49.                   <Order>1</Order>
  50.                   <PartitionID>1</PartitionID>
  51.                   <TypeID>0x27</TypeID>
  52.                 </ModifyPartition>
  53.                 <ModifyPartition wcm:action="add">
  54.                   <Format>NTFS</Format>
  55.                   <Label>OS</Label>
  56.                   <Letter>C</Letter>
  57.                   <Order>2</Order>
  58.                   <PartitionID>2</PartitionID>
  59.                 </ModifyPartition>
  60.               </ModifyPartitions>
  61.               <DiskID>0</DiskID>
  62.               <WillWipeDisk>true</WillWipeDisk>
  63.             </Disk>
  64.           </DiskConfiguration>
  65.           <ImageInstall>
  66.             <OSImage>
  67.               <InstallFrom>
  68.                 <MetaData wcm:action="add">
  69.                   <Key>/Image/Description</Key>
  70.                   <Value>Windows 10 Pro</Value>
  71.                 </MetaData>
  72.               </InstallFrom>
  73.               <InstallTo>
  74.                 <DiskID>0</DiskID>
  75.                 <PartitionID>2</PartitionID>
  76.               </InstallTo>
  77.             </OSImage>
  78.           </ImageInstall>
  79.           <UserData>
  80.             <AcceptEula>true</AcceptEula>
  81.             <FullName/>
  82.             <Organization/>
  83.             <ProductKey>
  84.               <Key/>
  85.             </ProductKey>
  86.           </UserData>
  87.         </component>
  88.       </settings>
  89.       <settings pass="offlineServicing">
  90.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-LUA-Settings" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  91.           <EnableLUA>false</EnableLUA>
  92.         </component>
  93.       </settings>
  94.       <settings pass="specialize">
  95.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-International-Core" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  96.           <InputLocale>0409:00000409</InputLocale>
  97.           <SystemLocale>en-US</SystemLocale>
  98.           <UILanguage>en-US</UILanguage>
  99.           <UILanguageFallback>en-US</UILanguageFallback>
  100.           <UserLocale>en-US</UserLocale>
  101.         </component>
  102.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Security-SPP-UX" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  103.           <SkipAutoActivation>true</SkipAutoActivation>
  104.         </component>
  105.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-SQMApi" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  106.           <CEIPEnabled>0</CEIPEnabled>
  107.         </component>
  108.       </settings>
  109.       <settings pass="oobeSystem">
  110.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  111.           <OOBE>
  112.             <HideEULAPage>true</HideEULAPage>
  113.             <HideOEMRegistrationScreen>true</HideOEMRegistrationScreen>
  114.             <HideOnlineAccountScreens>true</HideOnlineAccountScreens>
  115.             <HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE>
  116.             <NetworkLocation>Work</NetworkLocation>
  117.             <SkipUserOOBE>true</SkipUserOOBE>
  118.             <SkipMachineOOBE>true</SkipMachineOOBE>
  119.             <ProtectYourPC>3</ProtectYourPC>
  120.           </OOBE>
  121.           <AutoLogon>
  122.             <Password>
  123.               <Value>123456</Value>
  124.               <PlainText>true</PlainText>
  125.             </Password>
  126.             <Enabled>true</Enabled>
  127.             <Username>Administrator</Username>
  128.           </AutoLogon>
  129.           <UserAccounts>
  130.             <AdministratorPassword>
  131.               <Value>123456</Value>
  132.               <PlainText>true</PlainText>
  133.             </AdministratorPassword>
  134.           </UserAccounts>
  135.           <RegisteredOrganization/>
  136.           <RegisteredOwner/>
  137.           <TimeZone>Eastern Standard Time</TimeZone>
  138.           <FirstLogonCommands>
  139.             <SynchronousCommand wcm:action="add">
  140.               <CommandLine>powershell -ExecutionPolicy Bypass -NoExit -NoProfile f:\post-install.ps1</CommandLine>
  141.               <RequiresUserInput>false</RequiresUserInput>
  142.               <Order>1</Order>
  143.               <Description>Post Installation Script</Description>
  144.             </SynchronousCommand>
  145.           </FirstLogonCommands>
  146.         </component>
  147.       </settings>
  148.     </unattend>
  151.   post-install.ps1: |-
  152.     # Remove AutoLogin
  153.     # https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unattend/microsoft-windows-shell-setup-autologon-logoncount#logoncount-known-issue
  154.     reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v AutoAdminLogon /t REG_SZ /d 0 /f
  156.     # install Qemu Tools (Drivers)
  157.     Start-Process msiexec -Wait -ArgumentList '/i e:\virtio-win-gt-x64.msi /qn /passive /norestart'
  159.     # install Guest Agent
  160.     Start-Process msiexec -Wait -ArgumentList '/i e:\guest-agent\qemu-ga-x86_64.msi /qn /passive /norestart'
  162.     # Rename cached unattend.xml to avoid it is picked up by sysprep
  163.     mv C:\Windows\Panther\unattend.xml C:\Windows\Panther\unattend.install.xml
  165.     # Eject CD, to avoid that the autounattend.xml on the CD is picked up by sysprep
  166.     (new-object -COM Shell.Application).NameSpace(17).ParseName('F:').InvokeVerb('Eject')
  168.     # Run Sysprep and Shutdown
  169.     C:\Windows\System32\Sysprep\sysprep.exe /generalize /oobe /shutdown /mode:vm
  171. ---
  173. apiVersion: kubevirt.io/v1
  174. kind: VirtualMachine
  175. metadata:
  176.   annotations:
  177.     name.os.template.kubevirt.io/win10: Microsoft Windows 10
  178.     vm.kubevirt.io/validations: |
  179.       [
  180.         {
  181.           "name": "minimal-required-memory",
  182.           "path": "jsonpath::.spec.domain.resources.requests.memory",
  183.           "rule": "integer",
  184.           "message": "This VM requires more memory.",
  185.           "min": 2147483648
  186.         }, {
  187.           "name": "windows-virtio-bus",
  188.           "path": "jsonpath::.spec.domain.devices.disks[*].disk.bus",
  189.           "valid": "jsonpath::.spec.domain.devices.disks[*].disk.bus",
  190.           "rule": "enum",
  191.           "message": "virto disk bus type has better performance, install virtio drivers in VM and change bus type",
  192.           "values": ["virtio"],
  193.           "justWarning": true
  194.         }, {
  195.           "name": "windows-disk-bus",
  196.           "path": "jsonpath::.spec.domain.devices.disks[*].disk.bus",
  197.           "valid": "jsonpath::.spec.domain.devices.disks[*].disk.bus",
  198.           "rule": "enum",
  199.           "message": "disk bus has to be either virtio or sata or scsi",
  200.           "values": ["virtio", "sata", "scsi"]
  201.         }, {
  202.           "name": "windows-cd-bus",
  203.           "path": "jsonpath::.spec.domain.devices.disks[*].cdrom.bus",
  204.           "valid": "jsonpath::.spec.domain.devices.disks[*].cdrom.bus",
  205.           "rule": "enum",
  206.           "message": "cd bus has to be sata",
  207.           "values": ["sata"]
  208.         }
  209.       ]
  210.   name: win10-template
  211.   namespace: default
  212.   labels:
  213.     app: win10-template
  214.     flavor.template.kubevirt.io/medium: 'true'
  215.     os.template.kubevirt.io/win10: 'true'
  216.     vm.kubevirt.io/template: windows10-desktop-medium
  217.     vm.kubevirt.io/template.namespace: openshift
  218.     vm.kubevirt.io/template.revision: '1'
  219.     vm.kubevirt.io/template.version: v0.14.0
  220.     workload.template.kubevirt.io/desktop: 'true'
  221. spec:
  222.   runStrategy: RerunOnFailure
  223.   dataVolumeTemplates:
  224.     - metadata:
  225.         name: win10-template-windows-iso
  226.       spec:
  227.         pvc:
  228.           accessModes:
  229.             - ReadWriteOnce
  230.           resources:
  231.             requests:
  232.               storage: 20Gi
  233.         source:
  234.           pvc:
  235.             name: windows10-iso
  236.             namespace: default
  237.     - metadata:
  238.         name: win10-template
  239.       spec:
  240.         pvc:
  241.           accessModes:
  242.             - ReadWriteOnce
  243.           resources:
  244.             requests:
  245.               storage: 25Gi
  246.           volumeMode: Filesystem
  247.         source:
  248.           blank: {}
  249.   template:
  250.     metadata:
  251.       annotations:
  252.         vm.kubevirt.io/flavor: medium
  253.         vm.kubevirt.io/os: windows10
  254.         vm.kubevirt.io/workload: desktop
  255.       labels:
  256.         flavor.template.kubevirt.io/medium: 'true'
  257.         kubevirt.io/domain: win10-template
  258.         kubevirt.io/size: medium
  259.         os.template.kubevirt.io/win10: 'true'
  260.         vm.kubevirt.io/name: win10-template
  261.         workload.template.kubevirt.io/desktop: 'true'
  262.     spec:
  263.       domain:
  264.         clock:
  265.           timer:
  266.             hpet:
  267.               present: false
  268.             hyperv: {}
  269.             pit:
  270.               tickPolicy: delay
  271.             rtc:
  272.               tickPolicy: catchup
  273.           utc: {}
  274.         cpu:
  275.           cores: 1
  276.           sockets: 1
  277.           threads: 1
  278.         devices:
  279.           disks:
  280.             - bootOrder: 1
  281.               disk:
  282.                 bus: virtio
  283.               name: win10-template
  284.             - bootOrder: 2
  285.               cdrom:
  286.                 bus: sata
  287.               name: windows-iso
  288.             - cdrom:
  289.                 bus: sata
  290.               name: windows-guest-tools
  291.             - name: sysprep
  292.               cdrom:
  293.                 bus: sata
  294.           inputs:
  295.             - bus: usb
  296.               name: tablet
  297.               type: tablet
  298.           interfaces:
  299.             - masquerade: {}
  300.               model: virtio
  301.               name: default
  302.         features:
  303.           acpi: {}
  304.           apic: {}
  305.           hyperv:
  306.             reenlightenment: {}
  307.             ipi: {}
  308.             synic: {}
  309.             synictimer:
  310.               direct: {}
  311.             spinlocks:
  312.               spinlocks: 8191
  313.             reset: {}
  314.             relaxed: {}
  315.             vpindex: {}
  316.             runtime: {}
  317.             tlbflush: {}
  318.             frequencies: {}
  319.             vapic: {}
  320.         machine:
  321.           type: pc-q35-rhel8.4.0
  322.         resources:
  323.           requests:
  324.             memory: 4Gi
  325.       hostname: win10-template
  326.       networks:
  327.         - name: default
  328.           pod: {}
  329.       volumes:
  330.         - dataVolume:
  331.             name: win10-iso
  332.           name: windows-iso
  333.         - dataVolume:
  334.             name: win10-template-windows-iso
  335.           name: win10-template
  336.         - containerDisk:
  337.             image: kubevirt/virtio-container-disk
  338.           name: windows-guest-tools
  339.         - name: sysprep
  340.           sysprep:
  341.             configMap:
  342.               name: win10-template-configmap

Launching a VM from template

From the above example after the sysprep command is executed in the post-install.ps1 and the vm is in shutdown state, A new VM can be launched from the base win10-template with additional changes mentioned from the below unattend.xml in sysprep-config. The new VM can take upto 5 minutes to be in running state since Windows goes through oobe setup in the background with the customizations specified in the below unattend.xml file.

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: sysprep-config
  5. data:
  6.   autounattend.xml: |-
  7.     <?xml version="1.0" encoding="utf-8"?>
  8.     <!-- responsible for installing windows, ignored on sysprepped images -->
  9.   unattend.xml: |-
  10.     <?xml version="1.0" encoding="utf-8"?>
  11.     <unattend xmlns="urn:schemas-microsoft-com:unattend">
  12.       <settings pass="oobeSystem">
  13.         <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
  14.           <OOBE>
  15.             <HideEULAPage>true</HideEULAPage>
  16.             <HideOEMRegistrationScreen>true</HideOEMRegistrationScreen>
  17.             <HideOnlineAccountScreens>true</HideOnlineAccountScreens>
  18.             <HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE>
  19.             <NetworkLocation>Work</NetworkLocation>
  20.             <SkipUserOOBE>true</SkipUserOOBE>
  21.             <SkipMachineOOBE>true</SkipMachineOOBE>
  22.             <ProtectYourPC>3</ProtectYourPC>
  23.           </OOBE>
  24.           <AutoLogon>
  25.             <Password>
  26.             <Value>123456</Value>
  27.               <PlainText>true</PlainText>
  28.             </Password>
  29.             <Enabled>true</Enabled>
  30.         <Username>Administrator</Username>
  31.     </AutoLogon>
  32.     <UserAccounts>
  33.          <AdministratorPassword>
  34.                 <Value>123456</Value>
  35.                 <PlainText>true</PlainText>
  36.         </AdministratorPassword>
  37.           </UserAccounts>
  38.           <RegisteredOrganization>Kuebvirt</RegisteredOrganization>
  39.           <RegisteredOwner>Kubevirt</RegisteredOwner>
  40.           <TimeZone>Eastern Standard Time</TimeZone>
  41.                 <FirstLogonCommands>
  42.                     <SynchronousCommand wcm:action="add">
  43.                     <CommandLine>powershell -ExecutionPolicy Bypass -NoExit -WindowStyle Hidden -NoProfile d:\customize.ps1</CommandLine>
  44.                         <RequiresUserInput>false</RequiresUserInput>
  45.                         <Order>1</Order>
  46.                 <Description>Customize Script</Description>
  47.             </SynchronousCommand>
  48.                 </FirstLogonCommands>
  49.         </component>
  50.       </settings>
  51.     </unattend>
  52.   customize.ps1: |-
  53.     # Enable RDP
  54.     Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0
  55.     Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
  58.     # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse
  59.     # Install the OpenSSH Server
  60.     Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
  61.     # Start the sshd service
  62.     Start-Service sshd
  64.     Set-Service -Name sshd -StartupType 'Automatic'
  66.     # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration
  67.     # use powershell as default shell for ssh
  68.     New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force
  71.     # Add ssh authorized_key for administrator
  72.     # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement
  73.     $MyDir = $MyInvocation.MyCommand.Path | Split-Path -Parent
  74.     $PublicKey = Get-Content -Path $MyDir\id_rsa.pub
  75.     $authrized_keys_path = $env:ProgramData + "\ssh\administrators_authorized_keys" 
  76.     Add-Content -Path $authrized_keys_path -Value $PublicKey
  77.     icacls.exe $authrized_keys_path /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"
  80.     # install application via exe file installer from url
  81.     function Install-Exe {
  82.       $dlurl = $args[0]
  83.       $installerPath = Join-Path $env:TEMP (Split-Path $dlurl -Leaf)
  84.       Invoke-WebRequest -UseBasicParsing $dlurl -OutFile $installerPath
  85.       Start-Process -FilePath $installerPath -Args "/S" -Verb RunAs -Wait
  86.       Remove-Item $installerPath
  88.     }
  90.     # Wait for networking before running a task at startup
  91.     do {
  92.       $ping = test-connection -comp kubevirt.io -count 1 -Quiet
  93.     } until ($ping)
  95.     # Installing the Latest Notepad++ with PowerShell
  96.     $BaseUri = "https://notepad-plus-plus.org"
  97.     $BasePage = Invoke-WebRequest -Uri $BaseUri -UseBasicParsing
  98.     $ChildPath = $BasePage.Links | Where-Object { $_.outerHTML -like '*Current Version*' } | Select-Object -ExpandProperty href
  99.     $DownloadPageUri = $BaseUri + $ChildPath
  100.     $DownloadPage = Invoke-WebRequest -Uri $DownloadPageUri -UseBasicParsing
  101.     $DownloadUrl = $DownloadPage.Links | Where-Object { $_.outerHTML -like '*npp.*.Installer.x64.exe"*' } | Select-Object -ExpandProperty href
  102.     Install-Exe $DownloadUrl
  103.   id_rsa.pub: |-
  104.     ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J fedora@localhost.localdomain