KubeVirt Scheduler

Scheduling is the process of matching Pods/VMs to Nodes. By default, the scheduler used is kube-scheduler. Further details can be found at Kubernetes Scheduler Documentation.

Custom schedulers can be used if the default scheduler does not satisfy your needs. For instance, you might want to schedule VMs using a load aware scheduler such as Trimaran Schedulers.

Creating a Custom Scheduler

KubeVirt is compatible with custom schedulers. The configuration steps are described in the Official Kubernetes Documentation. Please note, the Kubernetes version KubeVirt is running on and the Kubernetes version used to build the custom scheduler have to match. To get the Kubernetes version KubeVirt is running on, you can run the following command:

  1. $ kubectl version
  2. Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:"a43c0904d0de10f92aa3956c74489c45e6453d6e", GitTreeState:"clean", BuildDate:"2022-08-17T18:28:56Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
  3. Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:"a43c0904d0de10f92aa3956c74489c45e6453d6e", GitTreeState:"clean", BuildDate:"2022-08-17T18:23:45Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Pay attention to the Server line. In this case, the Kubernetes version is v1.22.13. You have to checkout the matching Kubernetes version and build the Kubernetes project:

  1. $ cd kubernetes
  2. $ git checkout v1.22.13
  3. $ make

Then, you can follow the configuration steps described here. Additionally, the ClusterRole system:kube-scheduler needs permissions to use the verbs watch, list and get on StorageClasses.

  1. - apiGroups:                                                                                                   
  2.   - storage.k8s.io                                                                                             
  3.   resources:                                                                                                   
  4.   - storageclasses                                                                                             
  5.   verbs:                                                                                                       
  6.   - watch                                                                                                      
  7.   - list                                                                                                       
  8.   - get

Scheduling VMs with the Custom Scheduler

The second scheduler should be up and running. You can check it with:

  1. $ kubectl get all -n kube-system

The deployment my-scheduler should be up and running if everything is setup properly. In order to launch the VM using the custom scheduler, you need to set the SchedulerName in the VM’s spec to my-scheduler. Here is an example VM definition:

  1. apiVersion: kubevirt.io/v1
  2. kind: VirtualMachine
  3. metadata:
  4.   name: vm-fedora
  5. spec:
  6.   running: true
  7.   template:
  8.     spec:
  9.       schedulerName: my-scheduler
  10.       domain:
  11.         devices:
  12.           disks:
  13.             - name: containerdisk
  14.               disk:
  15.                 bus: virtio
  16.             - name: cloudinitdisk
  17.               disk:
  18.                 bus: virtio
  19.           rng: {}
  20.         resources:
  21.           requests:
  22.             memory: 1Gi
  23.       terminationGracePeriodSeconds: 180
  24.       volumes:
  25.         - containerDisk:
  26.             image: quay.io/containerdisks/fedora:latest
  27.           name: containerdisk
  28.         - cloudInitNoCloud:
  29.             userData: |-
  30.               #cloud-config
  31.               chpasswd:
  32.                 expire: false
  33.               password: fedora
  34.               user: fedora
  35.           name: cloudinitdisk

In case the specified SchedulerName does not match any existing scheduler, the virt-launcher pod will stay in state Pending, until the specified scheduler can be found. You can check if the VM has been scheduled using the my-scheduler checking the virt-launcher pod events associated with the VM. The pod should have been scheduled with my-scheduler.

  1. $ kubectl get pods
  2. NAME                             READY   STATUS    RESTARTS   AGE
  3. virt-launcher-vm-fedora-dpc87    2/2     Running   0          24m
  5. $ kubectl describe pod virt-launcher-vm-fedora-dpc87
  6. [...] 
  7. Events:
  8.   Type    Reason     Age   From              Message
  9.   ----    ------     ----  ----              -------
  10.   Normal  Scheduled  21m   my-scheduler  Successfully assigned default/virt-launcher-vm-fedora-dpc87 to node01
  11. [...]