Specifying nodes for OKD Virtualization components

The default scheduling for virtual machines (VMs) on bare metal nodes is appropriate. Optionally, you can specify the nodes where you want to deploy OKD Virtualization Operators, workloads, and controllers by configuring node placement rules.

You can configure node placement rules for some components after installing OKD Virtualization, but virtual machines cannot be present if you want to configure node placement rules for workloads.

About node placement rules for OKD Virtualization components

You can use node placement rules for the following tasks:

  • Deploy virtual machines only on nodes intended for virtualization workloads.

  • Deploy Operators only on infrastructure nodes.

  • Maintain separation between workloads.

Depending on the object, you can use one or more of the following rule types:

nodeSelector

Allows pods to be scheduled on nodes that are labeled with the key-value pair or pairs that you specify in this field. The node must have labels that exactly match all listed pairs.

affinity

Enables you to use more expressive syntax to set rules that match nodes with pods. Affinity also allows for more nuance in how the rules are applied. For example, you can specify that a rule is a preference, not a requirement. If a rule is a preference, pods are still scheduled when the rule is not satisfied.

tolerations

Allows pods to be scheduled on nodes that have matching taints. If a taint is applied to a node, that node only accepts pods that tolerate the taint.

Applying node placement rules

You can apply node placement rules by editing a Subscription, HyperConverged, or HostPathProvisioner object using the command line.

Prerequisites

  • The oc CLI tool is installed.

  • You are logged in with cluster administrator permissions.

Procedure

  1. Edit the object in your default editor by running the following command:

    1. $ oc edit <resource_type> <resource_name> -n {CNVNamespace}

  2. Save the file to apply the changes.

Node placement rule examples

You can specify node placement rules for a OKD Virtualization component by editing a Subscription, HyperConverged, or HostPathProvisioner object.

Subscription object node placement rule examples

To specify the nodes where OLM deploys the OKD Virtualization Operators, edit the Subscription object during OKD Virtualization installation.

Currently, you cannot configure node placement rules for the Subscription object by using the web console.

The Subscription object does not support the affinity node pplacement rule.

Example Subscription object with nodeSelector rule

  1. apiVersion: operators.coreos.com/v1alpha1
  2. kind: Subscription
  3. metadata:
  4.   name: hco-operatorhub
  5.   namespace: kubevirt-hyperconverged
  6. spec:
  7.   source: community-operators
  8.   sourceNamespace: openshift-marketplace
  9.   name: community-kubevirt-hyperconverged
  10.   startingCSV: kubevirt-hyperconverged-operator.v4.14.0
  11.   channel: "stable"
  12.   config:
  13.     nodeSelector:
  14.       example.io/example-infra-key: example-infra-value (1)
1OLM deploys the OKD Virtualization Operators on nodes labeled example.io/example-infra-key = example-infra-value.

Example Subscription object with tolerations rule

  1. apiVersion: operators.coreos.com/v1alpha1
  2. kind: Subscription
  3. metadata:
  4.   name: hco-operatorhub
  5.   namespace: kubevirt-hyperconverged
  6. spec:
  7.   source:  community-operators
  8.   sourceNamespace: openshift-marketplace
  9.   name: community-kubevirt-hyperconverged
  10.   startingCSV: kubevirt-hyperconverged-operator.v4.14.0
  11.   channel: "stable"
  12.   config:
  13.     tolerations:
  14.     - key: "key"
  15.       operator: "Equal"
  16.       value: "virtualization" (1)
  17.       effect: "NoSchedule"
1OLM deploys OKD Virtualization Operators on nodes labeled key = virtualization:NoSchedule taint. Only pods with the matching tolerations are scheduled on these nodes.

HyperConverged object node placement rule example

To specify the nodes where OKD Virtualization deploys its components, you can edit the nodePlacement object in the HyperConverged custom resource (CR) file that you create during OKD Virtualization installation.

Example HyperConverged object with nodeSelector rule

  1. apiVersion: hco.kubevirt.io/v1beta1
  2. kind: HyperConverged
  3. metadata:
  4.   name: kubevirt-hyperconverged
  5.   namespace: kubevirt-hyperconverged
  6. spec:
  7.   infra:
  8.     nodePlacement:
  9.       nodeSelector:
  10.         example.io/example-infra-key: example-infra-value (1)
  11.   workloads:
  12.     nodePlacement:
  13.       nodeSelector:
  14.         example.io/example-workloads-key: example-workloads-value (2)
1Infrastructure resources are placed on nodes labeled example.io/example-infra-key = example-infra-value.
2workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.

Example HyperConverged object with affinity rule

  1. apiVersion: hco.kubevirt.io/v1beta1
  2. kind: HyperConverged
  3. metadata:
  4.   name: kubevirt-hyperconverged
  5.   namespace: kubevirt-hyperconverged
  6. spec:
  7.   infra:
  8.     nodePlacement:
  9.       affinity:
  10.         nodeAffinity:
  11.           requiredDuringSchedulingIgnoredDuringExecution:
  12.             nodeSelectorTerms:
  13.             - matchExpressions:
  14.               - key: example.io/example-infra-key
  15.                 operator: In
  16.                 values:
  17.                 - example-infra-value (1)
  18.   workloads:
  19.     nodePlacement:
  20.       affinity:
  21.         nodeAffinity:
  22.           requiredDuringSchedulingIgnoredDuringExecution:
  23.             nodeSelectorTerms:
  24.             - matchExpressions:
  25.               - key: example.io/example-workloads-key (2)
  26.                 operator: In
  27.                 values:
  28.                 - example-workloads-value
  29.           preferredDuringSchedulingIgnoredDuringExecution:
  30.           - weight: 1
  31.             preference:
  32.               matchExpressions:
  33.               - key: example.io/num-cpus
  34.                 operator: Gt
  35.                 values:
  36.                 - 8 (3)
1Infrastructure resources are placed on nodes labeled example.io/example-infra-key = example-value.
2workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.
3Nodes that have more than eight CPUs are preferred for workloads, but if they are not available, pods are still scheduled.

Example HyperConverged object with tolerations rule

  1. apiVersion: hco.kubevirt.io/v1beta1
  2. kind: HyperConverged
  3. metadata:
  4.   name: kubevirt-hyperconverged
  5.   namespace: kubevirt-hyperconverged
  6. spec:
  7.   workloads:
  8.     nodePlacement:
  9.       tolerations: (1)
  10.       - key: "key"
  11.         operator: "Equal"
  12.         value: "virtualization"
  13.         effect: "NoSchedule"
1Nodes reserved for OKD Virtualization components are labeled with the key = virtualization:NoSchedule taint. Only pods with matching tolerations are scheduled on reserved nodes.

HostPathProvisioner object node placement rule example

You can edit the HostPathProvisioner object directly or by using the web console.

You must schedule the hostpath provisioner and the OKD Virtualization components on the same nodes. Otherwise, virtualization pods that use the hostpath provisioner cannot run. You cannot run virtual machines.

After you deploy a virtual machine (VM) with the hostpath provisioner (HPP) storage class, you can remove the hostpath provisioner pod from the same node by using the node selector. However, you must first revert that change, at least for that specific node, and wait for the pod to run before trying to delete the VM.

You can configure node placement rules by specifying nodeSelector, affinity, or tolerations for the spec.workload field of the HostPathProvisioner object that you create when you install the hostpath provisioner.

Example HostPathProvisioner object with nodeSelector rule

  1. apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
  2. kind: HostPathProvisioner
  3. metadata:
  4.   name: hostpath-provisioner
  5. spec:
  6.   imagePullPolicy: IfNotPresent
  7.   pathConfig:
  8.     path: "</path/to/backing/directory>"
  9.     useNamingPrefix: false
  10.   workload:
  11.     nodeSelector:
  12.       example.io/example-workloads-key: example-workloads-value (1)
1Workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.

Additional resources