Placing pods on specific nodes using node selectors
A node selector specifies a map of key/value pairs that are defined using custom labels on nodes and selectors specified in pods.
For the pod to be eligible to run on a node, the pod must have the same key/value node selector as the label on the node.
About node selectors
You can use node selectors on pods and labels on nodes to control where the pod is scheduled. With node selectors, OKD schedules the pods on nodes that contain matching labels.
You can use a node selector to place specific pods on specific nodes, cluster-wide node selectors to place new pods on specific nodes anywhere in the cluster, and project node selectors to place new pods in a project on specific nodes.
For example, as a cluster administrator, you can create an infrastructure where application developers can deploy pods only onto the nodes closest to their geographical location by including a node selector in every pod they create. In this example, the cluster consists of five data centers spread across two regions. In the U.S., label the nodes as us-east
, us-central
, or us-west
. In the Asia-Pacific region (APAC), label the nodes as apac-east
or apac-west
. The developers can add a node selector to the pods they create to ensure the pods get scheduled on those nodes.
A pod is not scheduled if the Pod
object contains a node selector, but no node has a matching label.
If you are using node selectors and node affinity in the same pod configuration, the following rules control pod placement onto nodes:
|
Node selectors on specific pods and nodes
You can control which node a specific pod is scheduled on by using node selectors and labels.
To use node selectors and labels, first label the node to avoid pods being descheduled, then add the node selector to the pod.
You cannot add a node selector directly to an existing scheduled pod. You must label the object that controls the pod, such as deployment config. |
For example, the following Node
object has the region: east
label:
Sample Node
object with a label
kind: Node
apiVersion: v1
metadata:
name: ip-10-0-131-14.ec2.internal
selfLink: /api/v1/nodes/ip-10-0-131-14.ec2.internal
uid: 7bc2580a-8b8e-11e9-8e01-021ab4174c74
resourceVersion: '478704'
creationTimestamp: '2019-06-10T14:46:08Z'
labels:
kubernetes.io/os: linux
failure-domain.beta.kubernetes.io/zone: us-east-1a
node.openshift.io/os_version: '4.5'
node-role.kubernetes.io/worker: ''
failure-domain.beta.kubernetes.io/region: us-east-1
node.openshift.io/os_id: fedora
beta.kubernetes.io/instance-type: m4.large
kubernetes.io/hostname: ip-10-0-131-14
beta.kubernetes.io/arch: amd64
region: east (1)
1 | Label to match the pod node selector. |
A pod has the type: user-node,region: east
node selector:
Sample Pod
object with node selectors
apiVersion: v1
kind: Pod
....
spec:
nodeSelector: (1)
region: east
type: user-node
1 | Node selectors to match the node label. |
When you create the pod using the example pod spec, it can be scheduled on the example node.
Default cluster-wide node selectors
With default cluster-wide node selectors, when you create a pod in that cluster, OKD adds the default node selectors to the pod and schedules the pod on nodes with matching labels.
For example, the following Scheduler
object has the default cluster-wide region=east
and type=user-node
node selectors:
Example Scheduler Operator Custom Resource
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
name: cluster
...
spec:
defaultNodeSelector: type=user-node,region=east
...
A node in that cluster has the type=user-node,region=east
labels:
Example Node
object
apiVersion: v1
kind: Node
metadata:
name: ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4
...
labels:
region: east
type: user-node
...
Example Pod
object with a node selector
apiVersion: v1
kind: Pod
...
spec:
nodeSelector:
region: east
...
When you create the pod using the example pod spec in the example cluster, the pod is created with the cluster-wide node selector and is scheduled on the labeled node:
Example pod list with the pod on the labeled node
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-s1 1/1 Running 0 20s 10.131.2.6 ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4 <none> <none>
If the project where you create the pod has a project node selector, that selector takes preference over a cluster-wide node selector. Your pod is not created or scheduled if the pod does not have the project node selector. |
Project node selectors
With project node selectors, when you create a pod in this project, OKD adds the node selectors to the pod and schedules the pods on a node with matching labels. If there is a cluster-wide default node selector, a project node selector takes preference.
For example, the following project has the region=east
node selector:
Example Namespace
object
apiVersion: v1
kind: Namespace
metadata:
name: east-region
annotations:
openshift.io/node-selector: "region=east"
...
The following node has the type=user-node,region=east
labels:
Example Node
object
apiVersion: v1
kind: Node
metadata:
name: ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4
...
labels:
region: east
type: user-node
...
When you create the pod using the example pod spec in this example project, the pod is created with the project node selectors and is scheduled on the labeled node:
Example Pod
object
apiVersion: v1
kind: Pod
metadata:
namespace: east-region
...
spec:
nodeSelector:
region: east
type: user-node
...
Example pod list with the pod on the labeled node
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-s1 1/1 Running 0 20s 10.131.2.6 ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4 <none> <none>
A pod in the project is not created or scheduled if the pod contains different node selectors. For example, if you deploy the following pod into the example project, it is not be created:
Example Pod
object with an invalid node selector
apiVersion: v1
kind: Pod
...
spec:
nodeSelector:
region: west
....
Using node selectors to control pod placement
You can use node selectors on pods and labels on nodes to control where the pod is scheduled. With node selectors, OKD schedules the pods on nodes that contain matching labels.
You add labels to a node, a compute machine set, or a machine config. Adding the label to the compute machine set ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or machine config do not persist if the node or machine goes down.
To add node selectors to an existing pod, add a node selector to the controlling object for that pod, such as a ReplicaSet
object, DaemonSet
object, StatefulSet
object, Deployment
object, or DeploymentConfig
object. Any existing pods under that controlling object are recreated on a node with a matching label. If you are creating a new pod, you can add the node selector directly to the Pod
spec.
You cannot add a node selector directly to an existing scheduled pod. |
Prerequisites
To add a node selector to existing pods, determine the controlling object for that pod. For example, the router-default-66d5cf9464-m2g75
pod is controlled by the router-default-66d5cf9464
replica set:
$ oc describe pod router-default-66d5cf9464-7pwkc
Name: router-default-66d5cf9464-7pwkc
Namespace: openshift-ingress
....
Controlled By: ReplicaSet/router-default-66d5cf9464
The web console lists the controlling object under ownerReferences
in the pod YAML:
ownerReferences:
- apiVersion: apps/v1
kind: ReplicaSet
name: router-default-66d5cf9464
uid: d81dd094-da26-11e9-a48a-128e7edf0312
controller: true
blockOwnerDeletion: true
Procedure
Add labels to a node by using a compute machine set or editing the node directly:
Use a
MachineSet
object to add labels to nodes managed by the compute machine set when a node is created:Run the following command to add labels to a
MachineSet
object:$ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]' -n openshift-machine-api
For example:
$ oc patch MachineSet abc612-msrtw-worker-us-east-1c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]' -n openshift-machine-api
You can alternatively apply the following YAML to add labels to a compute machine set:
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: <machineset>
namespace: openshift-machine-api
spec:
template:
spec:
metadata:
labels:
region: “east”
type: “user-node”
Verify that the labels are added to the
MachineSet
object by using theoc edit
command:For example:
$ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api
Example
MachineSet
objectapiVersion: machine.openshift.io/v1beta1
kind: MachineSet
....
spec:
...
template:
metadata:
...
spec:
metadata:
labels:
region: east
type: user-node
....
Add labels directly to a node:
Edit the
Node
object for the node:$ oc label nodes <name> <key>=<value>
For example, to label a node:
$ oc label nodes ip-10-0-142-25.ec2.internal type=user-node region=east
You can alternatively apply the following YAML to add labels to a node:
kind: Node
apiVersion: v1
metadata:
name: <node_name>
labels:
type: “user-node”
region: “east”
Verify that the labels are added to the node:
$ oc get nodes -l type=user-node,region=east
Example output
NAME STATUS ROLES AGE VERSION
ip-10-0-142-25.ec2.internal Ready worker 17m v1.25.0
Add the matching node selector to a pod:
To add a node selector to existing and future pods, add a node selector to the controlling object for the pods:
Example
ReplicaSet
object with labelskind: ReplicaSet
....
spec:
....
template:
metadata:
creationTimestamp: null
labels:
ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
pod-template-hash: 66d5cf9464
spec:
nodeSelector:
kubernetes.io/os: linux
node-role.kubernetes.io/worker: ''
type: user-node (1)
1 Add the node selector. To add a node selector to a specific, new pod, add the selector to the
Pod
object directly:Example
Pod
object with a node selectorapiVersion: v1
kind: Pod
....
spec:
nodeSelector:
region: east
type: user-node
You cannot add a node selector directly to an existing scheduled pod.
Creating default cluster-wide node selectors
You can use default cluster-wide node selectors on pods together with labels on nodes to constrain all pods created in a cluster to specific nodes.
With cluster-wide node selectors, when you create a pod in that cluster, OKD adds the default node selectors to the pod and schedules the pod on nodes with matching labels.
You configure cluster-wide node selectors by editing the Scheduler Operator custom resource (CR). You add labels to a node, a compute machine set, or a machine config. Adding the label to the compute machine set ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or machine config do not persist if the node or machine goes down.
You can add additional key/value pairs to a pod. But you cannot add a different value for a default key. |
Procedure
To add a default cluster-wide node selector:
Edit the Scheduler Operator CR to add the default cluster-wide node selectors:
$ oc edit scheduler cluster
Example Scheduler Operator CR with a node selector
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
name: cluster
...
spec:
defaultNodeSelector: type=user-node,region=east (1)
mastersSchedulable: false
1 Add a node selector with the appropriate <key>:<value>
pairs.After making this change, wait for the pods in the
openshift-kube-apiserver
project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.Add labels to a node by using a compute machine set or editing the node directly:
Use a compute machine set to add labels to nodes managed by the compute machine set when a node is created:
Run the following command to add labels to a
MachineSet
object:$ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]' -n openshift-machine-api (1)
1 Add a <key>/<value>
pair for each label.For example:
$ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]' -n openshift-machine-api
You can alternatively apply the following YAML to add labels to a compute machine set:
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: <machineset>
namespace: openshift-machine-api
spec:
template:
spec:
metadata:
labels:
region: “east”
type: “user-node”
Verify that the labels are added to the
MachineSet
object by using theoc edit
command:For example:
$ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api
Example
MachineSet
objectapiVersion: machine.openshift.io/v1beta1
kind: MachineSet
...
spec:
...
template:
metadata:
...
spec:
metadata:
labels:
region: east
type: user-node
...
Redeploy the nodes associated with that compute machine set by scaling down to
0
and scaling up the nodes:For example:
$ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
$ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
When the nodes are ready and available, verify that the label is added to the nodes by using the
oc get
command:$ oc get nodes -l <key>=<value>
For example:
$ oc get nodes -l type=user-node
Example output
NAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp Ready worker 61s v1.25.0
Add labels directly to a node:
Edit the
Node
object for the node:$ oc label nodes <name> <key>=<value>
For example, to label a node:
$ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east
You can alternatively apply the following YAML to add labels to a node:
kind: Node
apiVersion: v1
metadata:
name: <node_name>
labels:
type: “user-node”
region: “east”
Verify that the labels are added to the node using the
oc get
command:$ oc get nodes -l <key>=<value>,<key>=<value>
For example:
$ oc get nodes -l type=user-node,region=east
Example output
NAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 Ready worker 17m v1.25.0
Creating project-wide node selectors
You can use node selectors in a project together with labels on nodes to constrain all pods created in that project to the labeled nodes.
When you create a pod in this project, OKD adds the node selectors to the pods in the project and schedules the pods on a node with matching labels in the project. If there is a cluster-wide default node selector, a project node selector takes preference.
You add node selectors to a project by editing the Namespace
object to add the openshift.io/node-selector
parameter. You add labels to a node, a compute machine set, or a machine config. Adding the label to the compute machine set ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or machine config do not persist if the node or machine goes down.
A pod is not scheduled if the Pod
object contains a node selector, but no project has a matching node selector. When you create a pod from that spec, you receive an error similar to the following message:
Example error message
Error from server (Forbidden): error when creating "pod.yaml": pods "pod-4" is forbidden: pod node label selector conflicts with its project node label selector
You can add additional key/value pairs to a pod. But you cannot add a different value for a project key. |
Procedure
To add a default project node selector:
Create a namespace or edit an existing namespace to add the
openshift.io/node-selector
parameter:$ oc edit namespace <name>
Example output
apiVersion: v1
kind: Namespace
metadata:
annotations:
openshift.io/node-selector: "type=user-node,region=east" (1)
openshift.io/description: ""
openshift.io/display-name: ""
openshift.io/requester: kube:admin
openshift.io/sa.scc.mcs: s0:c30,c5
openshift.io/sa.scc.supplemental-groups: 1000880000/10000
openshift.io/sa.scc.uid-range: 1000880000/10000
creationTimestamp: "2021-05-10T12:35:04Z"
labels:
kubernetes.io/metadata.name: demo
name: demo
resourceVersion: "145537"
uid: 3f8786e3-1fcb-42e3-a0e3-e2ac54d15001
spec:
finalizers:
- kubernetes
1 Add the openshift.io/node-selector
with the appropriate<key>:<value>
pairs.Add labels to a node by using a compute machine set or editing the node directly:
Use a
MachineSet
object to add labels to nodes managed by the compute machine set when a node is created:Run the following command to add labels to a
MachineSet
object:$ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]' -n openshift-machine-api
For example:
$ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]' -n openshift-machine-api
You can alternatively apply the following YAML to add labels to a compute machine set:
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: <machineset>
namespace: openshift-machine-api
spec:
template:
spec:
metadata:
labels:
region: “east”
type: “user-node”
Verify that the labels are added to the
MachineSet
object by using theoc edit
command:For example:
$ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
Example output
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
...
spec:
...
template:
metadata:
...
spec:
metadata:
labels:
region: east
type: user-node
Redeploy the nodes associated with that compute machine set:
For example:
$ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
$ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
When the nodes are ready and available, verify that the label is added to the nodes by using the
oc get
command:$ oc get nodes -l <key>=<value>
For example:
$ oc get nodes -l type=user-node,region=east
Example output
NAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp Ready worker 61s v1.25.0
Add labels directly to a node:
Edit the
Node
object to add labels:$ oc label <resource> <name> <key>=<value>
For example, to label a node:
$ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-c-tgq49 type=user-node region=east
You can alternatively apply the following YAML to add labels to a node:
kind: Node
apiVersion: v1
metadata:
name: <node_name>
labels:
type: “user-node”
region: “east”
Verify that the labels are added to the
Node
object using theoc get
command:$ oc get nodes -l <key>=<value>
For example:
$ oc get nodes -l type=user-node,region=east
Example output
NAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 Ready worker 17m v1.25.0
Additional resources