Schedule GPUs

Configure and schedule GPUs for use as a resource by nodes in a cluster.

FEATURE STATE: Kubernetes v1.10 [beta]

Kubernetes includes experimental support for managing GPUs (graphical processing units) across several nodes.

This page describes how users can consume GPUs, and outlines some of the limitations in the implementation.

Using device plugins

Kubernetes implements device plugins to let Pods access specialized hardware features such as GPUs.

Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren’t responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.

As an administrator, you have to install GPU drivers from the corresponding hardware vendor on the nodes and run the corresponding device plugin from the GPU vendor. Here are some links to vendors’ instructions:

Once you have installed the plugin, your cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu.

You can consume these GPUs from your containers by requesting the custom GPU resource, the same way you request cpu or memory. However, there are some limitations in how you specify the resource requirements for custom devices.

GPUs are only supposed to be specified in the limits section, which means:

  • You can specify GPU limits without specifying requests, because Kubernetes will use the limit as the request value by default.
  • You can specify GPU in both limits and requests but these two values must be equal.
  • You cannot specify GPU requests without specifying limits.

Here’s an example manifest for a Pod that requests a GPU:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: example-vector-add
  5. spec:
  6.   restartPolicy: OnFailure
  7.   containers:
  8.     - name: example-vector-add
  9.       image: "registry.example/example-vector-add:v42"
  10.       resources:
  11.         limits:
  12.           gpu-vendor.example/example-gpu: 1 # requesting 1 GPU

Clusters containing different types of GPUs

If different nodes in your cluster have different types of GPUs, then you can use Node Labels and Node Selectors to schedule pods to appropriate nodes.

For example:

  1. # Label your nodes with the accelerator type they have.
  2. kubectl label nodes node1 accelerator=example-gpu-x100
  3. kubectl label nodes node2 accelerator=other-gpu-k915

That label key accelerator is just an example; you can use a different label key if you prefer.

Automatic node labelling

If you’re using AMD GPU devices, you can deploy Node Labeller. Node Labeller is a controller that automatically labels your nodes with GPU device properties.

At the moment, that controller can add labels for:

  • Device ID (-device-id)
  • VRAM Size (-vram)
  • Number of SIMD (-simd-count)
  • Number of Compute Unit (-cu-count)
  • Firmware and Feature Versions (-firmware)
  • GPU Family, in two letters acronym (-family)
    • SI - Southern Islands
    • CI - Sea Islands
    • KV - Kaveri
    • VI - Volcanic Islands
    • CZ - Carrizo
    • AI - Arctic Islands
    • RV - Raven
  1. kubectl describe node cluster-node-23
  1. Name:               cluster-node-23
  2. Roles:              <none>
  3. Labels:             beta.amd.com/gpu.cu-count.64=1
  4.                     beta.amd.com/gpu.device-id.6860=1
  5.                     beta.amd.com/gpu.family.AI=1
  6.                     beta.amd.com/gpu.simd-count.256=1
  7.                     beta.amd.com/gpu.vram.16G=1
  8.                     kubernetes.io/arch=amd64
  9.                     kubernetes.io/os=linux
  10.                     kubernetes.io/hostname=cluster-node-23
  11. Annotations:        node.alpha.kubernetes.io/ttl: 0
  12. …

With the Node Labeller in use, you can specify the GPU type in the Pod spec:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: cuda-vector-add
  5. spec:
  6.   restartPolicy: OnFailure
  7.   containers:
  8.     - name: cuda-vector-add
  9.       # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
  10.       image: "registry.k8s.io/cuda-vector-add:v0.1"
  11.       resources:
  12.         limits:
  13.           nvidia.com/gpu: 1
  14.   affinity:
  15.     nodeAffinity:
  16.       requiredDuringSchedulingIgnoredDuringExecution:
  17.         nodeSelectorTerms:
  18.         – matchExpressions:
  19.           – key: beta.amd.com/gpu.family.AI # Arctic Islands GPU family
  20.             operator: Exist

This ensures that the Pod will be scheduled to a node that has the GPU type you specified.