Container runtimes

You need to install a container runtime into each node in the cluster so that Pods can run there. This page outlines what is involved and describes related tasks for setting up nodes.

This page lists details for using several common container runtimes with Kubernetes, on Linux:

Note: For other operating systems, look for documentation specific to your platform.

Cgroup drivers

Control groups are used to constrain resources that are allocated to processes.

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager. Systemd has a tight integration with cgroups and allocates a cgroup per systemd unit. It’s possible to configure your container runtime and the kubelet to use cgroupfs. Using cgroupfs alongside systemd means that there will be two different cgroup managers.

A single cgroup manager simplifies the view of what resources are being allocated and will by default have a more consistent view of the available and in-use resources. When there are two cgroup managers on a system, you end up with two views of those resources. In the field, people have reported cases where nodes that are configured to use cgroupfs for the kubelet and Docker, but systemd for the rest of the processes, become unstable under resource pressure.

Changing the settings such that your container runtime and kubelet use systemd as the cgroup driver stabilized the system. To configure this for Docker, set native.cgroupdriver=systemd.

Caution:

Changing the cgroup driver of a Node that has joined a cluster is strongly not recommended.
If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors.

If you have automation that makes it feasible, replace the node with another using the updated configuration, or reinstall it using automation.

Container runtimes

Caution: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren’t responsible for these projects. This page follows CNCF website guidelines by listing projects alphabetically. To add a project to this list, read the content guide before submitting a change.

containerd

This section contains the necessary steps to use containerd as CRI runtime.

Use the following commands to install Containerd on your system:

Install and configure prerequisites:

  1. cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
  2. overlay
  3. br_netfilter
  4. EOF
  6. sudo modprobe overlay
  7. sudo modprobe br_netfilter
  9. # Setup required sysctl params, these persist across reboots.
  10. cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
  11. net.bridge.bridge-nf-call-iptables  = 1
  12. net.ipv4.ip_forward                 = 1
  13. net.bridge.bridge-nf-call-ip6tables = 1
  14. EOF
  16. # Apply sysctl params without reboot
  17. sudo sysctl --system

Install containerd:

  1. # (Install containerd)
  2. ## Set up the repository
  3. ### Install packages to allow apt to use a repository over HTTPS
  4. sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
  1. ## Add Docker's official GPG key
  2. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
  1. ## Add Docker apt repository.
  2. sudo add-apt-repository \
  3.     "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  4.     $(lsb_release -cs) \
  5.     stable"
  1. ## Install containerd
  2. sudo apt-get update && sudo apt-get install -y containerd.io
  1. # Configure containerd
  2. sudo mkdir -p /etc/containerd
  3. sudo containerd config default | sudo tee /etc/containerd/config.toml
  1. # Restart containerd
  2. sudo systemctl restart containerd
  1. # (Install containerd)
  2. ## Set up the repository
  3. ### Install required packages
  4. sudo yum install -y yum-utils device-mapper-persistent-data lvm2
  1. ## Add docker repository
  2. sudo yum-config-manager \
  3.     --add-repo \
  4.     https://download.docker.com/linux/centos/docker-ce.repo
  1. ## Install containerd
  2. sudo yum update -y && sudo yum install -y containerd.io
  1. ## Configure containerd
  2. sudo mkdir -p /etc/containerd
  3. sudo containerd config default > /etc/containerd/config.toml
  1. # Restart containerd
  2. sudo systemctl restart containerd
  1. # (Install containerd)
  2. # download containerd
  3. cmd /c curl -OL https://github.com/containerd/containerd/releases/download/v1.4.1/containerd-1.4.1-windows-amd64.tar.gz
  4. cmd /c tar xvf .\containerd-1.4.1-windows-amd64.tar.gz
  1. # extract and configure
  2. Copy-Item -Path ".\bin\" -Destination "$Env:ProgramFiles\containerd" -Recurse -Force
  3. cd $Env:ProgramFiles\containerd\
  4. .\containerd.exe config default | Out-File config.toml -Encoding ascii
  6. # review the configuration. depending on setup you may want to adjust:
  7. # - the sandbox_image (kubernetes pause image)
  8. # - cni bin_dir and conf_dir locations
  9. Get-Content config.toml
  1. # start containerd
  2. .\containerd.exe --register-service
  3. Start-Service containerd

systemd

To use the systemd cgroup driver in /etc/containerd/config.toml with runc, set

  1. [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  2.   ...
  3.   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  4.     SystemdCgroup = true

When using kubeadm, manually configure the cgroup driver for kubelet.

CRI-O

This section contains the necessary steps to install CRI-O as a container runtime.

Use the following commands to install CRI-O on your system:

Note: The CRI-O major and minor versions must match the Kubernetes major and minor versions. For more information, see the CRI-O compatibility matrix.

Install and configure prerequisites:

  1. sudo modprobe overlay
  2. sudo modprobe br_netfilter
  4. # Set up required sysctl params, these persist across reboots.
  5. cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
  6. net.bridge.bridge-nf-call-iptables  = 1
  7. net.ipv4.ip_forward                 = 1
  8. net.bridge.bridge-nf-call-ip6tables = 1
  9. EOF
  11. sudo sysctl --system

To install CRI-O on the following operating systems, set the environment variable OS to the appropriate value from the following table:

Operating system$OS
Debian UnstableDebian_Unstable
Debian TestingDebian_Testing

Then, set $VERSION to the CRI-O version that matches your Kubernetes version. For instance, if you want to install CRI-O 1.18, set VERSION=1.18. You can pin your installation to a specific release. To install version 1.18.3, set VERSION=1.18:1.18.3.

Then run

  1. cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
  2. deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
  3. EOF
  4. cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
  5. deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
  6. EOF
  8. curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
  9. curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
  11. sudo apt-get update
  12. sudo apt-get install cri-o cri-o-runc

To install on the following operating systems, set the environment variable OS to the appropriate field in the following table:

Operating system$OS
Ubuntu 20.04xUbuntu_20.04
Ubuntu 19.10xUbuntu_19.10
Ubuntu 19.04xUbuntu_19.04
Ubuntu 18.04xUbuntu_18.04

Then, set $VERSION to the CRI-O version that matches your Kubernetes version. For instance, if you want to install CRI-O 1.18, set VERSION=1.18. You can pin your installation to a specific release. To install version 1.18.3, set VERSION=1.18:1.18.3.

Then run

  1. cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
  2. deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
  3. EOF
  4. cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
  5. deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
  6. EOF
  8. curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
  9. curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -
  11. sudo apt-get update
  12. sudo apt-get install cri-o cri-o-runc

To install on the following operating systems, set the environment variable OS to the appropriate field in the following table:

Operating system$OS
Centos 8CentOS_8
Centos 8 StreamCentOS_8_Stream
Centos 7CentOS_7

Then, set $VERSION to the CRI-O version that matches your Kubernetes version. For instance, if you want to install CRI-O 1.18, set VERSION=1.18. You can pin your installation to a specific release. To install version 1.18.3, set VERSION=1.18:1.18.3.

Then run

  1. sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
  2. sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
  3. sudo yum install cri-o
  1. sudo zypper install cri-o

Set $VERSION to the CRI-O version that matches your Kubernetes version. For instance, if you want to install CRI-O 1.18, VERSION=1.18.

You can find available versions with:

  1. sudo dnf module list cri-o

CRI-O does not support pinning to specific releases on Fedora.

Then run

  1. sudo dnf module enable cri-o:$VERSION
  2. sudo dnf install cri-o

Start CRI-O:

  1. sudo systemctl daemon-reload
  2. sudo systemctl start crio

Refer to the CRI-O installation guide for more information.

Docker

On each of your nodes, install Docker CE.

The Kubernetes release notes list which versions of Docker are compatible with that version of Kubernetes.

Use the following commands to install Docker on your system:

  1. # (Install Docker CE)
  2. ## Set up the repository:
  3. ### Install packages to allow apt to use a repository over HTTPS
  4. sudo apt-get update && sudo apt-get install -y \
  5.   apt-transport-https ca-certificates curl software-properties-common gnupg2
  1. # Add Docker's official GPG key:
  2. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
  1. # Add the Docker apt repository:
  2. sudo add-apt-repository \
  3.   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  4.   $(lsb_release -cs) \
  5.   stable"
  1. # Install Docker CE
  2. sudo apt-get update && sudo apt-get install -y \
  3.   containerd.io=1.2.13-2 \
  4.   docker-ce=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) \
  5.   docker-ce-cli=5:19.03.11~3-0~ubuntu-$(lsb_release -cs)
  1. # Set up the Docker daemon
  2. cat <<EOF | sudo tee /etc/docker/daemon.json
  3. {
  4.   "exec-opts": ["native.cgroupdriver=systemd"],
  5.   "log-driver": "json-file",
  6.   "log-opts": {
  7.     "max-size": "100m"
  8.   },
  9.   "storage-driver": "overlay2"
  10. }
  11. EOF
  1. # Create /etc/systemd/system/docker.service.d
  2. sudo mkdir -p /etc/systemd/system/docker.service.d
  1. # Restart Docker
  2. sudo systemctl daemon-reload
  3. sudo systemctl restart docker
  1. # (Install Docker CE)
  2. ## Set up the repository
  3. ### Install required packages
  4. sudo yum install -y yum-utils device-mapper-persistent-data lvm2
  1. ## Add the Docker repository
  2. sudo yum-config-manager --add-repo \
  3.   https://download.docker.com/linux/centos/docker-ce.repo
  1. # Install Docker CE
  2. sudo yum update -y && sudo yum install -y \
  3.   containerd.io-1.2.13 \
  4.   docker-ce-19.03.11 \
  5.   docker-ce-cli-19.03.11
  1. ## Create /etc/docker
  2. sudo mkdir /etc/docker
  1. # Set up the Docker daemon
  2. cat <<EOF | sudo tee /etc/docker/daemon.json
  3. {
  4.   "exec-opts": ["native.cgroupdriver=systemd"],
  5.   "log-driver": "json-file",
  6.   "log-opts": {
  7.     "max-size": "100m"
  8.   },
  9.   "storage-driver": "overlay2",
  10.   "storage-opts": [
  11.     "overlay2.override_kernel_check=true"
  12.   ]
  13. }
  14. EOF
  1. # Create /etc/systemd/system/docker.service.d
  2. sudo mkdir -p /etc/systemd/system/docker.service.d
  1. # Restart Docker
  2. sudo systemctl daemon-reload
  3. sudo systemctl restart docker

If you want the docker service to start on boot, run the following command:

  1. sudo systemctl enable docker

Refer to the official Docker installation guides for more information.