Container Runtime

You need to install a container runtime into each edge node in the cluster so that the edge engine EdgeCore can be installed successfully and edge Pods can run there.

This page provides an outline of how to use several common container runtimes with KubeEdge.

containerd

Install and configure containerd

To begin, you will need to install containerd. Please refer to the containerd installation guide for instructions.

Container Runtime - 图1tip

If you use KubeEdge v1.15.0 or later, please install containerd v1.6.0 or a higher version.

If there are no containerd configuration files in the /etc/containerd/ directory, you can generate the configuration files and restart containerd by executing the following command.

  1. containerd config default > /etc/containerd/config.toml
  2. systemctl restart containerd

Container Runtime - 图2tip

If you installed containerd from a package (for example, RPM or .deb), you may find that the CRI integration plugin is disabled by default.

You need CRI support enabled to use containerd with Kubernetes. Make sure that cri is not included in the disabled_plugins list within /etc/containerd/config.toml; if you made changes to that file, also restart containerd.

If you need to update the sandbox(pause) image, you can modify the following settings in the containerd configuration file:

  1. [plugins."io.containerd.grpc.v1.cri"]
  2.   sandbox_image = "kubeedge/pause:3.6"

You can also get or update the cgroup driver for containerd through the containerd configuration file.

  1. [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  2.   ...
  3.   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  4.     SystemdCgroup = true

Container Runtime - 图3tip

If you apply these change in containerd configuration file, make sure to restart containerd。

Configure the runtime for EdgeCore using Keadm

When installing EdgeCore using Keadm, you need to set --remote-runtime-endpoint=unix:///run/containerd/containerd.sock.

Container Runtime - 图4tip

If you use KubeEdge v1.12 or earlier versions, you also need to set —runtimetype=remote when executing keadm join.

On Windows, use —remote-runtime-endpoint=npipe://./pipe/containerd-containerd to configure the CRI endpoint.

KubeEdge uses cgroupfs cgroup driver as default. If you want to use the systemd cgroup driver, ensure that containerd is configured with the systemd cgroup driver. And then set --cgroupdriver=systemd when executing keadm join.

Configure the runtime for EdgeCore using binary

If you install EdgeCore using the binary, you will need to update the configuration file edgecore.yaml and modify the following parameters:

In KubeEdge v1.15 and before:

  1. modules:
  2.   edged:
  3.     containerRuntime: remote
  4.     remoteImageEndpoint: unix:///run/containerd/containerd.sock
  5.     remoteRuntimeEndpoint: unix:///run/containerd/containerd.sock

KubeEdge uses cgroupfs cgroup driver as default. If you wish to use the systemd cgroup driver, you need to ensure that containerd is configured with the systemd cgroup driver. And then modify the following parameters in the edgecore.yaml:

  1. modules:
  2.   edged:
  3.     tailoredKubeletConfig:
  4.       cgroupDriver: systemd

CRI-O

Install and configure CRI-O

Please follow the CRI-O Installation Instructions to install CRI-O.

If you need to update the sandbox(pause) image, you can modify the following settings in the CRI-O configuration file (usually located at /etc/crio/crio.conf):

  1. [plugins."io.containerd.grpc.v1.cri"]
  2.   sandbox_image = "kubeedge/pause:3.6"

CRI-O uses systemd cgroup driver as default. If you need to switch to the cgroupfs cgroup driver, you can achieve this by editing the CRI-O configuration file (/etc/crio/crio.conf) and modifying the following settings:

  1. [crio.runtime]
  2. conmon_cgroup = "pod"
  3. cgroup_manager = "cgroupfs"

Container Runtime - 图5tip

You should also note the changed conmon_cgroup, which has to be set to the value pod when using CRI-O with cgroupfs.

Configure the runtime for EdgeCore using Keadm

When installing EdgeCore using Keadm, you need to set --remote-runtime-endpoint=unix:///var/run/crio/crio.sock.

Container Runtime - 图6tip

If you use KubeEdge v1.12 or earlier versions, you also need to set —runtimetype=remote when executing keadm join.

KubeEdge uses cgroupfs cgroup driver as default. If you want to use the systemd cgroup driver, ensure that CRI-O is configured with the systemd cgroup driver. And then set --cgroupdriver=systemd when executing keadm join.

Configure the runtime for EdgeCore using binary

If you install EdgeCore using the binary, you will need to update the configuration file edgecore.yaml and modify the following parameters:

In KubeEdge v1.15 and before:

  1. modules:
  2.   edged:
  3.     containerRuntime: remote
  4.     remoteImageEndpoint: unix:///var/run/crio/crio.sock
  5.     remoteRuntimeEndpoint: unix:///var/run/crio/crio.sock

KubeEdge uses cgroupfs cgroup driver as default. If you wish to use the systemd cgroup driver, you need to ensure that CRI-O is configured with the systemd cgroup driver. And then modify the following parameters in the edgecore.yaml:

  1. modules:
  2.   edged:
  3.     tailoredKubeletConfig:
  4.       cgroupDriver: systemd

Docker Engine

Container Runtime - 图7tip

Dockershim has been removed from KubeEdge v1.14. Users can’t use docker runtime directly to manage edge containers. Read the Dockershim Removal FAQ for further details.。

Install and configure Docker and cri-dockerd

The following installation steps are only applicable to KubeEdge v1.14 and later versions. If you use an earlier version, you only need to install Docker, configure --runtimetype=docker and --remote-runtime-endpoint=unix:///var/run/dockershim.sock when executing keadm join.

  1. Follow the Docker Engine Installation Guide to install Docker.
  2. Follow the cri-dockerd Installation Guide to install cri-dockerd.
  3. Install CNI Plugin

You can refer to the install_cni_plugins function in the kubeedge script for installing CNI plugins. It’s provided for reference purposes.

Configure the runtime for EdgeCore using Keadm

When installing EdgeCore using Keadm, you need to set --remote-runtime-endpoint=unix:///var/run/cri-dockerd.sock.

Container Runtime - 图8tip

When using cri-dockerd, the corresponding runtimetype is “remote”, not “docker”.

KubeEdge uses cgroupfs cgroup driver as default. If you want to use the systemd cgroup driver, ensure that Docker is configured with the systemd cgroup driver. And then set --cgroupdriver=systemd when executing keadm join.

Configure the runtime for EdgeCore using binary

If you install EdgeCore using the binary, you will need to update the configuration file edgecore.yaml and modify the following parameters:

In KubeEdge v1.15 and before:

  1. modules:
  2.   edged:
  3.     containerRuntime: remote
  4.     remoteImageEndpoint: unix:///var/run/cri-dockerd.sock
  5.     remoteRuntimeEndpoint: unix:///var/run/cri-dockerd.sock

KubeEdge uses cgroupfs cgroup driver as default. If you wish to use the systemd cgroup driver, you need to ensure that Docker is configured with the systemd cgroup driver. And then modify the following parameters in the edgecore.yaml:

  1. modules:
  2.   edged:
  3.     tailoredKubeletConfig:
  4.       cgroupDriver: systemd

Kata Containers

Kata Containers is a container runtime created to address security challenges in the multi-tenant, untrusted cloud environment. However, multi-tenancy support is still in KubeEdge’s backlog. If you have a downstream customized KubeEdge which supports multi-tenancy already then Kata Containers is a good option for a lightweight and secure container runtime.

Follow the install guide to install and configure containerd and Kata Containers.

If you have “kata-runtime” installed, run this command to check if your host system can run and create a Kata Container:

  1. kata-runtime kata-check

RuntimeClass is a feature for selecting the container runtime configuration to use to run a pod’s containers that is supported since containerd v1.2.0. If your containerd version is later than v1.2.0, you have two choices to configure containerd to use Kata Containers:

  • Kata Containers as a RuntimeClass
  • Kata Containers as a runtime for untrusted workloads

Suppose you have configured Kata Containers as the runtime for untrusted workloads. In order to verify whether it works on your edge node, you can run:

  1. cat nginx-untrusted.yaml
  2. apiVersion: v1
  3. kind: Pod
  4. metadata:
  5.   name: nginx-untrusted
  6.   annotations:
  7.     io.kubernetes.cri.untrusted-workload: "true"
  8. spec:
  9.   containers:
  10.   - name: nginx
  11.     image: nginx
  1. kubectl create -f nginx-untrusted.yaml
  3. # verify the container is running with qemu hypervisor on edge side,
  4. ps aux | grep qemu
  5. root      3941  3.0  1.0 2971576 174648 ?      Sl   17:38   0:02 /usr/bin/qemu-system-aarch64
  7. crictl pods
  8. POD ID              CREATED              STATE               NAME                NAMESPACE           ATTEMPT
  9. b1c0911644cb9       About a minute ago   Ready               nginx-untrusted     default             0

Virtlet

Make sure no libvirt is running on the worker nodes.

Steps

  1. Install CNI plugin:

    Download CNI plugin release and extract it:

    1. $ wget https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz
    3. # Extract the tarball
    4. $ mkdir cni
    5. $ tar -zxvf v0.2.0.tar.gz -C cni
    7. $ mkdir -p /opt/cni/bin
    8. $ cp ./cni/* /opt/cni/bin/

    Configure CNI plugin:

    1. $ mkdir -p /etc/cni/net.d/
    3. $ cat >/etc/cni/net.d/bridge.conf <<EOF
    4. {
    5.   "cniVersion": "0.3.1",
    6.   "name": "containerd-net",
    7.   "type": "bridge",
    8.   "bridge": "cni0",
    9.   "isGateway": true,
    10.   "ipMasq": true,
    11.   "ipam": {
    12.     "type": "host-local",
    13.     "subnet": "10.88.0.0/16",
    14.     "routes": [
    15.       { "dst": "0.0.0.0/0" }
    16.     ]
    17.   }
    18. }
    19. EOF

  2. Setup VM runtime: Use the script hack/setup-vmruntime.sh to set up a VM runtime. It makes use of the Arktos Runtime release to start three containers:

    vmruntime_vms vmruntime_libvirt vmruntime_virtlet