Configure the Kuma CNI

In order for traffic to flow through the Kuma data plane, all inbound and outbound traffic for a service needs to go through its data plane proxy. The recommended way of accomplishing this is via transparent proxying .

On Kubernetes it’s handled automatically by default with the initContainer kuma-init, but this container requires certain privileges.

Another option is to use the Kuma CNI. This frees every Pod in the mesh from requiring said privileges, which can make security compliance easier.

The CNI DaemonSet itself requires elevated privileges because it writes executables to the host filesystem as root.

Install the CNI using either kumactl or Helm . The default settings are tuned for OpenShift with Multus. To use it in other environments, set the relevant configuration parameters.

Kuma CNI applies NetworkAttachmentDefinitions to applications in any namespace with kuma.io/sidecar-injection label. To apply NetworkAttachmentDefinitions to applications not in a Mesh, add the label kuma.io/sidecar-injection with the value disabled to the namespace.

Installation

Below are the details of how to set up Kuma CNI in different environments using both kumactl and helm.

  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=05-cilium.conflist" \
  7.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=05-cilium.conflist" \
  7.   kuma kuma/kuma

For Cilium versions < 1.14 you should use cni.confName=05-cilium.conf as this has changed for version starting from Cilium 1.14.

  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-calico.conflist" \
  7.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-calico.conflist" \
  7.   kuma kuma/kuma
  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/var/lib/rancher/k3s/agent/etc/cni/net.d" \
  5.   --set "cni.binDir=/bin" \
  6.   --set "cni.confName=10-flannel.conflist" \
  7.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/var/lib/rancher/k3s/agent/etc/cni/net.d" \
  5.   --set "cni.binDir=/bin" \
  6.   --set "cni.confName=10-flannel.conflist" \
  7.   kuma kuma/kuma
  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-kindnet.conflist" \
  7.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-kindnet.conflist" \
  7.   kuma kuma/kuma
  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-azure.conflist" \
  7.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-azure.conflist" \
  7.   kuma kuma/kuma
  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=15-azure-swift-overlay.conflist" \
  7.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=15-azure-swift-overlay.conflist" \
  7.   kuma kuma/kuma
  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-aws.conflist" \
  7.   --set "runtime.kubernetes.injector.sidecarContainer.redirectPortInboundV6=0" \
  8.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/opt/cni/bin" \
  6.   --set "cni.confName=10-aws.conflist" \
  7.   --set "runtime.kubernetes.injector.sidecarContainer.redirectPortInboundV6=0" \
  8.   kuma kuma/kuma

Add redirectPortInboundV6=0 as EKS has IPv6 disabled by default.

You need to enable network-policy in your cluster (for existing clusters this redeploys the nodes).

  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/home/kubernetes/bin" \
  6.   --set "cni.confName=10-calico.conflist" \
  7.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.chained=true" \
  4.   --set "cni.netDir=/etc/cni/net.d" \
  5.   --set "cni.binDir=/home/kubernetes/bin" \
  6.   --set "cni.confName=10-calico.conflist" \
  7.   kuma kuma/kuma

  1. Follow the instructions in OpenShift 3.11 installation to get the MutatingAdmissionWebhook and ValidatingAdmissionWebhook enabled (this is required for regular Kuma installation).

  2. You need to grant privileged permission to kuma-cni service account:

  1. oc adm policy add-scc-to-user privileged -z kuma-cni -n kube-system
  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.containerSecurityContext.privileged=true" \
  4.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.containerSecurityContext.privileged=true" \
  4.   kuma kuma/kuma
  1. kumactl install control-plane \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.containerSecurityContext.privileged=true" \
  4.   | kubectl apply -f -

Before using Kuma with helm, please follow these steps to configure your local helm repo.

  1. helm install --create-namespace --namespace kuma-system \
  2.   --set "cni.enabled=true" \
  3.   --set "cni.containerSecurityContext.privileged=true" \
  4.   kuma kuma/kuma

Kuma CNI taint controller

To prevent a race condition described in this issue a new controller was implemented. The controller will taint a node with NoSchedule taint to prevent scheduling before the CNI DaemonSet is running and ready. Once the CNI DaemonSet is running and ready it will remove the taint and allow other pods to be scheduled into the node.

To disable the taint controller use the following env variable:

  1. KUMA_RUNTIME_KUBERNETES_NODE_TAINT_CONTROLLER_ENABLED="false"

Merbridge CNI with eBPF

To install merbridge CNI with eBPF append the following options to the command from installation:

To use Merbridge CNI with eBPF your environment has to use Kernel >= 5.7 and have cgroup2 available

  1. --set ... \
  2. --set "cni.enabled=true" \
  3. --set "experimental.ebpf.enabled=true"

Kuma CNI logs

Logs of the are available via kubectl logs.

eBPF CNI currently doesn’t have support for exposing its logs.