Getting Started Using Kind

This guide uses kind to demonstrate deployment and operation of Cilium in a multi-node Kubernetes cluster running locally on Docker.

Install Dependencies

  1. Install docker stable as described in Install Docker Engine
  2. Install kubectl version >= v1.14.0 as described in the Kubernetes Docs
  3. Install helm >= v3.0.3 per Helm documentation: Installing Helm
  4. Install kind >= v0.7.0 per kind documentation: Installation and Usage

Configure kind

Configuring kind cluster creation is done using a YAML configuration file. This step is necessary in order to disable the default CNI and replace it with Cilium.

Create a kind-config.yaml file based on the following template. It will create a cluster with 3 worker nodes and 1 control-plane node.

  1. kind: Cluster
  2. apiVersion: kind.x-k8s.io/v1alpha4
  3. nodes:
  4. - role: control-plane
  5. - role: worker
  6. - role: worker
  7. - role: worker
  8. networking:
  9.   disableDefaultCNI: true

By default, the latest version of Kubernetes from when the kind release was created is used.

To change the version of Kubernetes being run, image has to be defined for each node. See the Node Configuration documentation for more information.

Tip

By default, kind uses the following pod and service subnets:

  1. Networking.PodSubnet     = "10.244.0.0/16"
  2. Networking.ServiceSubnet = "10.96.0.0/12"

If any of these subnets conflicts with your local network address range, update the networking section of the kind configuration file to specify different subnets that do not conflict or you risk having connectivity issues when deploying Cilium. For example:

  1. networking:
  2.   disableDefaultCNI: true
  3.   podSubnet: "10.10.0.0/16"
  4.   serviceSubnet: "10.11.0.0/16"

Create a cluster

To create a cluster with the configuration defined above, pass the kind-config.yaml you created with the --config flag of kind.

  1. kind create cluster --config=kind-config.yaml

After a couple of seconds or minutes, a 4 nodes cluster should be created.

A new kubectl context (kind-kind) should be added to KUBECONFIG or, if unset, to ${HOME}/.kube/config:

  1. kubectl cluster-info --context kind-kind

Note

The cluster nodes will remain in state NotReady until Cilium is deployed. This behavior is expected.

Install Cilium

Note

Make sure you have Helm 3 installed. Helm 2 is no longer supported.

Setup Helm repository:

  1. helm repo add cilium https://helm.cilium.io/

Preload the cilium image into each worker node in the kind cluster:

  1. docker pull quay.io/cilium/cilium:v1.11.7
  2. kind load docker-image quay.io/cilium/cilium:v1.11.7

Then, install Cilium release via Helm:

  1. helm install cilium cilium/cilium --version 1.11.7 \
  2.    --namespace kube-system \
  3.    --set kubeProxyReplacement=partial \
  4.    --set hostServices.enabled=false \
  5.    --set externalIPs.enabled=true \
  6.    --set nodePort.enabled=true \
  7.    --set hostPort.enabled=true \
  8.    --set bpf.masquerade=false \
  9.    --set image.pullPolicy=IfNotPresent \
  10.    --set ipam.mode=kubernetes

Note

To fully enable Cilium’s kube-proxy replacement (Kubernetes Without kube-proxy), cgroup v2 needs to be enabled by setting the kernel systemd.unified_cgroup_hierarchy=1 parameter. Also, cgroup v1 controllers net_cls and net_prio have to be disabled, or cgroup v1 has to be disabled (e.g. by setting the kernel cgroup_no_v1="all" parameter). This ensures that Kind nodes have their own cgroup namespace, and Cilium can attach BPF programs at the right cgroup hierarchy. To verify this, run the following commands on the host, and check that the output values are different.

  1. $ sudo ls -al /proc/$(docker inspect -f '{{.State.Pid}}' kind-control-plane)/ns/cgroup
  2. $ sudo ls -al /proc/self/ns/cgroup

See the Pull Request for more details.

Validate the Installation

Cilium CLI

Manually

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

Linux

macOS

Other

  1. curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
  2. sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
  3. sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
  4. rm cilium-linux-amd64.tar.gz{,.sha256sum}
  1. curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-darwin-amd64.tar.gz{,.sha256sum}
  2. shasum -a 256 -c cilium-darwin-amd64.tar.gz.sha256sum
  3. sudo tar xzvfC cilium-darwin-amd64.tar.gz /usr/local/bin
  4. rm cilium-darwin-amd64.tar.gz{,.sha256sum}

See the full page of releases.

To validate that Cilium has been properly installed, you can run

  1. $ cilium status --wait
  2.    /¯¯\
  3. /¯¯\__/¯¯\    Cilium:         OK
  4. \__/¯¯\__/    Operator:       OK
  5. /¯¯\__/¯¯\    Hubble:         disabled
  6. \__/¯¯\__/    ClusterMesh:    disabled
  7.    \__/
  9. DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
  10. Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
  11. Containers:       cilium-operator    Running: 2
  12.                   cilium             Running: 2
  13. Image versions    cilium             quay.io/cilium/cilium:v1.9.5: 2
  14.                   cilium-operator    quay.io/cilium/operator-generic:v1.9.5: 2

Run the following command to validate that your cluster has proper network connectivity:

  1. $ cilium connectivity test
  2. ℹ️  Monitor aggregation detected, will skip some flow validation steps
  3. ✨ [k8s-cluster] Creating namespace for connectivity check...
  4. (...)
  5. ---------------------------------------------------------------------------------------------------------------------
  6. 📋 Test Report
  7. ---------------------------------------------------------------------------------------------------------------------
  8. ✅ 69/69 tests successful (0 warnings)

Congratulations! You have a fully functional Kubernetes cluster with Cilium. 🎉

You can monitor as Cilium and all required components are being installed:

  1. $ kubectl -n kube-system get pods --watch
  2. NAME                                    READY   STATUS              RESTARTS   AGE
  3. cilium-operator-cb4578bc5-q52qk         0/1     Pending             0          8s
  4. cilium-s8w5m                            0/1     PodInitializing     0          7s
  5. coredns-86c58d9df4-4g7dd                0/1     ContainerCreating   0          8m57s
  6. coredns-86c58d9df4-4l6b2                0/1     ContainerCreating   0          8m57s

It may take a couple of minutes for all components to come up:

  1. cilium-operator-cb4578bc5-q52qk         1/1     Running   0          4m13s
  2. cilium-s8w5m                            1/1     Running   0          4m12s
  3. coredns-86c58d9df4-4g7dd                1/1     Running   0          13m
  4. coredns-86c58d9df4-4l6b2                1/1     Running   0          13m

You can deploy the “connectivity-check” to test connectivity between pods. It is recommended to create a separate namespace for this.

  1. kubectl create ns cilium-test

Deploy the check with:

  1. kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/v1.11/examples/kubernetes/connectivity-check/connectivity-check.yaml

It will deploy a series of deployments which will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:

  1. $ kubectl get pods -n cilium-test
  2. NAME                                                     READY   STATUS    RESTARTS   AGE
  3. echo-a-76c5d9bd76-q8d99                                  1/1     Running   0          66s
  4. echo-b-795c4b4f76-9wrrx                                  1/1     Running   0          66s
  5. echo-b-host-6b7fc94b7c-xtsff                             1/1     Running   0          66s
  6. host-to-b-multi-node-clusterip-85476cd779-bpg4b          1/1     Running   0          66s
  7. host-to-b-multi-node-headless-dc6c44cb5-8jdz8            1/1     Running   0          65s
  8. pod-to-a-79546bc469-rl2qq                                1/1     Running   0          66s
  9. pod-to-a-allowed-cnp-58b7f7fb8f-lkq7p                    1/1     Running   0          66s
  10. pod-to-a-denied-cnp-6967cb6f7f-7h9fn                     1/1     Running   0          66s
  11. pod-to-b-intra-node-nodeport-9b487cf89-6ptrt             1/1     Running   0          65s
  12. pod-to-b-multi-node-clusterip-7db5dfdcf7-jkjpw           1/1     Running   0          66s
  13. pod-to-b-multi-node-headless-7d44b85d69-mtscc            1/1     Running   0          66s
  14. pod-to-b-multi-node-nodeport-7ffc76db7c-rrw82            1/1     Running   0          65s
  15. pod-to-external-1111-d56f47579-d79dz                     1/1     Running   0          66s
  16. pod-to-external-fqdn-allow-google-cnp-78986f4bcf-btjn7   1/1     Running   0          66s

Note

If you deploy the connectivity check to a single node cluster, pods that check multi-node functionalities will remain in the Pending state. This is expected since these pods need at least 2 nodes to be scheduled successfully.

Once done with the test, remove the cilium-test namespace:

  1. kubectl delete ns cilium-test

Next Steps

Troubleshooting

Unable to contact k8s api-server

In the Cilium agent logs you will see:

  1. level=info msg="Establishing connection to apiserver" host="https://10.96.0.1:443" subsys=k8s
  2. level=error msg="Unable to contact k8s api-server" error="Get https://10.96.0.1:443/api/v1/namespaces/kube-system: dial tcp 10.96.0.1:443: connect: no route to host" ipAddr="https://10.96.0.1:443" subsys=k8s
  3. level=fatal msg="Unable to initialize Kubernetes subsystem" error="unable to create k8s client: unable to create k8s client: Get https://10.96.0.1:443/api/v1/namespaces/kube-system: dial tcp 10.96.0.1:443: connect: no route to host" subsys=daemon

As Kind is running nodes as containers in Docker, they’re sharing your host machines’ kernel. If Host-Reachable Services wasn’t disabled, the eBPF programs attached by Cilium may be out of date and no longer routing api-server requests to the current kind-control-plane container.

Recreating the kind cluster and using the helm command Install Cilium will detach the inaccurate eBPF programs.

Crashing Cilium agent pods

Check if Cilium agent pods are crashing with following logs. This may indicate that you are deploying a kind cluster in an environment where Cilium is already running (for example, in the Cilium development VM). This can also happen if you have other overlapping BPF cgroup type programs attached to the parent cgroup hierarchy of the kind container nodes. In such cases, either tear down Cilium, or manually detach the overlapping BPF cgroup programs running in the parent cgroup hierarchy by following the bpftool documentation. For more information, see the Pull Request.

  1. level=warning msg="+ bpftool cgroup attach /var/run/cilium/cgroupv2 connect6 pinned /sys/fs/bpf/tc/globals/cilium_cgroups_connect6" subsys=datapath-loader
  2. level=warning msg="Error: failed to attach program" subsys=datapath-loader
  3. level=warning msg="+ RETCODE=255" subsys=datapath-loader

Cluster Mesh

With Kind we can simulate Cluster Mesh in a sandbox too.

Kind Configuration

This time we need to create (2) config.yaml, one for each kubernetes cluster. We will explicitly configure their pod-network-cidr and service-cidr to not overlap.

Example kind-cluster1.yaml:

  1. kind: Cluster
  2. apiVersion: kind.x-k8s.io/v1alpha4
  3. nodes:
  4. - role: control-plane
  5. - role: worker
  6. - role: worker
  7. - role: worker
  8. networking:
  9.   disableDefaultCNI: true
  10.   podSubnet: "10.0.0.0/16"
  11.   serviceSubnet: "10.1.0.0/16"

Example kind-cluster2.yaml:

  1. kind: Cluster
  2. apiVersion: kind.x-k8s.io/v1alpha4
  3. nodes:
  4. - role: control-plane
  5. - role: worker
  6. - role: worker
  7. - role: worker
  8. networking:
  9.   disableDefaultCNI: true
  10.   podSubnet: "10.2.0.0/16"
  11.   serviceSubnet: "10.3.0.0/16"

Create Kind Clusters

We can now create the respective clusters:

  1. kind create cluster --name=cluster1 --config=kind-cluster1.yaml
  2. kind create cluster --name=cluster2 --config=kind-cluster2.yaml

Setting up Cluster Mesh

We can deploy Cilium, and complete setup by following the Cluster Mesh guide with Setting up Cluster Mesh. For Kind, we’ll want to deploy the NodePort service into the kube-system namespace.