Getting Started Using Minikube

This guide uses minikube to demonstrate deployment and operation of Cilium in a single-node Kubernetes cluster. The minikube VM requires approximately 5GB of RAM and supports hypervisors like VirtualBox that run on Linux, macOS, and Windows.

Install kubectl & minikube

  1. Install kubectl version >= v1.10.0 as described in the Kubernetes Docs
  2. Install minikube >= v1.3.1 as per minikube documentation: Install Minikube.

Note

It is important to validate that you have minikube v1.3.1 installed. Older versions of minikube are shipping a kernel configuration that is not compatible with the TPROXY requirements of Cilium >= 1.6.0.

  1. minikube version
  2. minikube version: v1.3.1
  3. commit: ca60a424ce69a4d79f502650199ca2b52f29e631
  1. Create a minikube cluster:
  1. minikube start --network-plugin=cni --memory=4096

Note

If minikube is deployed as a container (that is if docker is the configured driver), then kube-proxy replacement features like host-reachable services may not work (GitHub issue). If you experience Kubernetes service load-balancing issues, then set any other driver from the supported list.

  1. minikube start --cni=cilium --memory=4096
  2. # Only available for minikube >= v1.12.1

Note

From minikube v1.12.1+, cilium networking plugin can be enabled directly with --cni=cilium parameter in minikube start command. With this flag enabled, minikube will not only mount eBPF file system but also deploy quick-install.yaml automatically. However, this may not install the latest version of cilium.

  1. Mount the eBPF filesystem
  1. minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf

Note

In case of installing Cilium for a specific Kubernetes version, the --kubernetes-version vx.y.z parameter can be appended to the minikube start command for bootstrapping the local cluster. By default, minikube will install the most recent version of Kubernetes.

Install Cilium

Install Cilium as DaemonSet into your new Kubernetes cluster. The DaemonSet will automatically install itself as Kubernetes CNI plugin.

With quick-install.yaml

With experimental-install.yaml

Note

quick-install.yaml is a pre-rendered Cilium chart template. The template is generated using helm template command with default configuration parameters without any customization.

In case of installing Cilium with CRIO, please see CRIO instructions.

  1. kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml

Warning

experimental-install.yaml is a pre-rendered Cilium chart template with experimental features enabled. These features may include unreleased or beta features that are not considered production-ready. While it provides a convenient way to try out experimental features, It should only be used in testing environments.

  1. kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/experimental-install.yaml

Validate the Installation

You can monitor as Cilium and all required components are being installed:

  1. kubectl -n kube-system get pods --watch
  2. NAME                                    READY   STATUS              RESTARTS   AGE
  3. cilium-operator-cb4578bc5-q52qk         0/1     Pending             0          8s
  4. cilium-s8w5m                            0/1     PodInitializing     0          7s
  5. coredns-86c58d9df4-4g7dd                0/1     ContainerCreating   0          8m57s
  6. coredns-86c58d9df4-4l6b2                0/1     ContainerCreating   0          8m57s

It may take a couple of minutes for all components to come up:

  1. cilium-operator-cb4578bc5-q52qk         1/1     Running   0          4m13s
  2. cilium-s8w5m                            1/1     Running   0          4m12s
  3. coredns-86c58d9df4-4g7dd                1/1     Running   0          13m
  4. coredns-86c58d9df4-4l6b2                1/1     Running   0          13m

Deploy the connectivity test

You can deploy the “connectivity-check” to test connectivity between pods. It is recommended to create a separate namespace for this.

  1. kubectl create ns cilium-test

Deploy the check with:

  1. kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/v1.9/examples/kubernetes/connectivity-check/connectivity-check.yaml

It will deploy a series of deployments which will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:

  1. $ kubectl get pods -n cilium-test
  2. NAME                                                     READY   STATUS    RESTARTS   AGE
  3. echo-a-76c5d9bd76-q8d99                                  1/1     Running   0          66s
  4. echo-b-795c4b4f76-9wrrx                                  1/1     Running   0          66s
  5. echo-b-host-6b7fc94b7c-xtsff                             1/1     Running   0          66s
  6. host-to-b-multi-node-clusterip-85476cd779-bpg4b          1/1     Running   0          66s
  7. host-to-b-multi-node-headless-dc6c44cb5-8jdz8            1/1     Running   0          65s
  8. pod-to-a-79546bc469-rl2qq                                1/1     Running   0          66s
  9. pod-to-a-allowed-cnp-58b7f7fb8f-lkq7p                    1/1     Running   0          66s
  10. pod-to-a-denied-cnp-6967cb6f7f-7h9fn                     1/1     Running   0          66s
  11. pod-to-b-intra-node-nodeport-9b487cf89-6ptrt             1/1     Running   0          65s
  12. pod-to-b-multi-node-clusterip-7db5dfdcf7-jkjpw           1/1     Running   0          66s
  13. pod-to-b-multi-node-headless-7d44b85d69-mtscc            1/1     Running   0          66s
  14. pod-to-b-multi-node-nodeport-7ffc76db7c-rrw82            1/1     Running   0          65s
  15. pod-to-external-1111-d56f47579-d79dz                     1/1     Running   0          66s
  16. pod-to-external-fqdn-allow-google-cnp-78986f4bcf-btjn7   1/1     Running   0          66s

Note

If you deploy the connectivity check to a single node cluster, pods that check multi-node functionalities will remain in the Pending state. This is expected since these pods need at least 2 nodes to be scheduled successfully.

Specify Environment Variables

Specify the namespace in which Cilium is installed as CILIUM_NAMESPACE environment variable. Subsequent commands reference this environment variable.

  1. export CILIUM_NAMESPACE=kube-system

Enable Hubble for Cluster-Wide Visibility

Hubble is the component for observability in Cilium. To obtain cluster-wide visibility into your network traffic, deploy Hubble Relay and the UI as follows on your existing installation:

Installation via Helm

Installation via quick-hubble-install.yaml

If you installed Cilium via helm install, you may enable Hubble Relay and UI with the following command:

  1. helm upgrade cilium cilium/cilium --version 1.9.8 \
  2.    --namespace $CILIUM_NAMESPACE \
  3.    --reuse-values \
  4.    --set hubble.listenAddress=":4244" \
  5.    --set hubble.relay.enabled=true \
  6.    --set hubble.ui.enabled=true

On Cilium 1.9.1 and older, the Cilium agent pods will be restarted in the process.

If you installed Cilium 1.9.2 or newer via the provided quick-install.yaml, you may deploy Hubble Relay and UI on top of your existing installation with the following command:

  1. kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-hubble-install.yaml

Installation via quick-hubble-install.yaml only works if the installed Cilium version is 1.9.2 or newer. Users of Cilium 1.9.0 or 1.9.1 are encouraged to upgrade to a newer version by applying the most recent Cilium quick-install.yaml first.

Alternatively, it is possible to manually generate a YAML manifest for the Cilium DaemonSet and Hubble Relay/UI as follows. The generated YAML can be applied on top of an existing installation:

  1. # Set this to your installed Cilium version
  2. export CILIUM_VERSION=1.9.1
  3. # Please set any custom Helm values you may need for Cilium,
  4. # such as for example `--set operator.replicas=1` on single-cluster nodes.
  5. helm template cilium cilium/cilium --version $CILIUM_VERSION \\
  6.    --namespace $CILIUM_NAMESPACE \\
  7.    --set hubble.tls.auto.method="cronJob" \\
  8.    --set hubble.listenAddress=":4244" \\
  9.    --set hubble.relay.enabled=true \\
  10.    --set hubble.ui.enabled=true > cilium-with-hubble.yaml
  11. # This will modify your existing Cilium DaemonSet and ConfigMap
  12. kubectl apply -f cilium-with-hubble.yaml

The Cilium agent pods will be restarted in the process.

Once the Hubble UI pod is started, use port forwarding for the hubble-ui service. This allows opening the UI locally on a browser:

  1. kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-ui --address 0.0.0.0 --address :: 12000:80

And then open http://localhost:12000/ to access the UI.

Hubble UI is not the only way to get access to Hubble data. A command line tool, the Hubble CLI, is also available. It can be installed by following the instructions below:

Linux

MacOS

Windows

Download the latest hubble release:

  1. export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
  2. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz"
  3. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz.sha256sum"
  4. sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
  5. tar zxf hubble-linux-amd64.tar.gz

and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

  1. sudo mv hubble /usr/local/bin

Download the latest hubble release:

  1. export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
  2. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz"
  3. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz.sha256sum"
  4. shasum -a 256 -c hubble-darwin-amd64.tar.gz.sha256sum
  5. tar zxf hubble-darwin-amd64.tar.gz

and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

  1. sudo mv hubble /usr/local/bin

Download the latest hubble release:

  1. curl -LO "https://raw.githubusercontent.com/cilium/hubble/master/stable.txt"
  2. set /p HUBBLE_VERSION=<stable.txt
  3. curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz"
  4. curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum"
  5. certutil -hashfile hubble-windows-amd64.tar.gz SHA256
  6. type hubble-windows-amd64.tar.gz.sha256sum
  7. :: verify that the checksum from the two commands above match
  8. tar zxf hubble-windows-amd64.tar.gz

and move the hubble.exe CLI to a directory listed in the %PATH% environment variable after extracting it from the tarball.

Similarly to the UI, use port forwarding for the hubble-relay service to make it available locally:

  1. kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-relay --address 0.0.0.0 --address :: 4245:80

In a separate terminal window, run the hubble status command specifying the Hubble Relay address:

  1. $ hubble --server localhost:4245 status
  2. Healthcheck (via localhost:4245): Ok
  3. Current/Max Flows: 5455/16384 (33.29%)
  4. Flows/s: 11.30
  5. Connected Nodes: 4/4

If Hubble Relay reports that all nodes are connected, as in the example output above, you can now use the CLI to observe flows of the entire cluster:

  1. hubble --server localhost:4245 observe

If you encounter any problem at this point, you may seek help on Slack.

Tip

Hubble CLI configuration can be persisted using a configuration file or environment variables. This avoids having to specify options specific to a particular environment every time a command is run. Run hubble help config for more information.

For more information about Hubble and its components, see the Observability section.

Next steps

Now that you have a Kubernetes cluster with Cilium up and running, you can take a couple of next steps to explore various capabilities: