Setting Up Cilium in AlibabaCloud ENI Mode (beta)

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

Note

The AlibabaCloud ENI integration is still subject to some limitations. See Limitations for details.

Create a Cluster on AlibabaCloud

Setup a Kubernetes on AlibabaCloud. You can use any method you prefer. The quickest way is to create an ACK (Alibaba Cloud Container Service for Kubernetes) cluster and to replace the CNI plugin with Cilium. For more details on how to set up an ACK cluster please follow the official documentation.

Disable ACK CNI (ACK Only)

If you are running an ACK cluster, you should delete the ACK CNI.

Cilium will manage ENIs instead of the ACK CNI, so any running DaemonSet from the list below has to be deleted to prevent conflicts.

  • kube-flannel-ds

  • terway

  • terway-eni

  • terway-eniip

Note

If you are using ACK with Flannel (DaemonSet kube-flannel-ds), the Cloud Controller Manager (CCM) will create route (Pod CIDR) in VPC. If your cluster is an Managed Kubernetes you cannot disable this behavior. Please consider creating a new cluster.

  1. kubectl -n kube-system delete daemonset <terway>

The next step is to remove CRD below created by terway* CNI

  1. kubectl delete crd \
  2.     ciliumclusterwidenetworkpolicies.cilium.io \
  3.     ciliumendpoints.cilium.io \
  4.     ciliumidentities.cilium.io \
  5.     ciliumnetworkpolicies.cilium.io \
  6.     ciliumnodes.cilium.io \
  7.     bgpconfigurations.crd.projectcalico.org \
  8.     clusterinformations.crd.projectcalico.org \
  9.     felixconfigurations.crd.projectcalico.org \
  10.     globalnetworkpolicies.crd.projectcalico.org \
  11.     globalnetworksets.crd.projectcalico.org \
  12.     hostendpoints.crd.projectcalico.org \
  13.     ippools.crd.projectcalico.org \
  14.     networkpolicies.crd.projectcalico.org

Create AlibabaCloud Secrets

Before installing Cilium, a new Kubernetes Secret with the AlibabaCloud Tokens needs to be added to your Kubernetes cluster. This Secret will allow Cilium to gather information from the AlibabaCloud API which is needed to implement ToGroups policies.

AlibabaCloud Access Keys

To create a new access token the following guide can be used. These keys need to have certain RAM Permissions:

  1. {
  2.   "Version": "1",
  3.   "Statement": [{
  4.       "Action": [
  5.         "ecs:CreateNetworkInterface",
  6.         "ecs:DescribeNetworkInterfaces",
  7.         "ecs:AttachNetworkInterface",
  8.         "ecs:DetachNetworkInterface",
  9.         "ecs:DeleteNetworkInterface",
  10.         "ecs:DescribeInstanceAttribute",
  11.         "ecs:DescribeInstanceTypes",
  12.         "ecs:AssignPrivateIpAddresses",
  13.         "ecs:UnassignPrivateIpAddresses",
  14.         "ecs:DescribeInstances",
  15.         "ecs:DescribeSecurityGroups"
  16.       ],
  17.       "Resource": [
  18.         "*"
  19.       ],
  20.       "Effect": "Allow"
  21.     },
  22.     {
  23.       "Action": [
  24.         "vpc:DescribeVSwitches",
  25.         "vpc:ListTagResources",
  26.         "vpc:DescribeVpcs"
  27.       ],
  28.       "Resource": [
  29.         "*"
  30.       ],
  31.       "Effect": "Allow"
  32.     }
  33.   ]
  34. }

As soon as you have the access tokens, the following secret needs to be added, with each empty string replaced by the associated value as a base64-encoded string:

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: cilium-alibabacloud
  5.   namespace: kube-system
  6. type: Opaque
  7. data:
  8.   ALIBABA_CLOUD_ACCESS_KEY_ID: ""
  9.   ALIBABA_CLOUD_ACCESS_KEY_SECRET: ""

The base64 command line utility can be used to generate each value, for example:

  1. $ echo -n "access_key" | base64
  2. YWNjZXNzX2tleQ==

This secret stores the AlibabaCloud credentials, which will be used to connect to the AlibabaCloud API.

  1. $ kubectl create -f cilium-secret.yaml

Deploy Cilium

Note

Make sure you have Helm 3 installed. Helm 2 is no longer supported.

Setup Helm repository:

  1. helm repo add cilium https://helm.cilium.io/

Deploy Cilium release via Helm:

  1. helm install cilium cilium/cilium --version 1.12.0 \
  2.   --namespace kube-system \
  3.   --set alibabacloud.enabled=true \
  4.   --set ipam.mode=alibabacloud \
  5.   --set enableIPv4Masquerade=false \
  6.   --set tunnel=disabled

Note

You must ensure that the security groups associated with the ENIs (eth1, eth2, …) allow for egress traffic to go outside of the VPC. By default, the security groups for pod ENIs are derived from the primary ENI (eth0).

Validate the Installation

Cilium CLIManually

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

LinuxmacOSOther

  1. CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
  2. CLI_ARCH=amd64
  3. if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
  4. curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
  5. sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
  6. sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
  7. rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
  1. CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
  2. CLI_ARCH=amd64
  3. if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
  4. curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
  5. shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum
  6. sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin
  7. rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}

See the full page of releases.

To validate that Cilium has been properly installed, you can run

  1. $ cilium status --wait
  2.    /¯¯\
  3. /¯¯\__/¯¯\    Cilium:         OK
  4. \__/¯¯\__/    Operator:       OK
  5. /¯¯\__/¯¯\    Hubble:         disabled
  6. \__/¯¯\__/    ClusterMesh:    disabled
  7.    \__/
  9. DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
  10. Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
  11. Containers:       cilium-operator    Running: 2
  12.                   cilium             Running: 2
  13. Image versions    cilium             quay.io/cilium/cilium:v1.9.5: 2
  14.                   cilium-operator    quay.io/cilium/operator-generic:v1.9.5: 2

Run the following command to validate that your cluster has proper network connectivity:

  1. $ cilium connectivity test
  2. ℹ️  Monitor aggregation detected, will skip some flow validation steps
  3. ✨ [k8s-cluster] Creating namespace for connectivity check...
  4. (...)
  5. ---------------------------------------------------------------------------------------------------------------------
  6. 📋 Test Report
  7. ---------------------------------------------------------------------------------------------------------------------
  8. ✅ 69/69 tests successful (0 warnings)

Congratulations! You have a fully functional Kubernetes cluster with Cilium. 🎉

You can monitor as Cilium and all required components are being installed:

  1. $ kubectl -n kube-system get pods --watch
  2. NAME                                    READY   STATUS              RESTARTS   AGE
  3. cilium-operator-cb4578bc5-q52qk         0/1     Pending             0          8s
  4. cilium-s8w5m                            0/1     PodInitializing     0          7s
  5. coredns-86c58d9df4-4g7dd                0/1     ContainerCreating   0          8m57s
  6. coredns-86c58d9df4-4l6b2                0/1     ContainerCreating   0          8m57s

It may take a couple of minutes for all components to come up:

  1. cilium-operator-cb4578bc5-q52qk         1/1     Running   0          4m13s
  2. cilium-s8w5m                            1/1     Running   0          4m12s
  3. coredns-86c58d9df4-4g7dd                1/1     Running   0          13m
  4. coredns-86c58d9df4-4l6b2                1/1     Running   0          13m

You can deploy the “connectivity-check” to test connectivity between pods. It is recommended to create a separate namespace for this.

  1. kubectl create ns cilium-test

Deploy the check with:

  1. kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/kubernetes/connectivity-check/connectivity-check.yaml

It will deploy a series of deployments which will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:

  1. $ kubectl get pods -n cilium-test
  2. NAME                                                     READY   STATUS    RESTARTS   AGE
  3. echo-a-76c5d9bd76-q8d99                                  1/1     Running   0          66s
  4. echo-b-795c4b4f76-9wrrx                                  1/1     Running   0          66s
  5. echo-b-host-6b7fc94b7c-xtsff                             1/1     Running   0          66s
  6. host-to-b-multi-node-clusterip-85476cd779-bpg4b          1/1     Running   0          66s
  7. host-to-b-multi-node-headless-dc6c44cb5-8jdz8            1/1     Running   0          65s
  8. pod-to-a-79546bc469-rl2qq                                1/1     Running   0          66s
  9. pod-to-a-allowed-cnp-58b7f7fb8f-lkq7p                    1/1     Running   0          66s
  10. pod-to-a-denied-cnp-6967cb6f7f-7h9fn                     1/1     Running   0          66s
  11. pod-to-b-intra-node-nodeport-9b487cf89-6ptrt             1/1     Running   0          65s
  12. pod-to-b-multi-node-clusterip-7db5dfdcf7-jkjpw           1/1     Running   0          66s
  13. pod-to-b-multi-node-headless-7d44b85d69-mtscc            1/1     Running   0          66s
  14. pod-to-b-multi-node-nodeport-7ffc76db7c-rrw82            1/1     Running   0          65s
  15. pod-to-external-1111-d56f47579-d79dz                     1/1     Running   0          66s
  16. pod-to-external-fqdn-allow-google-cnp-78986f4bcf-btjn7   1/1     Running   0          66s

Note

If you deploy the connectivity check to a single node cluster, pods that check multi-node functionalities will remain in the Pending state. This is expected since these pods need at least 2 nodes to be scheduled successfully.

Once done with the test, remove the cilium-test namespace:

  1. kubectl delete ns cilium-test

Next Steps

Limitations

  • The Alibaba ENI integration of Cilium is currently only enabled for IPv4.

  • Only work with instance support ENI, refer to Instance families.