Setting up Support for External Workloads (beta)

This is a step-by-step guide on how to add external workloads (such as VMs) in to your Kubernetes cluster and to enforce security policies to restrict access.

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

Prerequisites

  • Cilium must be configured to use Kubernetes for identity allocation (identityAllocationMode set to crd). This is the default for new installations.
  • External workloads must run a recent enough kernel for k8s service access from the external host to work, see Host-Reachable Services for details.
  • External workloads must have IP connectivity with the nodes in your cluster. This requirement is typically met by establishing peering or VPN tunnels between the networks of the nodes of your cluster and your external workloads.
  • All external workloads must have a unique IP address assigned them. Node IPs of such nodes and your clusters must not conflict with each other.
  • The network between the external workloads and your cluster must allow the node-cluster communication. The exact ports are documented in the Firewall Rules section.
  • This guide assumes your external workload manages domain name resolution service via systemd.

Prepare your cluster

Enable support for external workloads

Note

First, make sure you have Helm 3 installed. Helm 2 is no longer supported.

Setup Helm repository:

  1. helm repo add cilium https://helm.cilium.io/

Your cluster must be configured with support for external workloads enabled. This can also be done by passing --set externalWorkloads.enabled=true to helm install when installing or updating Cilium:

  1. helm install cilium cilium/cilium --version 1.9.8         \
  2.   --namespace kube-system                   \
  3.   --set externalWorkloads.enabled=true

This will add a deployment for clustermesh-apiserver into your cluster, as well as the related cluster resources, such as TLS secrets.

TLS configuration

By default TLS secrets are created by helm. This has the downside that each time you run helm the TLS secrets will be re-created, forcing each external workload to be reconfigured to be able to connect to your cluster. There are two ways to get around this. You can enable a cluster job to create TLS secrets instead. This way the CA cert secrets are reused as long as they are not removed from your cluster:

  1. helm install cilium cilium/cilium --version 1.9.8                     \
  2.   --namespace kube-system                               \
  3.   --set externalWorkloads.enabled=true                  \
  4.   --set clustermesh.apiserver.tls.auto.method=cronJob

Alternatively, you can use your own CA certs. You can either create the clustermesh-apiserver-ca-cert secret in your Cilium install namespace (e.g., kube-system) yourself, or pass the CA cert and key to helm:

  1. helm install cilium cilium/cilium --version 1.9.8                     \
  2.   --namespace kube-system                               \
  3.   --set externalWorkloads.enabled=true                  \
  4.   --set clustermesh.apiserver.tls.auto.method=cronJob   \
  5.   --set clustermesh.apiserver.tls.ca.cert="<< base64 encoded CA certificate >>"   \
  6.   --set clustermesh.apiserver.tls.ca.key="<< base64 encoded CA key >>"

You can also allow the cronJob create the secrets using the first approach above and then save the generated CA certs to be used with the second approach above:

  1. kubectl -n kube-system get secret clustermesh-apiserver-ca-cert -o yaml > clustermesh-apiserver-ca-cert.yaml

Expose cluster to external workloads

clustermesh-apiserver must be exposed to the external workloads. By default this is done using a NodePort service on port 32379. It is also possible to use the LoadBalancer service type that, depending on your cloud provider, allows use of a static IP, making configuring the external workloads easier.

NodePort

GCP

AWS

NodePort is the default service type. Get the Node IP to use with:

  1. kubectl get node -o jsonpath='{.items[0].status.addresses[?(@.type == "InternalIP")].address}{"\n"}'

Add the following values to the helm install cilium command:

  1. --set clustermesh.apiserver.service.type=LoadBalancer
  2. --set clustermesh.apiserver.service.annotations."cloud.google.com/load-balancer-type"=Internal

It is also possible to use an IP address from a pre-defined subnet. In this example, gke-vip-subnet is the name of the subnet that must have been created before helm install cilium (see Google documentation for details):

  1. --set clustermesh.apiserver.service.annotations."networking.gke.io/internal-load-balancer-subnet"="gke-vip-subnet"
  2. --set clustermesh.apiserver.service.loadBalancerIP="xxx.xxx.xxx.xxx"

xxx.xxx.xxx.xxx must be an IP from gke-vip-subnet.

If not using a pre-set IP, get the service IP with:

  1. kubectl -n kube-system get svc clustermesh-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'
  1. --set clustermesh.apiserver.service.type=LoadBalancer
  2. --set clustermesh.apiserver.service.annotations."service.beta.kubernetes.io/aws-load-balancer-internal"="true"

If not using a pre-set IP, get the service IP with:

  1. kubectl -n kube-system get svc clustermesh-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'

Note

Make sure that you use the namespace in which cilium is running. Depending on which installation method you chose, this could be kube-system or cilium.

Tell your cluster about external workloads

To allow an external workload to join your cluster, the cluster must be informed about each such workload. This is done by adding a CiliumExternalWorkload (CEW) resource for each external workload. CEW resource specifies the name and labels (including namespace) for the workload. For now you must also allocate a small IP CIDR that must be unique to the workload. For example, for a VM named runtime that is to join the default namespace, you could create a file runtime.yaml with the following contents:

  1. apiVersion: cilium.io/v2
  2. kind: CiliumExternalWorkload
  3. metadata:
  4.   name: runtime
  5.   labels:
  6.     io.kubernetes.pod.namespace: default
  7. spec:
  8.   ipv4-alloc-cidr: 10.192.1.0/30

Apply this with:

  1. kubectl apply -f runtime.yaml

Extract the TLS keys for external workloads

Cilium control plane performs TLS based authentication and encryption. For this purpose, the TLS keys and certificates of clustermesh-apiserver need to be made available to external workload that wish to join the cluster.

Extract the TLS certs from your k8s cluster using extract-external-workload-certs.sh:

  1. curl -LO https://raw.githubusercontent.com/cilium/cilium/v1.9/contrib/k8s/extract-external-workload-certs.sh
  2. chmod +x extract-external-workload-certs.sh
  3. ./extract-external-workload-certs.sh

This saves the certs (ca.crt, tls.crt, tls.key) to the current directory. These files need to be copied to your external workload.

Install and configure Cilium on external workloads

Log in to the external workload. First make sure the hostname matches the name used in the CiliumExternalWorkload resource:

  1. hostname

By now you should be able to find the corresponding resource in your k8s cluster (<name> is the output from hostname above):

  1. kubectl get cew <name>

Next, copy the saved TLS certs from your kubectl command line to the external workload. Then download the install-external-workload.sh script:

  1. curl -LO https://raw.githubusercontent.com/cilium/cilium/v1.9/contrib/k8s/install-external-workload.sh
  2. chmod +x install-external-workload.sh

Before you continue you need to pre-pull the cilium image and stop the system service that would update /etc/resolv.conf:

  1. docker pull cilium/cilium:v1.9.8
  2. sudo cp /etc/resolv.conf /etc/resolv.conf.orig
  3. sudo systemctl disable systemd-resolved.service
  4. sudo service systemd-resolved stop

Then, assuming they are in the same directory:

NodePort

LoadBalancer

  1. CLUSTER_ADDR=<node-ip> CILIUM_IMAGE=cilium/cilium:v1.9.8 ./install-external-workload.sh

<node-ip> is the node IP you extracted from the k8s cluster above.

  1. CLUSTER_ADDR=<load-balancer-ip>:2379 CILIUM_IMAGE=cilium/cilium:v1.9.8 ./install-external-workload.sh

<load-balancer-ip> is the load balancer IP you extracted from the k8s cluster above.

This command launches the Cilium agent in a docker container named cilium and copies the cilium CLI to your host. This needs sudo permissions, so you may be asked for a password.

This command waits until the node has been connected to the cluster and the cluster services are available. Then it re-configures /etc/resolv.conf with the IP address of the kube-dns service.

Note

If your node has multiple IP addresses you may need to tell Cilium agent which IP to use. To this end add HOST_IP=<ip-address> to the beginning of the command line above.

Verify basic connectivity

Next you can check the status of the Cilium agent in your external workload:

  1. cilium status

You should see something like:

  1. etcd: 1/1 connected, lease-ID=7c02748328e75f57, lock lease-ID=7c02748328e75f59, has-quorum=true: 192.168.36.11:32379 - 3.4.13 (Leader)
  2. ...

Check that cluster DNS works:

  1. nslookup clustermesh-apiserver.kube-system.svc.cluster.local

Inspecting status changes in the cluster

The following command in your cluster should show the external workload IPs and their Cilium security IDs:

  1. kubectl get cew

External workloads should also be visible as Cilium Endpoints:

  1. kubectl get cep

Apply Cilium Network Policy to enforce traffic from external workloads

From the external workload, ping the backend IP of clustermesh-apiserver service to verify connectivity:

  1. ping $(cilium service list get -o jsonpath='{[?(@.spec.flags.name=="clustermesh-apiserver")].spec.backend-addresses[0].ip}')

The ping should keep running also when the following CCNP is applied in your cluster:

  1. apiVersion: cilium.io/v2
  2. kind: CiliumClusterwideNetworkPolicy
  3. metadata:
  4.   name: test-ccnp
  5.   namespace: kube-system
  6. spec:
  7.   endpointSelector:
  8.     matchLabels:
  9.       k8s-app: clustermesh-apiserver
  10.   ingress:
  11.   - fromEndpoints:
  12.     - matchLabels:
  13.         io.kubernetes.pod.name: runtime
  14.   - toPorts:
  15.     - ports:
  16.       - port: "2379"
  17.         protocol: TCP

The ping should stop if you delete these lines from the policy (e.g., kubectl edit ccnp test-ccnp):

  1. - fromEndpoints:
  2.   - matchLabels:
  3.       io.kubernetes.pod.name: runtime

Clean-up

You can remove the Cilium installation from your external workload by first removing the Cilium docker image and Cilium CLI tool:

  1. docker rm -f cilium
  2. sudo rm /usr/bin/cilium

Then restore the domain name resolver configuration:

  1. sudo cp /etc/resolv.conf.orig /etc/resolv.conf
  2. sudo systemctl enable systemd-resolved.service
  3. sudo service systemd-resolved start

Conclusion

With the above we have enabled policy-based communication between external workloads and pods in your Kubernetes cluster. We have also established service load-balancing from external workloads to your cluster backends, and configured domain name lookup in the external workload to be served by kube-dns of your cluster.