Adding non-Kubernetes workloads to your mesh

In this guide, we’ll walk you through an example of mesh expansion: setting up and configuring an example non-Kubernetes workload and adding it to your Linkerd mesh.

Overall flow

In this guide, we’ll take you through how to:

  1. Install the Linkerd proxy onto a virtual or physical machine outside the Kubernetes cluster.
  2. Configure network rules so traffic is routed through the proxy.
  3. Register the external workload in the mesh.
  4. Exercise traffic patterns and apply authorization policies that affect the external workload.

We’ll be using SPIRE as our identity mechanism to generate a workload identity.

Prerequisites

You will need:

  • A functioning Linkerd installation and its trust anchor.
  • A cluster that you have elevated privileges to. For local development, you can use kind or k3d.
  • A physical or virtual machine.
  • NET_CAP privileges on the machine, so iptables rules can be modified.
  • IP connectivity from the machine to every pod in the mesh.
  • A working DNS setup such that the machine is able to resolve DNS names for in-cluster Kubernetes workloads.

Getting the current trust anchor and key

To be able to use mutual TLS across cluster boundaries, the off-cluster machine and the cluster need to have a shared trust anchor. For the purposes of this tutorial, we will assume that you have access to the trust anchor certificate and secret key for your Linkerd deployment and placed it in files called ca.key and ca.crt.

Install SPIRE on your machine

Linkerd’s proxies normally obtain TLS certificates from the identity component of Linkerd’s control plane. In order to attest their identity, they use the Kubernetes Service Account token that is provided to each Pod.

Since our external workload lives outside of Kubernetes, the concept of Service Account tokens does not exist. Instead, we turn to the SPIFFE framework and its SPIRE implementation to create identities for off-cluster resources. Thus, for mesh expansion, we configure the Linkerd proxy to obtain its certificates directly from SPIRE instead of the Linkerd’s identity service. The magic of SPIFFE is that these certificates are compatible with those generated by Linkerd on the cluster.

In production, you may already have your own identity infrastructure built on top of SPIFFE that can be used by the proxies on external machines. For this tutorial however, we can take you through installing and setting up a minimal SPIRE environment on your machine. To begin with you need to install SPIRE by downloading it from the SPIRE GitHub releases page. For example:

  1. wget https://github.com/spiffe/SPIRE/releases/download/v1.8.2/SPIRE-1.8.2-linux-amd64-musl.tar.gz
  2. tar zvxf SPIRE-1.8.2-linux-amd64-musl.tar.gz
  3. cp -r SPIRE-1.8.2/. /opt/SPIRE/

Then you need to configure the SPIRE server on your machine:

  1. cat >/opt/SPIRE/server.cfg <<EOL
  2. server {
  3.     bind_address = "127.0.0.1"
  4.     bind_port = "8081"
  5.     trust_domain = "root.linkerd.cluster.local"
  6.     data_dir = "/opt/SPIRE/data/server"
  7.     log_level = "DEBUG"
  8.     ca_ttl = "168h"
  9.     default_x509_svid_ttl = "48h"
  10. }
  12. plugins {
  13.     DataStore "sql" {
  14.         plugin_data {
  15.             database_type = "sqlite3"
  16.             connection_string = "/opt/SPIRE/data/server/datastore.sqlite3"
  17.         }
  18.     }
  20.     KeyManager "disk" {
  21.         plugin_data {
  22.             keys_path = "/opt/SPIRE/data/server/keys.json"
  23.         }
  24.     }
  26.     NodeAttestor "join_token" {
  27.         plugin_data {}
  28.     }
  30.     UpstreamAuthority "disk" {
  31.         plugin_data {
  32.             cert_file_path = "/opt/SPIRE/certs/ca.crt"
  33.             key_file_path = "/opt/SPIRE/certs/ca.key"
  34.             }
  35.     }
  36. }
  37. EOL

This file configures the SPIRE server. It assumes that your root cert and key that you have installed Linkerd with are placed in the /opt/SPIRE/certs directory.

Additionally, you need will need to configure the SPIRE agent:

  1. cat >/opt/SPIRE/agent.cfg <<EOL
  2. agent {
  3.     data_dir = "/opt/SPIRE/data/agent"
  4.     log_level = "DEBUG"
  5.     trust_domain = "root.linkerd.cluster.local"
  6.     server_address = "localhost"
  7.     server_port = 8081
  9.     # Insecure bootstrap is NOT appropriate for production use but is ok for
  10.     # simple testing/evaluation purposes.
  11.     insecure_bootstrap = true
  12. }
  14. plugins {
  15.    KeyManager "disk" {
  16.         plugin_data {
  17.             directory = "/opt/SPIRE/data/agent"
  18.         }
  19.     }
  21.     NodeAttestor "join_token" {
  22.         plugin_data {}
  23.     }
  25.     WorkloadAttestor "unix" {
  26.         plugin_data {}
  27.     }
  28. }
  29. EOL

Now you need to start the server and provide a registration policy for your workload. The server is the component that issues certificates. To begin with, start the SPIRE server and verify that it is healthy:

  1. SPIRE-server run -config ./server.cfg &&
  2. SPIRE-server healthcheck

Now you need to register the agent and run it. The agent queries the SPIRE server to attest (authenticate) workloads.

  1. AGENT_TOKEN=$(SPIRE-server token generate -spiffeID spiffe://root.linkerd.cluster.local/agent -output json | jq -r '.value')
  2. SPIRE-agent run -config ./agent.cfg -joinToken "$AGENT_TOKEN" &
  3. SPIRE-agent healthcheck
  5. Agent is healthy.

After both the server and agent are running, you need to provide a registration policy for your workload. For the sake of simplicity, we can put together a simple registration policy that hands out a predefined SPIFFE identity to any process that runs under the root UID.

  1. SPIRE-server entry create -parentID spiffe://root.linkerd.cluster.local/agent  \
  2.     -spiffeID spiffe://root.linkerd.cluster.local/external-workload -selector unix:uid:$(id -u root)
  4. Entry ID      : ac5e2354-596a-4059-85f7-5b76e3bb53b3
  5. SPIFFE ID     : spiffe://root.linkerd.cluster.local/external-workload
  6. Parent ID     : spiffe://root.linkerd.cluster.local/agent
  7. TTL           : 3600
  8. Selector      : unix:uid:0

Registering the external workload with the mesh

For Linkerd to know about the external workload and be able to route traffic to it, we need to supply some information. This is done via an ExternalWorkload CRD that needs to be present in the cluster. Create one now:

  1. machine_IP=<the ip address of your machine>
  2. kubectl --context=west apply -f - <<EOF
  3. apiVersion: workload.linkerd.io/v1alpha1
  4. kind: ExternalWorkload
  5. metadata:
  6.   name: external-workload
  7.   namespace: mixed-env
  8.   labels:
  9.     location: vm
  10.     app: legacy-app
  11.     workload_name: external-workload
  12. spec:
  13.   meshTls:
  14.     identity: "spiffe://root.linkerd.cluster.local/external-workload"
  15.     serverName: "external-workload.cluster.local"
  16.   workloadIPs:
  17.   - ip: $machine_IP
  18.   ports:
  19.   - port: 80
  20.     name: http
  21. status:
  22.   conditions:
  23.     - type: Ready
  24.       status: "True"
  25.       lastTransitionTime: "2024-01-24T11:53:43Z"
  26. EOF

This will create an ExternalWorkload resource that will be used to discover workloads that live outside of Kubernetes. A Service object can select over these resources the same way it selects over Pods, but more on that later.

Installing the Linkerd proxy on the machine

We need to install and run the Linkerd proxy on the machine. Typically the proxy runs as a container in Kubernetes. The container itself has some additional machinery that is specific to bootstrapping identity in Kubernetes environments. When in a foreign environment, we do not need this functionality, so we can simply get the proxy binary:

  1. LINKERD_VERSION=enterprise-2.15.0
  2. mkdir /opt/linkerd-proxy && cs /opt/linkerd-proxy
  3. id=$(docker create cr.l5d.io/linkerd/proxy:$LINKERD_VERSION)
  4. docker cp $id:/usr/lib/linkerd/linkerd2-proxy ./linkerd-proxy
  5. docker rm -v $id

Configuring and running the proxy

The machine network configuration needs to be set up for traffic to be steered through the proxy. This could be done by adding the following iptables rules:

  1. PROXY_INBOUND_PORT=4143
  2. PROXY_OUTBOUND_PORT=4140
  3. PROXY_USER_UID=$(id -u root)
  4. # default inbound and outbound ports to ignore
  5. INBOUND_PORTS_TO_IGNORE="4190,4191,4567,4568"
  6. OUTBOUND_PORTS_TO_IGNORE="4567,4568"
  8. iptables -t nat -N PROXY_INIT_REDIRECT
  9. # ignore inbound ports
  10. iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports $INBOUND_PORTS_TO_IGNORE -j RETURN
  11. # redirect all incoming traffic to proxy's inbound port
  12. iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port $PROXY_INBOUND_PORT
  13. iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT
  15. # outbound rules
  16. iptables -t nat -N PROXY_INIT_OUTPUT
  17. # ignore proxy user
  18. iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner $PROXY_USER_UID -j RETURN
  19. # ignore loopback
  20. iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN
  21. # ignore outbound ports
  22. iptables -t nat -A PROXY_INIT_OUTPUT -p tcp --match multiport --dports $OUTBOUND_PORTS_TO_IGNORE -j RETURN
  23. # redirect all outgoing traffic proxy's outbound port
  24. iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port $PROXY_OUTBOUND_PORT
  25. iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT
  26. iptables-save -t nat

These rules ensure that traffic is correctly routed through the proxy. Now that this is done, we need to run the proxy with the correct environment variables set up:

  1. export LINKERD2_PROXY_IDENTITY_SERVER_ID="spiffe://root.linkerd.cluster.local/external-workload"
  2. export LINKERD2_PROXY_IDENTITY_SERVER_NAME="external-workload.cluster.local"
  3. export LINKERD2_PROXY_POLICY_WORKLOAD="{\"ns\":\"mixed-env\", \"external_workload\":\"external-workload\"}"
  4. export LINKERD2_PROXY_DESTINATION_CONTEXT="{\"ns\":\"mixed-env\", \"nodeName\":\"my-vm\", \"external_workload\":\"external-workload\"}"
  5. export LINKERD2_PROXY_DESTINATION_SVC_ADDR="linkerd-dst-headless.linkerd.svc.cluster.local.:8086"
  6. export LINKERD2_PROXY_DESTINATION_SVC_NAME="linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local"
  7. export LINKERD2_PROXY_POLICY_SVC_NAME="linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local"
  8. export LINKERD2_PROXY_POLICY_SVC_ADDR="linkerd-policy.linkerd.svc.cluster.local.:8090"
  9. export LINKERD2_PROXY_IDENTITY_SPIRE_SOCKET="unix:///tmp/spire-agent/public/api.sock"
  10. export LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS=`cat /opt/SPIRE/crts/ca.crt`
  12. ./linkerd-proxy

Start an application workload on the machine

Now that the proxy is running on the machine, you can start another workload on it that will be reachable from within the cluster. Make sure that you run this application under a user account different than the one the proxy is using. Let’s use the bb utility to mimic a workload:

  1. docker run -p 80:80 buoyantio/bb:v0.0.5 terminus \
  2.     --h1-server-port 80 \
  3.     --response-text hello-from-external-vm

Send encrypted traffic from and to the machine

Now that everything is running, you can send traffic from an in-cluster workload to the machine. Let’s start by creating our client as a workload in the cluster:

  1. kubectl apply -f - <<EOF
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   name: client
  6.   namespace: mixed-env
  7. ---
  8. apiVersion: v1
  9. kind: Pod
  10. metadata:
  11.   name: client
  12.   namespace: mixed-env
  13.   annotations:
  14.     linkerd.io/inject: enabled
  15. spec:
  16.   volumes:
  17.   - name: shared-data
  18.     emptyDir: {}
  19.   containers:
  20.   - name: client
  21.     image: cr.l5d.io/linkerd/client:current
  22.     command:
  23.       - "sh"
  24.       - "-c"
  25.       - >
  26.         while true; do
  27.           sleep 3600;
  28.         done        
  29.   serviceAccountName: client
  30. EOF

You can also create a service that selects over both the machine as well as an in-cluster workload:

  1. kubectl apply -f - <<EOF
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: legacy-app
  6.   namespace: mixed-env
  7. spec:
  8.   type: ClusterIP
  9.   selector:
  10.     app: legacy-app
  11.   ports:
  12.   - port: 80
  13.     protocol: TCP
  14.     name: one
  15. ---
  16. apiVersion: v1
  17. kind: Service
  18. metadata:
  19.   name: legacy-app-cluster
  20.   namespace: mixed-env
  21. spec:
  22.   type: ClusterIP
  23.   selector:
  24.     app: legacy-app
  25.     location: cluster
  26.   ports:
  27.   - port: 80
  28.     protocol: TCP
  29.     name: one
  30. ---
  31. apiVersion: apps/v1
  32. kind: Deployment
  33. metadata:
  34.   namespace: mixed-env
  35.   name: legacy-app
  36. spec:
  37.   replicas: 1
  38.   selector:
  39.     matchLabels:
  40.       app: legacy-app
  41.   template:
  42.     metadata:
  43.       labels:
  44.         app: legacy-app
  45.         location: cluster
  46.       annotations:
  47.         linkerd.io/inject: enabled
  48.     spec:
  49.       containers:
  50.         - name: legacy-app
  51.           image: buoyantio/bb:v0.0.5
  52.           command: [ "sh", "-c"]
  53.           args:
  54.           - "/out/bb terminus --h1-server-port 80 --response-text hello-from-$POD_NAME --fire-and-forget"
  55.           ports:
  56.             - name: http-port
  57.               containerPort: 80
  58.           env:
  59.             - name: POD_NAME
  60.               valueFrom:
  61.                 fieldRef:
  62.                   fieldPath: metadata.name
  63. EOF

Now you can ssh into the client pod and observe traffic being load-balanced between both the in-cluster workload and the machine:

  1. kubectl exec -c client --stdin --tty client -n mixed-env -- bash
  2. while sleep 1; do curl -s http://legacy-app.mixed-env.svc.cluster.local:80/who-am-i| jq .; done
  4. {
  5.   "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-571813026",
  6.   "payload": "hello-from-external-workload"
  7. }
  8. {
  9.   "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-599832807",
  10.   "payload": "hello-from-legacy-app-d4446455b-2fgcr"
  11. }
  12. {
  13.   "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-634437030",
  14.   "payload": "hello-from-external-workload"
  15. }
  16. {
  17.   "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-667578518",
  18.   "payload": "hello-from-external-workload"
  19. }

Similarly, you can send traffic from the machine to the cluster:

  1. while sleep 1; do curl -s http://legacy-app-cluster.mixed-env.svc.cluster.local:80/who-am-i| jq .; done
  3. # You should start seeing responses from the in-cluster workload.
  4. {
  5.   "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-824112662",
  6.   "payload": "hello-from-legacy-app-6bb4854789-x4wbw"
  7. }
  8. {
  9.   "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-858574572",
  10.   "payload": "hello-from-legacy-app-6bb4854789-x4wbw"
  11. }
  12. {
  13.   "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-895218927",
  14.   "payload": "hello-from-legacy-app-6bb4854789-x4wbw"
  15. }

Use authorization policies with machines

Although the identity of the proxy running on the machine is not tied to a Kubernetes service account, there is still an attested identity that can be used to define authorization policies. Let’s limit the kind of traffic that can reach our in-cluster workload. Create a Server resource now:

  1. kubectl apply -f - <<EOF
  2. apiVersion: policy.linkerd.io/v1beta2
  3. kind: Server
  4. metadata:
  5.   name: in-cluster-endpoint
  6.   namespace: mixed-env
  7.   annotations:
  8.     config.linkerd.io/default-inbound-policy: "deny"
  9. spec:
  10.   podSelector:
  11.     matchLabels:
  12.       app: legacy-app
  13.   port: http
  14.   proxyProtocol: HTTP/1
  15. EOF

You can observe that we no longer get responses when we try and target the in-cluster workload from the machine. This is because our default policy is deny. We can fix that by explicitly allowing traffic from the machine by creating a policy that allows its SPIFFE id:

  1. kubectl apply -f - <<EOF
  2. apiVersion: policy.linkerd.io/v1beta2
  3. kind: Server
  4. metadata:
  5.   name: in-cluster-endpoint
  6.   namespace: mixed-env
  7.   annotations:
  8.     config.linkerd.io/default-inbound-policy: "deny"
  9. spec:
  10.   podSelector:
  11.     matchLabels:
  12.       app: legacy-app
  13.   port: http
  14.   proxyProtocol: HTTP/1
  15. apiVersion: policy.linkerd.io/v1alpha1
  16. kind: AuthorizationPolicy
  17. metadata:
  18.   name: in-cluster-endpoint-authn
  19.   namespace: mixed-env
  20. spec:
  21.   targetRef:
  22.     group: policy.linkerd.io
  23.     kind: Server
  24.     name: in-cluster-endpoint
  25.   requiredAuthenticationRefs:
  26.   - name: in-cluster-endpoint-mtls
  27.     kind: MeshTLSAuthentication
  28.     group: policy.linkerd.io
  29. ---
  30. apiVersion: policy.linkerd.io/v1alpha1
  31. kind: MeshTLSAuthentication
  32. metadata:
  33.   name: in-cluster-endpoint-mtls
  34.   namespace: mixed-env
  35. spec:
  36.   identities:
  37.     - "spiffe://root.linkerd.cluster.local/external-workload"
  38. EOF

When this policy is applied, you can observe that traffic is allowed from the machine to the in-cluster workload. Similarly, you can attach policies to an external workload object by using the externalWorkloadSelector field of the Server object.

That’s it

Congrats! You have successfully meshed a non-Kubernetes workload with Linkerd and demonstrated secure, reliable communication between it and the meshed pods on your cluster.