Multi-cluster Federated Services

Linkerd’s multicluster extension can create federated services which act as a union of multiple services in different clusters with the same name and namespace. By sending traffic to the federated service, that traffic will be load balanced among all endpoints of that service in all linked clusters. This allows the client to be cluster agnostic, balance traffic across multiple clusters, and be resiliant to the failure of any individual cluster.

Federated services send traffic directly to the pods of the member services rather than through a gateway. Therefore, federated services have the same requirements as pod-to-pod multicluster services:

  • The clusters must be on a flat network. In other words, pods from one cluster must be able to address and connect to pods in the other cluster.
  • The clusters must have the same trust root.
  • Any clients connecting to the federated service must be meshed.

This guide will walk you through creating a federated service to load balance traffic to a service which exists in multiple clusters. A federated service can include services from any number of clusters, but in this guide we’ll create a federated service for a service that spans 3 clusters.

Prerequisites

  • Three clusters. We will refer to them as west, east, and north in this guide.
  • The clusters must be on a flat network. In other words, pods from one cluster must be able to address and connect to pods in the other cluster.
  • Each of these clusters should be configured as kubectl contexts. We’d recommend you use the names west, east, and north so that you can follow along with this guide. It is easy to rename contexts with kubectl, so don’t feel like you need to keep them all named this way forever.

Step 1: Installing Linkerd and Linkerd-Viz

First, install Linkerd and Linkerd-Viz into all three clusters, as described in the multicluster guide. Make sure to take care that all clusters share a common trust anchor.

Step 2: Installing Linkerd-Multicluster

We will install the multicluster extension into all three clusters. We can install without the gateway because federated services use direct pod-to-pod communication.

  1. > linkerd --context west multicluster install --gateway=false | kubectl --context west apply -f -
  2. > linkerd --context west check
  4. > linkerd --context east multicluster install --gateway=false | kubectl --context east apply -f -
  5. > linkerd --context east check
  7. > linkerd --context north multicluster install --gateway=false | kubectl --context north apply -f -
  8. > linkerd --context north check

Step 3: Linking the Clusters

We use the linkerd multicluster link command to link the east and north cluster to the west cluster. This is exactly the same as in the regular Multicluster guide except that we pass the --gateway=false flag to create a Link which doesn’t require a gateway.

  1. > linkerd --context east multicluster link --cluster-name=east --gateway=false | kubectl --context west apply -f -
  2. > linkerd --context north multicluster link --cluster-name=north --gateway=false | kubectl --context west apply -f -
  3. > linkerd --context west check

Step 4: Deploy a Service

For our guide, we’ll deploy the bb service, which is a simple server that just returns a static response. We deploy it into all three clusters but configure each one with a different response string so that we can tell the responses apart:

  1. > cat <<EOF | linkerd --context east inject - | kubectl --context east apply -f -
  2. ---
  3. apiVersion: v1
  4. kind: Namespace
  5. metadata:
  6.   name: mc-demo
  7. ---
  8. apiVersion: apps/v1
  9. kind: Deployment
  10. metadata:
  11.   name: bb
  12.   namespace: mc-demo
  13. spec:
  14.   replicas: 1
  15.   selector:
  16.     matchLabels:
  17.       app: bb
  18.   template:
  19.     metadata:
  20.       labels:
  21.         app: bb
  22.     spec:
  23.       containers:
  24.       - name: terminus
  25.         image: buoyantio/bb:v0.0.6
  26.         args:
  27.         - terminus
  28.         - "--h1-server-port=8080"
  29.         - "--response-text=hello from east\n"
  30.         ports:
  31.         - containerPort: 8080
  32. ---
  33. apiVersion: v1
  34. kind: Service
  35. metadata:
  36.   name: bb
  37.   namespace: mc-demo
  38. spec:
  39.   ports:
  40.   - name: http
  41.     port: 8080
  42.     targetPort: 8080
  43.   selector:
  44.     app: bb
  45. EOF
  47. > cat <<EOF | linkerd --context north inject - | kubectl --context north apply -f -
  48. ---
  49. apiVersion: v1
  50. kind: Namespace
  51. metadata:
  52.   name: mc-demo
  53. ---
  54. apiVersion: apps/v1
  55. kind: Deployment
  56. metadata:
  57.   name: bb
  58.   namespace: mc-demo
  59. spec:
  60.   replicas: 1
  61.   selector:
  62.     matchLabels:
  63.       app: bb
  64.   template:
  65.     metadata:
  66.       labels:
  67.         app: bb
  68.     spec:
  69.       containers:
  70.       - name: terminus
  71.         image: buoyantio/bb:v0.0.6
  72.         args:
  73.         - terminus
  74.         - "--h1-server-port=8080"
  75.         - "--response-text=hello from north\n"
  76.         ports:
  77.         - containerPort: 8080
  78. ---
  79. apiVersion: v1
  80. kind: Service
  81. metadata:
  82.   name: bb
  83.   namespace: mc-demo
  84. spec:
  85.   ports:
  86.   - name: http
  87.     port: 8080
  88.     targetPort: 8080
  89.   selector:
  90.     app: bb
  91. EOF
  93. > cat <<EOF | linkerd --context west inject - | kubectl --context west apply -f -
  94. ---
  95. apiVersion: v1
  96. kind: Namespace
  97. metadata:
  98.   name: mc-demo
  99. ---
  100. apiVersion: apps/v1
  101. kind: Deployment
  102. metadata:
  103.   name: bb
  104.   namespace: mc-demo
  105. spec:
  106.   replicas: 1
  107.   selector:
  108.     matchLabels:
  109.       app: bb
  110.   template:
  111.     metadata:
  112.       labels:
  113.         app: bb
  114.     spec:
  115.       containers:
  116.       - name: terminus
  117.         image: buoyantio/bb:v0.0.6
  118.         args:
  119.         - terminus
  120.         - "--h1-server-port=8080"
  121.         - "--response-text=hello from west\n"
  122.         ports:
  123.         - containerPort: 8080
  124. ---
  125. apiVersion: v1
  126. kind: Service
  127. metadata:
  128.   name: bb
  129.   namespace: mc-demo
  130. spec:
  131.   ports:
  132.   - name: http
  133.     port: 8080
  134.     targetPort: 8080
  135.   selector:
  136.     app: bb
  137. EOF

Step 5: Label the services

We now set a label on the services to indicate that they should join a federated service.

  1. > kubectl --context east -n mc-demo label svc/bb mirror.linkerd.io/federated=member
  2. > kubectl --context north -n mc-demo label svc/bb mirror.linkerd.io/federated=member
  3. > kubectl --context west -n mc-demo label svc/bb mirror.linkerd.io/federated=member

You should immediately see a federated service created in the west cluster:

  1. > kubectl --context west -n mc-demo get svc
  2. NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
  3. bb-federated   ClusterIP   10.43.56.245   <none>        8080/TCP   114s

We can also check the status subresource of each of the Link resources to see which services have joined federated services or if there are any errors.

  1. > kubectl --context west -n linkerd-multicluster get link/east -ojsonpath='{.status.federatedServices}' | jq .
  2. [
  3.   {
  4.     "conditions": [
  5.       {
  6.         "lastTransitionTime": "2024-11-07T19:53:01Z",
  7.         "localRef": {
  8.           "group": "",
  9.           "kind": "Service",
  10.           "name": "bb-federated",
  11.           "namespace": "mc-demo"
  12.         },
  13.         "message": "",
  14.         "reason": "Mirrored",
  15.         "status": "True",
  16.         "type": "Mirrored"
  17.       }
  18.     ],
  19.     "controllerName": "linkerd.io/service-mirror",
  20.     "remoteRef": {
  21.       "group": "",
  22.       "kind": "Service",
  23.       "name": "bb",
  24.       "namespace": "mc-demo"
  25.     }
  26.   }
  27. ]
  28. > kubectl --context west -n linkerd-multicluster get link/north -ojsonpath='{.status.federatedService
  29. s}' | jq .
  30. [
  31.   {
  32.     "conditions": [
  33.       {
  34.         "lastTransitionTime": "2024-11-07T19:53:06Z",
  35.         "localRef": {
  36.           "group": "",
  37.           "kind": "Service",
  38.           "name": "bb-federated",
  39.           "namespace": "mc-demo"
  40.         },
  41.         "message": "",
  42.         "reason": "Mirrored",
  43.         "status": "True",
  44.         "type": "Mirrored"
  45.       }
  46.     ],
  47.     "controllerName": "linkerd.io/service-mirror",
  48.     "remoteRef": {
  49.       "group": "",
  50.       "kind": "Service",
  51.       "name": "bb",
  52.       "namespace": "mc-demo"
  53.     }
  54.   }
  55. ]

Step 6: Send some traffic!

We’ll create a deployment that uses curl to generate traffic to the bb-federated service.

  1. > cat <<EOF | linkerd --context west inject - | kubectl --context west apply -f -
  2. ---
  3. apiVersion: apps/v1
  4. kind: Deployment
  5. metadata:
  6.   name: traffic
  7.   namespace: mc-demo
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: traffic
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: traffic
  17.     spec:
  18.       containers:
  19.       - args:
  20.         - -c
  21.         - |
  22.           while true
  23.           do curl -s http://bb-federated:8080
  24.           echo
  25.           sleep 1
  26.           done
  27.         command:
  28.         - /bin/sh
  29.         image: curlimages/curl
  30.         name: traffic
  31. EOF

Looking at the logs from this deployment, we can see that the federated service is distributing requests across all three clusters:

  1. > kubectl --context west -n mc-demo logs deploy/traffic -c traffic
  2. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-407945949","payload":"hello from east\n"}
  3. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-420928530","payload":"hello from west\n"}
  4. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-433442439","payload":"hello from north\n"}
  5. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-445418175","payload":"hello from west\n"}
  6. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-457469540","payload":"hello from west\n"}
  7. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-469729132","payload":"hello from west\n"}
  8. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-481971153","payload":"hello from west\n"}
  9. {"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-496032705","payload":"hello from east\n"}
  10. ...

Next Steps

We now have a federated service that balances traffic accross services in three clusters. Additional clusters can be added simply by linking the new cluster and adding the mirror.linkerd.io/federated=member label to the services that you wish to add to the federated service. Similarly, services can be removed from the federated service at any time by removing the label.

You may notice that the bb-federated federated service exists only in the west cluster and not in the east or north clusters. This is because Links are directional and to keep this guide simple, we only linked north and east to west, and not the other way around. If we were to create links in both directions between all three clusters, we would get a bb-federated service in all three clusters.

Troubleshooting

  • The first step of troubleshooting should be to run the linkerd check command in each of the clusters. In particular, look for the linkerd-multicluster checks and ensure that all linked clusters are listed:
  1. linkerd-multicluster
  2. --------------------
  3. √ Link CRD exists
  4. √ Link resources are valid
  5.         * east
  6.         * north
  7. √ remote cluster access credentials are valid
  8.         * east
  9.         * north
  10. √ clusters share trust anchors
  11.         * east
  12.         * north
  13. √ service mirror controller has required permissions
  14.         * east
  15.         * north
  16. √ service mirror controllers are running
  17.         * east
  18.         * north
  • Check the status subresource of the Link resource. If any services failed to join the federated service, they will appear as an error here.
  • If a service that should join a federated service is not present in the Link status, ensure that the service matches the federated service label selector (mirror.linkerd.io/federated=memeber by default).
  • Use the linkerd diagnostics endpoints command to see all of the endpoints in a federated service:
  1. > linkerd --context west diagnostics endpoints bb-federated.mc-demo.svc.cluster.local:8080
  2. NAMESPACE   IP            PORT   POD                   SERVICE
  3. mc-demo     10.42.0.108   8080   bb-85f9bbc898-j7fbq   bb.mc-demo
  4. mc-demo     10.23.1.43    8080   bb-7d9f44c6fd-9s848   bb.mc-demo
  5. mc-demo     10.23.0.42    8080   bb-74c6c64948-j5drn   bb.mc-demo