Multi-cluster communication with StatefulSets

Linkerd’s multi-cluster extension works by “mirroring” service information between clusters. Exported services in a target cluster will be mirrored as clusterIP replicas. By default, every exported service will be mirrored as clusterIP. When running workloads that require a headless service, such as StatefulSets, Linkerd’s multi-cluster extension can be configured with support for headless services to preserve the service type. Exported services that are headless will be mirrored in a source cluster as headless, preserving functionality such as DNS record creation and the ability to address an individual pod.

This guide will walk you through installing and configuring Linkerd and the multi-cluster extension with support for headless services and will exemplify how a StatefulSet can be deployed in a target cluster. After deploying, we will also look at how to communicate with an arbitrary pod from the target cluster’s StatefulSet from a client in the source cluster. For a more detailed overview on how multi-cluster support for headless services work, check out multi-cluster communication.

Prerequisites

  • Two Kubernetes clusters. They will be referred to as east and west with east being the “source” cluster and “west” the target cluster respectively. These can be in any cloud or local environment, this guide will make use of k3d to configure two local clusters.
  • smallstep/CLI to generate certificates for Linkerd installation.
  • linkerd:stable-2.11.0 to install Linkerd.

To help with cluster creation and installation, there is a demo repository available. Throughout the guide, we will be using the scripts from the repository, but you can follow along without cloning or using the scripts.

Install Linkerd multi-cluster with headless support

To start our demo and see everything in practice, we will go through a multi-cluster scenario where a pod in an east cluster will try to communicate to an arbitrary pod from a west cluster.

The first step is to clone the demo repository on your local machine.

  1. # clone example repository
  2. $ git clone [email protected]:mateiidavid/l2d-k3d-statefulset.git
  3. $ cd l2d-k3d-statefulset

The second step consists of creating two k3d clusters named east and west, where the east cluster is the source and the west cluster is the target. When creating our clusters, we need a shared trust root. Luckily, the repository you have just cloned includes a handful of scripts that will greatly simplify everything.

  1. # create k3d clusters
  2. $ ./create.sh
  4. # list the clusters
  5. $ k3d cluster list
  6. NAME   SERVERS   AGENTS   LOADBALANCER
  7. east   1/1       0/0      true
  8. west   1/1       0/0      true

Once our clusters are created, we will install Linkerd and the multi-cluster extension. Finally, once both are installed, we need to link the two clusters together so their services may be mirrored. To enable support for headless services, we will pass an additional --set "enableHeadlessServices=true flag to linkerd multicluster link. As before, these steps are automated through the provided scripts, but feel free to have a look!

  1. # Install Linkerd and multicluster, output to check should be a success
  2. $ ./install.sh
  4. # Next, link the two clusters together
  5. $ ./link.sh

Perfect! If you’ve made it this far with no errors, then it’s a good sign. In the next chapter, we’ll deploy some services and look at how communication works.

Pod-to-Pod: from east, to west

With our install steps out of the way, we can now focus on our pod-to-pod communication. First, we will deploy our pods and services:

  • We will mesh the default namespaces in east and west.
  • In west, we will deploy an nginx StatefulSet with its own headless service, nginx-svc.

  • In east, our script will deploy a curl pod that will then be used to curl the nginx service.

    1. # deploy services and mesh namespaces
    2. $ ./deploy.sh
    4. # verify both clusters
    5. #
    6. # verify east
    7. $ kubectl --context=k3d-east get pods
    8. NAME                    READY   STATUS        RESTARTS   AGE
    9. curl-56dc7d945d-96r6p   2/2     Running       0          7s
    11. # verify west has headless service
    12. $ kubectl --context=k3d-west get services
    13. NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
    14. kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   10m
    15. nginx-svc    ClusterIP   None         <none>        80/TCP    8s
    17. # verify west has statefulset
    18. #
    19. # this may take a while to come up
    20. $ kubectl --context=k3d-west get pods
    21. NAME          READY   STATUS    RESTARTS   AGE
    22. nginx-set-0   2/2     Running   0          53s
    23. nginx-set-1   2/2     Running   0          43s
    24. nginx-set-2   2/2     Running   0          36s

Before we go further, let’s have a look at the endpoints object for the nginx-svc:

  1. $ kubectl --context=k3d-west get endpoints nginx-svc -o yaml
  2. ...
  3. subsets:
  4. - addresses:
  5.   - hostname: nginx-set-0
  6.     ip: 10.42.0.31
  7.     nodeName: k3d-west-server-0
  8.     targetRef:
  9.       kind: Pod
  10.       name: nginx-set-0
  11.       namespace: default
  12.       resourceVersion: "114743"
  13.       uid: 7049f1c1-55dc-4b7b-a598-27003409d274
  14.   - hostname: nginx-set-1
  15.     ip: 10.42.0.32
  16.     nodeName: k3d-west-server-0
  17.     targetRef:
  18.       kind: Pod
  19.       name: nginx-set-1
  20.       namespace: default
  21.       resourceVersion: "114775"
  22.       uid: 60df15fd-9db0-4830-9c8f-e682f3000800
  23.   - hostname: nginx-set-2
  24.     ip: 10.42.0.33
  25.     nodeName: k3d-west-server-0
  26.     targetRef:
  27.       kind: Pod
  28.       name: nginx-set-2
  29.       namespace: default
  30.       resourceVersion: "114808"
  31.       uid: 3873bc34-26c4-454d-bd3d-7c783de16304

We can see, based on the endpoints object that the service has three endpoints, with each endpoint having an address (or IP) whose hostname corresponds to a StatefulSet pod. If we were to do a curl to any of these endpoints directly, we would get an answer back. We can test this out by applying the curl pod to the west cluster:

  1. $ kubectl --context=k3d-west apply -f east/curl.yml
  2. $ kubectl --context=k3d-west get pods
  3. NAME                    READY   STATUS            RESTARTS   AGE
  4. nginx-set-0             2/2     Running           0          5m8s
  5. nginx-set-1             2/2     Running           0          4m58s
  6. nginx-set-2             2/2     Running           0          4m51s
  7. curl-56dc7d945d-s4n8j   0/2     PodInitializing   0          4s
  9. $ kubectl --context=k3d-west exec -it curl-56dc7d945d-s4n8j -c curl -- bin/sh
  10. /$ # prompt for curl pod

If we now curl one of these instances, we will get back a response.

  1. # exec'd on the pod
  2. / $ curl nginx-set-0.nginx-svc.default.svc.west.cluster.local
  3. "<!DOCTYPE html>
  4. <html>
  5. <head>
  6. <title>Welcome to nginx!</title>
  7. <style>
  8.     body {
  9.         width: 35em;
  10.         margin: 0 auto;
  11.         font-family: Tahoma, Verdana, Arial, sans-serif;
  12.     }
  13. </style>
  14. </head>
  15. <body>
  16. <h1>Welcome to nginx!</h1>
  17. <p>If you see this page, the nginx web server is successfully installed and
  18. working. Further configuration is required.</p>
  20. <p>For online documentation and support please refer to
  21. <a href="http://nginx.org/">nginx.org</a>.<br/>
  22. Commercial support is available at
  23. <a href="http://nginx.com/">nginx.com</a>.</p>
  25. <p><em>Thank you for using nginx.</em></p>
  26. </body>
  27. </html>"

Now, let’s do the same, but this time from the east cluster. We will first export the service.

  1. $ kubectl --context=k3d-west label service nginx-svc mirror.linkerd.io/exported="true"
  2. service/nginx-svc labeled
  4. $ kubectl --context=k3d-east get services
  5. NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
  6. kubernetes         ClusterIP   10.43.0.1       <none>        443/TCP   20h
  7. nginx-svc-west     ClusterIP   None            <none>        80/TCP    29s
  8. nginx-set-0-west   ClusterIP   10.43.179.60    <none>        80/TCP    29s
  9. nginx-set-1-west   ClusterIP   10.43.218.18    <none>        80/TCP    29s
  10. nginx-set-2-west   ClusterIP   10.43.245.244   <none>        80/TCP    29s

If we take a look at the endpoints object, we will notice something odd, the endpoints for nginx-svc-west will have the same hostnames, but each hostname will point to one of the services we see above:

  1. $ kubectl --context=k3d-east get endpoints nginx-svc-west -o yaml
  2. subsets:
  3. - addresses:
  4.   - hostname: nginx-set-0
  5.     ip: 10.43.179.60
  6.   - hostname: nginx-set-1
  7.     ip: 10.43.218.18
  8.   - hostname: nginx-set-2
  9.     ip: 10.43.245.244

This is what we outlined at the start of the tutorial. Each pod from the target cluster (west), will be mirrored as a clusterIP service. We will see in a second why this matters.

  1. $ kubectl --context=k3d-east get pods
  2. NAME                    READY   STATUS    RESTARTS   AGE
  3. curl-56dc7d945d-96r6p   2/2     Running   0          23m
  5. # exec and curl
  6. $ kubectl --context=k3d-east exec pod curl-56dc7d945d-96r6p -it -c curl -- bin/sh
  7. # we want to curl the same hostname we see in the endpoints object above.
  8. # however, the service and cluster domain will now be different, since we
  9. # are in a different cluster.
  10. #
  11. / $ curl nginx-set-0.nginx-svc-west.default.svc.east.cluster.local
  12. <!DOCTYPE html>
  13. <html>
  14. <head>
  15. <title>Welcome to nginx!</title>
  16. <style>
  17.     body {
  18.         width: 35em;
  19.         margin: 0 auto;
  20.         font-family: Tahoma, Verdana, Arial, sans-serif;
  21.     }
  22. </style>
  23. </head>
  24. <body>
  25. <h1>Welcome to nginx!</h1>
  26. <p>If you see this page, the nginx web server is successfully installed and
  27. working. Further configuration is required.</p>
  29. <p>For online documentation and support please refer to
  30. <a href="http://nginx.org/">nginx.org</a>.<br/>
  31. Commercial support is available at
  32. <a href="http://nginx.com/">nginx.com</a>.</p>
  34. <p><em>Thank you for using nginx.</em></p>
  35. </body>
  36. </html>

As you can see, we get the same response back! But, nginx is in a different cluster. So, what happened behind the scenes?

  1. When we mirrored the headless service, we created a clusterIP service for each pod. Since services create DNS records, naming each endpoint with the hostname from the target gave us these pod FQDNs (nginx-set-0.(...).cluster.local).
  2. Curl resolved the pod DNS name to an IP address. In our case, this IP would be 10.43.179.60.
  3. Once the request is in-flight, the linkerd2-proxy intercepts it. It looks at the IP address and associates it with our clusterIP service. The service itself points to the gateway, so the proxy forwards the request to the target cluster gateway. This is the usual multi-cluster scenario.
  4. The gateway in the target cluster looks at the request and looks-up the original destination address. In our case, since this is an “endpoint mirror”, it knows it has to go to nginx-set-0.nginx-svc in the same cluster.
  5. The request is again forwarded by the gateway to the pod, and the response comes back.

And that’s it! You can now send requests to pods across clusters. Querying any of the 3 StatefulSet pods should have the same results.

Note

To mirror a headless service as headless, the service’s endpoints must also have at least one named address (e.g a hostname for an IP), otherwise, there will be no endpoints to mirror so the service will be mirrored as clusterIP. A headless service may under normal conditions also be created without exposing a port; the mulit-cluster service-mirror does not support this, however, since the lack of ports means we cannot create a service that passes Kubernetes validation.

Cleanup

To clean-up, you can remove both clusters entirely using the k3d CLI:

  1. $ k3d cluster delete east
  2. cluster east deleted
  3. $ k3d cluster delete west
  4. cluster west deleted