Multi-zone deployment

Multi-zone deployment

About

Kuma supports running your service mesh in multiple zones. It is even possible to run with a mix of Kubernetes and Universal zones. Your mesh environment can include multiple isolated service meshes (multi-tenancy), and workloads running in different regions, on different clouds, or in different datacenters. A zone can be a Kubernetes cluster, a VPC, or any other deployment you need to include in the same distributed mesh environment.

How it works

Kuma manages service connectivity – establishing and maintaining connections across zones in the mesh – with the zone ingress and with a DNS resolver.

The DNS resolver is embedded in each data plane proxy and configured through XDS. It resolves each service address to a virtual IP address for all service-to-service communication.

The global control plane and the zone control planes communicate to synchronize resources such as Kuma policy configurations over Kuma Discovery Service (KDS), which is a protocol based on xDS.

A zone ingress is not an API gateway. Instead, it is specific to internal cross-zone communication within the mesh. API gateways are supported in Kuma gateway mode which can be deployed in addition to zone ingresses.

Components of a multi-zone deployment

A multi-zone deployment includes:

The global control plane:
- Accept connections only from zone control planes.
- Accept creation and changes to policies that will be applied to the data plane proxies.
- Send policies down to zone control planes.
- Send zone ingresses down to zone control plane.
- Keep an inventory of all data plane proxies running in all zones (this is only done for observability but is not required for operations).
- Reject connections from data plane proxies.
The zone control planes:
- Accept connections from data plane proxies started within this zone.
- Receive policy updates from the global control plane.
- Send data plane proxies and zone ingress changes to the global control plane.
- Compute and send configurations using XDS to the local data plane proxies.
- Update list of services which exist in the zone in the zone ingress.
- Reject policy changes that do not come from global.
The data plane proxies:
- Connect to the local zone control plane.
- Receive configurations using XDS from the local zone control plane.
- Connect to other local data plane proxies.
- Connect to zone ingresses for sending cross zone traffic.
- Receive traffic from local data plane proxies and local zone ingresses.
The zone ingress:
- Receive XDS configuration from the local zone control plane.
- Proxy traffic from other zone data plane proxies to local data plane proxies.
(optional) The zone egress:
- Receive XDS configuration from the local zone control plane.
- Proxy traffic from local data plane proxies:
  - to zone ingress proxies from other zones;
  - to external services from local zone;

Usage

To set up a multi-zone deployment we will need to:

Set up the global control plane
Set up the zone control planes
Verify control plane connectivity
Set up cross-zone communication between data plane proxies

Set up the global control plane

The global control plane must run on a dedicated cluster, and cannot be assigned to a zone.

Kubernetes
Helm
Universal

The global control plane on Kubernetes must reside on its own Kubernetes cluster, to keep its resources separate from the resources the zone control planes create during synchronization.

Run:

kumactl install control-plane --mode=global | kubectl apply -f -

Find the external IP and port of the global-remote-sync service in the kuma-system namespace:

kubectl get services -n kuma-system
NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                  AGE
kuma-system   global-remote-sync     LoadBalancer   10.105.9.10     35.226.196.103   5685:30685/TCP                                                           89s
kuma-system   kuma-control-plane     ClusterIP      10.105.12.133   <none>           5681/TCP,443/TCP,5676/TCP,5677/TCP,5678/TCP,5679/TCP,5682/TCP,5653/UDP   90s

In this example the value is 35.226.196.103:5685. You pass this as the value of <global-kds-address> when you set up the zone control planes.

Set the controlPlane.mode value to global in the chart (values.yaml), then install. On the command line, run:
```
helm install kuma --namespace kuma-system --set controlPlane.mode=global kuma/kuma
```
Or you can edit the chart and pass the file to the helm install kuma command. To get the default values, run:
```
helm show values kuma/kuma
```

Find the external IP and port of the global-remote-sync service in the kuma-system namespace:

kubectl get services -n kuma-system
NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                  AGE
kuma-system   global-remote-sync     LoadBalancer   10.105.9.10     35.226.196.103   5685:30685/TCP                                                           89s
kuma-system   kuma-control-plane     ClusterIP      10.105.12.133   <none>           5681/TCP,443/TCP,5676/TCP,5677/TCP,5678/TCP,5679/TCP,5682/TCP,5653/UDP   90s

By default, it’s exposed on port 5685. In this example the value is 35.226.196.103:5685. You pass this as the value of <global-kds-address> when you set up the zone control planes.

Set up the global control plane, and add the global environment variable:
```
KUMA_MODE=global kuma-cp run
```