Multi-Cluster Federation Overview

In Consul, federation is the act of joining two or more Consul datacenters. When datacenters are joined, Consul servers in each datacenter can communicate with one another. This enables the following features:

  • Services on all clusters can make calls to each other through Consul Service Mesh.
  • Intentions can be used to enforce rules about which services can communicate across all clusters.
  • L7 Routing Rules can enable multi-cluster failover and traffic splitting.
  • The Consul UI has a drop-down menu that lets you navigate between datacenters.

Traditional WAN Federation vs. WAN Federation Via Mesh Gateways

Consul provides two mechanisms for WAN (Wide Area Network) federation:

  1. Traditional WAN Federation
  2. WAN Federation Via Mesh Gateways (newly available in Consul 1.8.0)

Traditional WAN Federation

With traditional WAN federation, all Consul servers must be exposed on the wide area network. In the Kubernetes context this is often difficult to set up. It would require that each Consul server pod is running on a Kubernetes node with an IP address that is routable from all other Kubernetes clusters. Often Kubernetes clusters are deployed into private subnets that other clusters cannot route to without additional network devices and configuration.

The Kubernetes solution to the problem of exposing pods is load balancer services but these can’t be used with traditional WAN federation because it requires proxying both UDP and TCP and Kubernetes load balancers only proxy TCP. In addition, each Consul server would need its own load balancer because each server needs a unique address. This would increase cost and complexity.

Traditional WAN Federation

WAN Federation Via Mesh Gateways

To solve the problems that occurred with traditional WAN federation, Consul 1.8.0 now supports WAN federation via mesh gateways. This mechanism only requires that mesh gateways are exposed with routable addresses, not Consul servers. We can front the mesh gateway pods with a single Kubernetes service and all traffic flows between datacenters through the mesh gateways.

WAN Federation Via Mesh Gateway

Network Requirements

Clusters/datacenters can be federated even if they have overlapping pod IP spaces or if they’re on different cloud providers or platforms. Kubernetes clusters can even be federated with Consul datacenters running on virtual machines (and vice versa). Because the communication between clusters is end-to-end encrypted, mesh gateways can even be exposed on the public internet.

There are three networking requirements:

  1. When Consul servers in secondary datacenters first start up, they must be able to make calls directly to the primary datacenter’s mesh gateways.

  2. Once the Consul servers in secondary datacenters have made that initial call to the primary datacenter’s mesh gateways, the mesh gateways in the secondary datacenter will be able to start. From this point onwards, all communication between servers will flow first to the local mesh gateways, and then to the remote mesh gateways. This means all mesh gateways across datacenters must be able to route to one another.

    For example, if using a load balancer service in front of each cluster’s mesh gateway pods, the load balancer IP must be routable from the other mesh gateway pods. If using a public load balancer, this is guaranteed. If using a private load balancer then you’ll need to make sure that its IP/DNS address is routable from your other clusters.

  3. If ACLs are enabled, primary clusters must be able to make requests to the Kubernetes API URLs of secondary clusters.

Next Steps

Now that you have an overview of federation, proceed to either the Federation Between Kubernetes Clusters or Federation Between VMs and Kubernetes pages depending on your use case.