Antrea Multi-cluster User Guide

Table of Contents

Antrea Multi-cluster implements Multi-cluster Service API, which allows users to create multi-cluster Services that can be accessed cross clusters in a ClusterSet. Antrea Multi-cluster also supports Antrea ClusterNetworkPolicy replication. A Multi-cluster ClusterSet admin can define ClusterNetworkPolicies to be replicated across the entire ClusterSet and enforced in all member clusters. Antrea Multi-cluster was first introduced in Antrea v1.5.0, and the ClusterNetworkPolicy replication feature is supported since Antrea v1.6.0. In Antrea v1.7.0, the Multi-cluster Gateway feature was added that supports routing multi-cluster Service traffic through tunnels among clusters.

Quick Start

Please refer to the Quick Start Guide to learn how to build a ClusterSet with two clusters quickly.

Installation

Preparation

We assume an Antrea version >= v1.8.0 is used in this guide, and the Antrea version is set to an environment variable TAG. For example, the following command sets the Antrea version to v1.8.0.

  1. export TAG=v1.8.0

To use the latest version of Antrea Multi-cluster from the Antrea main branch, you can change the YAML manifest path to: https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/ when applying or downloading an Antrea YAML manifest.

Since Antrea 1.7.0, multi-cluster Services require an Antrea Multi-cluster Gateway to be set up in each member cluster to route Service traffic across clusters, and Service CIDRs (ClusterIP ranges) must not overlap among clusters. To support Multi-cluster Gateways, antrea-agent must be deployed with the Multicluster feature enabled in a member cluster. You can set the following configuration parameters in antrea-agent.conf of the Antrea deployment manifest to enable the Multicluster feature:

  1. antrea-agent.conf: |
  2. ...
  3.   featureGates:
  4. ...
  5.     Multicluster: true
  6. ...
  7.   multicluster:
  8.     enable: true
  9.     namespace: ""

At the moment, Multi-cluster Gateway only works with the Antrea encap traffic mode, and all member clusters in a ClusterSet must use the same tunnel type.

Deploy Antrea Multi-cluster Controller

A Multi-cluster ClusterSet is comprised of a single leader cluster and at least two member clusters. Antrea Multi-cluster Controller needs to be deployed in the leader and all member clusters. A cluster can serve as the leader, and meanwhile also be a member cluster of the ClusterSet. To deploy Multi-cluster Controller in a dedicated leader cluster, please refer to Deploy in a Dedicated Leader cluster. To deploy Multi-cluster Controller in a member cluster, please refer to Deploy in a Member Cluster. To deploy Multi-cluster Controller in a dual-role cluster, please refer to Deploy Leader and Member in One Cluster.

Deploy in a Dedicated Leader Cluster

  1. Run the following command to import Multi-cluster CRDs in the leader cluster:

    1. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml

  2. Install Multi-cluster Controller in the leader cluster. Since Multi-cluster Controller runs as a namespaced Deployment, you should create the Namespace first, and then apply the deployment manifest with the Namespace.

  1. kubectl create ns antrea-multicluster
  2. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml

The Multi-cluster Controller in the leader cluster will be deployed in Namespace antrea-multicluster by default. If you’d like to use another Namespace, you can change antrea-multicluster to the desired Namespace in antrea-multicluster-leader-namespaced.yml, for example:

  1. $kubectl create ns '<desired-namespace>'
  2. $curl -L https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml
  3. $sed 's/antrea-multicluster/<desired-namespace>/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f -

Deploy in a Member Cluster

You can run the following command to install Multi-cluster Controller in a member cluster. The command will run the controller in the “member” mode in the kube-system Namespace. If you want to use a different Namespace other than kube-system, you can edit antrea-multicluster-member.yml and change kube-system to the desired Namespace.

  1. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml

Deploy Leader and Member in One Cluster

We need to run two instances of Multi-cluster Controller in the dual-role cluster, one in leader mode and another in member mode.

  1. Follow the steps in section Deploy in a Dedicated Leader Cluster to deploy the leader controller and import the Multi-cluster CRDs.
  2. Follow the steps in section Deploy in a Member Cluster to deploy the member controller.

Create ClusterSet

An Antrea Multi-cluster ClusterSet should include at least one leader cluster and two member clusters. As an example, in the following sections we will create a ClusterSet test-clusterset which has two member clusters with cluster ID test-cluster-east and test-cluster-west respectively, and one leader cluster with ID test-cluster-north.

Set up Access to Leader Cluster

We first need to set up access to the leader cluster’s API server for all member clusters. We recommend creating one ServiceAccount for each member for fine-grained access control.

  1. Apply the following YAML manifest in the leader cluster to set up access for test-cluster-east:

    1. apiVersion: v1
    2. kind: ServiceAccount
    3. metadata:
    4.   name: member-east
    5.   namespace: antrea-multicluster
    6. ---
    7. apiVersion: v1
    8. kind: Secret
    9. metadata:
    10.   name: member-east-token
    11.   namespace: antrea-multicluster
    12.   annotations:
    13.     kubernetes.io/service-account.name: member-east
    14. type: kubernetes.io/service-account-token
    15. ---
    16. apiVersion: rbac.authorization.k8s.io/v1
    17. kind: RoleBinding
    18. metadata:
    19.   name: member-east
    20.   namespace: antrea-multicluster
    21. roleRef:
    22.   apiGroup: rbac.authorization.k8s.io
    23.   kind: Role
    24.   name: antrea-mc-member-cluster-role
    25. subjects:
    26.   - kind: ServiceAccount
    27.     name: member-east
    28.     namespace: antrea-multicluster

  2. Generate the token Secret manifest from the leader cluster, and create a Secret with the manifest in member cluster test-cluster-east, e.g.:

    1. # Generate the file 'member-east-token.yml' from your leader cluster
    2. kubectl get secret member-east-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:'  -e   '^kind' -e '  ca.crt' -e '  token:' -e '^type' -e '  namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' >  member-east-token.yml
    3. # Apply 'member-east-token.yml' to the member cluster.
    4. kubectl apply -f member-east-token.yml --kubeconfig=/path/to/kubeconfig-of-member-test-cluster-east

  3. Replace all east to west and repeat step 1/2 for the other member cluster test-cluster-west.

Initialize ClusterSet

In all clusters, a ClusterSet CR must be created to define the ClusterSet, and two ClusterClaim CRs must be created to claim the ClusterSet and claim the cluster is a member of the ClusterSet.

  • Create ClusterClaim and ClusterSet in the leader cluster test-cluster-north with the following YAML manifest (you can also refer to leader-clusterset-template.yml):
  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: antrea-multicluster
  6. value: test-cluster-north
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: antrea-multicluster
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: antrea-multicluster
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  • Create ClusterClaim and ClusterSet in member cluster test-cluster-east with the following YAML manifest (you can also refer to member-clusterset-template.yml):
  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: kube-system
  6. value: test-cluster-east
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: kube-system
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: kube-system
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  23.       secret: "member-east-token"
  24.       server: "https://172.18.0.1:6443"
  25.   namespace: antrea-multicluster

Note: update server: "https://172.18.0.1:6443" in the ClusterSet spec to the correct leader cluster API server address.

  • Create ClusterClaim and ClusterSet in member cluster test-cluster-west:
  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: kube-system
  6. value: test-cluster-west
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: kube-system
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: kube-system
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  23.       secret: "member-west-token"
  24.       server: "https://172.18.0.1:6443"
  25.   namespace: antrea-multicluster

Initialize ClusterSet for a Dual-role Cluster

If you want to make the leader cluster test-cluster-north also a member cluster of the ClusterSet, make sure you follow the steps in Deploy Leader and Member in One Cluster and repeat the steps in Set up Access to Leader Cluster as well (don’t forget replace all east to north when you repeat the steps).

Then create the ClusterClaim and ClusterSet CRs in cluster test-cluster-north in the kube-system Namespace (where the member Multi-cluster Controller runs):

  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: kube-system
  6. value: test-cluster-north
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: kube-system
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: kube-system
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  23.       secret: "member-north-token"
  24.       server: "https://172.18.0.1:6443"
  25.   namespace: antrea-multicluster

Multi-cluster Gateway Configuration

Multi-cluster Gateways are required to support multi-cluster Service access across member clusters. Each member cluster should have one Node be specified as its Multi-cluster Gateway. Multi-cluster Service traffic is routed among clusters through the tunnels between Gateways.

After a member cluster joins a ClusterSet, and the Multicluster feature is enabled on antrea-agent, you can select one Node of the cluster to serve as the Multi-cluster Gateway by adding an annotation: multicluster.antrea.io/gateway=true to the K8s Node. For example, you can run the following command to annotate Node node-1 as the Multi-cluster Gateway:

  1. $kubectl annotate node node-1 multicluster.antrea.io/gateway=true

Multi-cluster Controller in the member cluster will detect the Gateway Node, and create a Gateway CR with the same name as the Node. You can check it with command:

  1. $kubectl get gateway -n kube-system
  2. NAME          GATEWAY IP     INTERNAL IP    AGE
  3. node-1        10.17.27.55    10.17.27.55    10s

internalIP of the Gateway is used for the tunnels between the Gateway Node and other Nodes in the local cluster, while gatewayIP is used for the tunnels to remote Gateways of other member clusters. Multi-cluster Controller discovers the IP addresses from the K8s Node resource of the Gateway Node. It will always use InternalIP of the K8s Node as the Gateway’s internalIP. For gatewayIP, there are several possibilities:

  • By default, the K8s Node’s InternalIP is used as gatewayIP too.
  • You can choose to use the K8s Node’s ExternalIP as gatewayIP, by changing the configuration option gatewayIPPrecedence to value: external, when deploying the member Multi-cluster Controller. The configration option is defined in ConfigMap antrea-mc-controller-config-*** in antrea-multicluster-member.yml.
  • When the Gateway Node has a separate IP for external communication or is associated with a public IP (e.g. an Elastic IP on AWS), but the IP is not added to the K8s Node, you can still choose to use the IP as gatewayIP, by adding an annotation: multicluster.antrea.io/gateway-ip=<ip-address> to the K8s Node.

When selecting the Multi-cluster Gateway Node, you need to make sure the resulted gatewayIP can be reached from the remote Gateways. You may need to configure firewall or security groups properly to allow the tunnels between Gateway Nodes. As of now, only IPv4 Gateway IPs are supported.

After the Gateway is detected, Multi-cluster Controller will be responsible for exporting the cluster’s network information to other member clusters through the leader cluster, including the cluster’s Gateway IP and Service CIDR. Multi-cluster Controller will try to discover the cluster’s Service CIDR automatically, but you can also manually specify the serviceCIDR option in ConfigMap antrea-mc-controller-config-***. In other member clusters, a ClusterInfoImport CR will be created for the cluster which includes the exported network information. For example, in cluster test-cluster-west, you you can see a ClusterInfoImport CR with name test-cluster-east-clusterinfo is created for cluster test-cluster-east:

  1. $kubectl get clusterinfoimport -n kube-system
  2. NAME                            CLUSTER ID          SERVICE CIDR   AGE
  3. test-cluster-east-clusterinfo   test-cluster-east   110.96.0.0/20  10s

Make sure you repeat the same step to assign a Gateway Node in all member clusters. Once you confirm that all Gateway and ClusterInfoImport are created correctly, you can follow the Multi-cluster Service section to create multi-cluster Services and verify cross-cluster Service access.

Multi-cluster Service

After you set up a ClusterSet properly, you can create a ServiceExport CR to export a Service from one cluster to other clusters in the Clusterset, like the example below:

  1. apiVersion: multicluster.x-k8s.io/v1alpha1
  2. kind: ServiceExport
  3. metadata:
  4.   name: nginx
  5.   namespace: default

For example, once you export the default/nginx Service in member cluster test-cluster-west, it will be automatically imported in member cluster test-cluster-east. A Service and an Endpoints with name default/antrea-mc-nginx will be created in test-cluster-east, as well as a ServcieImport CR with name default/nginx. Now, Pods in test-cluster-east can access the imported Service using its ClusterIP, and the requests will be routed to the backend nginx Pods in test-cluster-west. You can check the imported Service and ServiceImport with commands:

  1. $kubectl get serviceimport antrea-mc-nginx -n default
  2. NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
  3. antrea-mc-nginx ClusterIP   10.107.57.62 <none>        443/TCP   10s
  5. $kubectl get serviceimport nginx -n default
  6. NAME      TYPE           IP                AGE
  7. nginx     ClusterSetIP   ["10.19.57.62"]   10s

As part of the Service export/import process, in the leader cluster, two ResourceExport CRs will be created in the Multi-cluster Controller Namespace, for the exported Service and Endpoints respectively, as well as two ResourceImport CRs. You can check them in the leader cluster with commands:

  1. $kubectl get resourceexport -n antrea-multicluster
  2. NAME                                         CLUSTER ID          KIND          NAMESPACE     NAME      AGE
  3. test-cluster-west-default-nginx-endpoints    test-cluster-west   Endpoints     default       nginx     30s
  4. test-cluster-west-default-nginx-service      test-cluster-west   Service       default       nginx     30s
  6. $kubectl get resourceimport -n antrea-multicluster
  7. NAME                      KIND            NAMESPACE       NAME           AGE
  8. default-nginx-endpoints   Endpoints       default         nginx          99s
  9. default-nginx-service     ServiceImport   default         nginx          99s

When there is any new change on the exported Service, the imported multi-cluster Service resources will be updated accordingly. Multiple member clusters can export the same Service (with the same name and Namespace). In this case, the imported Service in a member cluster will include endpoints from all the export clusters, and the Service requests will be load-balanced to all these clusters. Even when the client Pod’s cluster also exported the Service, the Service requests may be routed to other clusters, and the endpoints from the local cluster do not take precedence. A Service cannot have conflicted definitions in different export clusters, otherwise only the first export will be replicated to other clusters; other exports as well as new updates to the Service will be ingored, until user fixes the conflicts. For example, after a member cluster exported a Service: default/nginx with TCP Port 80, other clusters can only export the same Service with the same Ports definition including Port names. At the moment, Antrea Multi-cluster supports only IPv4 multi-cluster Services.

Multi-cluster ClusterNetworkPolicy Replication

Since Antrea v1.6.0, Multi-cluster admins can specify certain ClusterNetworkPolicies to be replicated and enforced across the entire ClusterSet. This is especially useful for ClusterSet admins who want all clusters in the ClusterSet to be applied with a consistent security posture (for example, all Namespaces in all clusters can only communicate with Pods in their own Namespaces). For more information regarding Antrea ClusterNetworkPolicy (ACNP), please refer to this document.

To achieve such ACNP replication across clusters, admins can, in the leader cluster of a ClusterSet, create a ResourceExport CR of kind AntreaClusterNetworkPolicy which contains the ClusterNetworkPolicy spec they wish to be replicated. The ResourceExport should be created in the Namespace where the ClusterSet’s leader Multi-cluster Controller runs.

  1. apiVersion: multicluster.crd.antrea.io/v1alpha1
  2. kind: ResourceExport
  3. metadata:
  4.   name: strict-namespace-isolation-for-test-clusterset
  5.   namespace: antrea-multicluster # Namespace that Multi-cluster Controller is deployed
  6. spec:
  7.   kind: AntreaClusterNetworkPolicy
  8.   name: strict-namespace-isolation # In each importing cluster, an ACNP of name antrea-mc-strict-namespace-isolation will be created with the spec below
  9.   clusternetworkpolicy:
  10.     priority: 1
  11.     tier: securityops
  12.     appliedTo:
  13.       - namespaceSelector: {} # Selects all Namespaces in the member cluster
  14.     ingress:
  15.       - action: Pass
  16.         from:
  17.           - namespaces:
  18.               match: Self # Skip drop rule for traffic from Pods in the same Namespace
  19.           - podSelector:
  20.               matchLabels:
  21.                 k8s-app: kube-dns # Skip drop rule for traffic from the core-dns components
  22.       - action: Drop
  23.         from:
  24.           - namespaceSelector: {} # Drop from Pods from all other Namespaces

The above sample spec will create an ACNP in each member cluster which implements strict Namespace isolation for that cluster.

Note that because the Tier that an ACNP refers to must exist before the ACNP is applied, an importing cluster may fail to create the ACNP to be replicated, if the Tier in the ResourceExport spec cannot be found in that particular cluster. If there are such failures, the ACNP creation status of failed member clusters will be reported back to the leader cluster as K8s Events, and can be checked by describing the ResourceImport of the original ResourceExport:

  1. $kubectl describe resourceimport -A
  2. Name:         strict-namespace-isolation-antreaclusternetworkpolicy
  3. Namespace:    antrea-multicluster
  4. API Version:  multicluster.crd.antrea.io/v1alpha1
  5. Kind:         ResourceImport
  6. Spec:
  7.   Clusternetworkpolicy:
  8.     Applied To:
  9.       Namespace Selector:
  10.     Ingress:
  11.       Action:          Pass
  12.       Enable Logging:  false
  13.       From:
  14.         Namespaces:
  15.           Match:  Self
  16.         Pod Selector:
  17.           Match Labels:
  18.             k8s-app:   kube-dns
  19.       Action:          Drop
  20.       Enable Logging:  false
  21.       From:
  22.         Namespace Selector:
  23.     Priority:  1
  24.     Tier:      random
  25.   Kind:        AntreaClusterNetworkPolicy
  26.   Name:        strict-namespace-isolation
  27.   ...
  28. Events:
  29.   Type    Reason               Age    From                       Message
  30.   ----    ------               ----   ----                       -------
  31.   Warning ACNPImportFailed     2m11s  resourceimport-controller  ACNP Tier random does not exist in the importing cluster test-cluster-west

In future releases, some additional tooling may become available to automate the creation of ResourceExports for ACNPs, and provide a user-friendly way to define Multi-cluster NetworkPolicies to be enforced in the ClusterSet.

Build Antrea Multi-cluster Image

If you’d like to build Antrea Multi-cluster Docker image locally, you can follow the following steps:

  1. Go to your local antrea source tree, run make antrea-mc-controller, and you will get a new image named antrea/antrea-mc-controller:latest locally.
  2. Run docker save antrea/antrea-mc-controller:latest > antrea-mcs.tar to save the image.
  3. Copy the image file antrea-mcs.tar to the Nodes of your local cluster.
  4. Run docker load < antrea-mcs.tar in each Node of your local cluster.

Known Issue

We recommend user to redeploy or update Antrea Multi-cluster Controller through kubectl apply. If you are using kubectl delete -f * and kubectl create -f * to redeploy Controller in the leader cluster, you might encounter a known issue in ResourceExport CRD cleanup. To avoid this issue, please delete any ResourceExport CRs in the leader cluster first, and make sure kubectl get resourceexport -A returns empty result before you can redeploy Multi-cluster Controller.

All ResourceExports can be deleted with the following command:

  1. kubectl get resourceexport -A -o json | jq -r '.items[]|[.metadata.namespace,.metadata.name]|join(" ")' | xargs -n2 bash -c 'kubectl delete -n $0 resourceexport/$1'