Antrea Multi-cluster User Guide

Table of Contents

Antrea Multi-cluster implements Multi-cluster Service API, which allows users to create multi-cluster Services that can be accessed cross clusters in a ClusterSet. Antrea Multi-cluster also extends Antrea native NetworkPolicy to support Multi-cluster NetworkPolicy rules that apply to cross-cluster traffic, and ClusterNetworkPolicy replication that allows a ClusterSet admin to create ClusterNetworkPolicies which are replicated across the entire ClusterSet and enforced in all member clusters. Antrea Multi-cluster was first introduced in Antrea v1.5.0. In Antrea v1.7.0, the Multi-cluster Gateway feature was added that supports routing multi-cluster Service traffic through tunnels among clusters. The ClusterNetworkPolicy replication feature is supported since Antrea v1.6.0, and Multi-cluster NetworkPolicy rules are supported since Antrea v1.10.0.

Quick Start

Please refer to the Quick Start Guide to learn how to build a ClusterSet with two clusters quickly.

Installation

In this guide, all Multi-cluster installation and ClusterSet configuration are done by applying Antrea Multi-cluster YAML manifests. Actually, all operations can also be done with antctl Multi-cluster commands, which may be more convenient in many cases. You can refer to the Quick Start Guide and antctl Guide to learn how to use the Multi-cluster commands.

Preparation

We assume an Antrea version >= v1.8.0 is used in this guide, and the Antrea version is set to an environment variable TAG. For example, the following command sets the Antrea version to v1.8.0.

  1. export TAG=v1.8.0

To use the latest version of Antrea Multi-cluster from the Antrea main branch, you can change the YAML manifest path to: https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/ when applying or downloading an Antrea YAML manifest.

Multi-cluster Services and multi-cluster Pod-to-Pod connectivity, in particular configuration (please check the corresponding sections to learn more information), requires an Antrea Multi-cluster Gateway to be set up in each member cluster by default to route Service and Pod traffic across clusters. To support Multi-cluster Gateways, antrea-agent must be deployed with the Multicluster feature enabled in a member cluster. You can set the following configuration parameters in antrea-agent.conf of the Antrea deployment manifest to enable the Multicluster feature:

  1. antrea-agent.conf: |
  2. ...
  3.   featureGates:
  4. ...
  5.     Multicluster: true
  6. ...
  7.   multicluster:
  8.     enableGateway: true
  9.     namespace: "" # Change to the Namespace where antrea-mc-controller is deployed.

In order for Multi-cluster features to work, it is necessary for enableGateway to be set to true by the user, except when Pod-to-Pod direct connectivity already exists (e.g., provided by the cloud provider) and endpointIPType is configured as PodIP. Details can be found in Multi-cluster Services. Please note that Multi-cluster NetworkPolicy always requires Gateway.

Prior to Antrea v1.11.0, Multi-cluster Gateway only works with Antrea encap traffic mode, and all member clusters in a ClusterSet must use the same tunnel type. Since Antrea v1.11.0, Multi-cluster Gateway also works with the Antrea noEncap, hybrid and networkPolicyOnly modes. For noEncap and hybrid modes, Antrea Multi-cluster deployment is the same as encap mode. For networkPolicyOnly mode, we need extra Antrea configuration changes to support Multi-cluster Gateway. Please check the deployment guide for more information.

Deploy Antrea Multi-cluster Controller

A Multi-cluster ClusterSet is comprised of a single leader cluster and at least two member clusters. Antrea Multi-cluster Controller needs to be deployed in the leader and all member clusters. A cluster can serve as the leader, and meanwhile also be a member cluster of the ClusterSet. To deploy Multi-cluster Controller in a dedicated leader cluster, please refer to Deploy in a Dedicated Leader cluster. To deploy Multi-cluster Controller in a member cluster, please refer to Deploy in a Member Cluster. To deploy Multi-cluster Controller in a dual-role cluster, please refer to Deploy Leader and Member in One Cluster.

Deploy in a Dedicated Leader Cluster

  1. Run the following command to import Multi-cluster CRDs in the leader cluster:

    1. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml

  2. Install Multi-cluster Controller in the leader cluster. Since Multi-cluster Controller runs as a namespaced Deployment, you should create the Namespace first, and then apply the deployment manifest with the Namespace.

  1. kubectl create ns antrea-multicluster
  2. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml

The Multi-cluster Controller in the leader cluster will be deployed in Namespace antrea-multicluster by default. If you’d like to use another Namespace, you can change antrea-multicluster to the desired Namespace in antrea-multicluster-leader-namespaced.yml, for example:

  1. $kubectl create ns '<desired-namespace>'
  2. $curl -L https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml
  3. $sed 's/antrea-multicluster/<desired-namespace>/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f -

Deploy in a Member Cluster

You can run the following command to install Multi-cluster Controller in a member cluster. The command will run the controller in the “member” mode in the kube-system Namespace. If you want to use a different Namespace other than kube-system, you can edit antrea-multicluster-member.yml and change kube-system to the desired Namespace.

  1. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml

Deploy Leader and Member in One Cluster

We need to run two instances of Multi-cluster Controller in the dual-role cluster, one in leader mode and another in member mode.

  1. Follow the steps in section Deploy in a Dedicated Leader Cluster to deploy the leader controller and import the Multi-cluster CRDs.
  2. Follow the steps in section Deploy in a Member Cluster to deploy the member controller.

Create ClusterSet

An Antrea Multi-cluster ClusterSet should include at least one leader cluster and two member clusters. As an example, in the following sections we will create a ClusterSet test-clusterset which has two member clusters with cluster ID test-cluster-east and test-cluster-west respectively, and one leader cluster with ID test-cluster-north.

Set up Access to Leader Cluster

We first need to set up access to the leader cluster’s API server for all member clusters. We recommend creating one ServiceAccount for each member for fine-grained access control.

  1. Apply the following YAML manifest in the leader cluster to set up access for test-cluster-east:

    1. apiVersion: v1
    2. kind: ServiceAccount
    3. metadata:
    4.   name: member-east
    5.   namespace: antrea-multicluster
    6. ---
    7. apiVersion: v1
    8. kind: Secret
    9. metadata:
    10.   name: member-east-token
    11.   namespace: antrea-multicluster
    12.   annotations:
    13.     kubernetes.io/service-account.name: member-east
    14. type: kubernetes.io/service-account-token
    15. ---
    16. apiVersion: rbac.authorization.k8s.io/v1
    17. kind: RoleBinding
    18. metadata:
    19.   name: member-east
    20.   namespace: antrea-multicluster
    21. roleRef:
    22.   apiGroup: rbac.authorization.k8s.io
    23.   kind: Role
    24.   name: antrea-mc-member-cluster-role
    25. subjects:
    26.   - kind: ServiceAccount
    27.     name: member-east
    28.     namespace: antrea-multicluster

  2. Generate the token Secret manifest from the leader cluster, and create a Secret with the manifest in member cluster test-cluster-east, e.g.:

    1. # Generate the file 'member-east-token.yml' from your leader cluster
    2. kubectl get secret member-east-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:'  -e   '^kind' -e '  ca.crt' -e '  token:' -e '^type' -e '  namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' >  member-east-token.yml
    3. # Apply 'member-east-token.yml' to the member cluster.
    4. kubectl apply -f member-east-token.yml --kubeconfig=/path/to/kubeconfig-of-member-test-cluster-east

  3. Replace all east to west and repeat step 1/2 for the other member cluster test-cluster-west.

Initialize ClusterSet

In all clusters, a ClusterSet CR must be created to define the ClusterSet, and two ClusterClaim CRs must be created to claim the ClusterSet and claim the cluster is a member of the ClusterSet.

  • Create ClusterClaim and ClusterSet in the leader cluster test-cluster-north with the following YAML manifest (you can also refer to leader-clusterset-template.yml):
  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: antrea-multicluster
  6. value: test-cluster-north
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: antrea-multicluster
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: antrea-multicluster
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  • Create ClusterClaim and ClusterSet in member cluster test-cluster-east with the following YAML manifest (you can also refer to member-clusterset-template.yml):
  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: kube-system
  6. value: test-cluster-east
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: kube-system
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: kube-system
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  23.       secret: "member-east-token"
  24.       server: "https://172.18.0.1:6443"
  25.   namespace: antrea-multicluster

Note: update server: "https://172.18.0.1:6443" in the ClusterSet spec to the correct leader cluster API server address.

  • Create ClusterClaim and ClusterSet in member cluster test-cluster-west:
  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: kube-system
  6. value: test-cluster-west
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: kube-system
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: kube-system
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  23.       secret: "member-west-token"
  24.       server: "https://172.18.0.1:6443"
  25.   namespace: antrea-multicluster

Initialize ClusterSet for a Dual-role Cluster

If you want to make the leader cluster test-cluster-north also a member cluster of the ClusterSet, make sure you follow the steps in Deploy Leader and Member in One Cluster and repeat the steps in Set up Access to Leader Cluster as well (don’t forget replace all east to north when you repeat the steps).

Then create the ClusterClaim and ClusterSet CRs in cluster test-cluster-north in the kube-system Namespace (where the member Multi-cluster Controller runs):

  1. apiVersion: multicluster.crd.antrea.io/v1alpha2
  2. kind: ClusterClaim
  3. metadata:
  4.   name: id.k8s.io
  5.   namespace: kube-system
  6. value: test-cluster-north
  7. ---
  8. apiVersion: multicluster.crd.antrea.io/v1alpha2
  9. kind: ClusterClaim
  10. metadata:
  11.   name: clusterset.k8s.io
  12.   namespace: kube-system
  13. value: test-clusterset
  14. ---
  15. apiVersion: multicluster.crd.antrea.io/v1alpha1
  16. kind: ClusterSet
  17. metadata:
  18.   name: test-clusterset
  19.   namespace: kube-system
  20. spec:
  21.   leaders:
  22.     - clusterID: test-cluster-north
  23.       secret: "member-north-token"
  24.       server: "https://172.18.0.1:6443"
  25.   namespace: antrea-multicluster

Multi-cluster Gateway Configuration

Multi-cluster Gateways are responsible for establishing tunnels between clusters. Each member cluster should have one Node serving as its Multi-cluster Gateway. Multi-cluster Service traffic is routed among clusters through the tunnels between Gateways.

Below is a table about communication support for different configurations.

Pod-to-Pod connectivity provided by underlayGateway EnabledMC EndpointTypesCross-cluster Service/Pod communications
NoNoN/ANo
YesNoPodIPYes
NoYesPodIP/ClusterIPYes
YesYesPodIP/ClusterIPYes

After a member cluster joins a ClusterSet, and the Multicluster feature is enabled on antrea-agent, you can select a Node of the cluster to serve as the Multi-cluster Gateway by adding an annotation: multicluster.antrea.io/gateway=true to the K8s Node. For example, you can run the following command to annotate Node node-1 as the Multi-cluster Gateway:

  1. $kubectl annotate node node-1 multicluster.antrea.io/gateway=true

You can annotate multiple Nodes in a member cluster as the candidates for Multi-cluster Gateway, but only one Node will be selected as the active Gateway. Before Antrea v1.9.0, the Gateway Node is just randomly selected and will never change unless the Node or its gateway annotation is deleted. Starting with Antrea v1.9.0, Antrea Multi-cluster Controller will guarantee a “ready” Node is selected as the Gateway, and when the current Gateway Node’s status changes to not “ready”, Antrea will try selecting another “ready” Node from the candidate Nodes to be the Gateway.

Once a Gateway Node is decided, Multi-cluster Controller in the member cluster will create a Gateway CR with the same name as the Node. You can check it with command:

  1. $kubectl get gateway -n kube-system
  2. NAME          GATEWAY IP     INTERNAL IP    AGE
  3. node-1        10.17.27.55    10.17.27.55    10s

internalIP of the Gateway is used for the tunnels between the Gateway Node and other Nodes in the local cluster, while gatewayIP is used for the tunnels to remote Gateways of other member clusters. Multi-cluster Controller discovers the IP addresses from the K8s Node resource of the Gateway Node. It will always use InternalIP of the K8s Node as the Gateway’s internalIP. For gatewayIP, there are several possibilities:

  • By default, the K8s Node’s InternalIP is used as gatewayIP too.
  • You can choose to use the K8s Node’s ExternalIP as gatewayIP, by changing the configuration option gatewayIPPrecedence to value: external, when deploying the member Multi-cluster Controller. The configration option is defined in ConfigMap antrea-mc-controller-config in antrea-multicluster-member.yml.
  • When the Gateway Node has a separate IP for external communication or is associated with a public IP (e.g. an Elastic IP on AWS), but the IP is not added to the K8s Node, you can still choose to use the IP as gatewayIP, by adding an annotation: multicluster.antrea.io/gateway-ip=<ip-address> to the K8s Node.

When choosing a candidate Node for Multi-cluster Gateway, you need to make sure the resulted gatewayIP can be reached from the remote Gateways. You may need to configure firewall or security groups properly to allow the tunnels between Gateway Nodes. As of now, only IPv4 Gateway IPs are supported.

After the Gateway is created, Multi-cluster Controller will be responsible for exporting the cluster’s network information to other member clusters through the leader cluster, including the cluster’s Gateway IP and Service CIDR. Multi-cluster Controller will try to discover the cluster’s Service CIDR automatically, but you can also manually specify the serviceCIDR option in ConfigMap antrea-mc-controller-config. In other member clusters, a ClusterInfoImport CR will be created for the cluster which includes the exported network information. For example, in cluster test-cluster-west, you you can see a ClusterInfoImport CR with name test-cluster-east-clusterinfo is created for cluster test-cluster-east:

  1. $kubectl get clusterinfoimport -n kube-system
  2. NAME                            CLUSTER ID          SERVICE CIDR   AGE
  3. test-cluster-east-clusterinfo   test-cluster-east   110.96.0.0/20  10s

Make sure you repeat the same step to assign a Gateway Node in all member clusters. Once you confirm that all Gateway and ClusterInfoImport are created correctly, you can follow the Multi-cluster Service section to create multi-cluster Services and verify cross-cluster Service access.

Multi-cluster WireGuard Encryption

Since Antrea v1.12.0, Antrea Multi-cluster supports WireGuard tunnel between member clusters. If WireGuard is enabled, the WireGuard interface and routes will be created by Antrea Agent on the Gateway Node, and all cross-cluster traffic will be encrypted and forwarded to the WireGuard tunnel.

To enable the WireGuard encryption, the TrafficEncryptMode in Multi-cluster configuration should be set to wireGuard and the enableGateway field should be set to true as follows:

  1. antrea-agent.conf: |
  2.     featureGates:
  3. ...
  4.     Multicluster: true
  5. ...
  6.     multicluster:
  7.       enableGateway: true
  8.       trafficEncryptionMode: "wireGuard"
  9.       wireGuard:
  10.         port: 51821

Multi-cluster Service

After you set up a ClusterSet properly, you can create a ServiceExport CR to export a Service from one cluster to other clusters in the Clusterset, like the example below:

  1. apiVersion: multicluster.x-k8s.io/v1alpha1
  2. kind: ServiceExport
  3. metadata:
  4.   name: nginx
  5.   namespace: default

For example, once you export the default/nginx Service in member cluster test-cluster-west, it will be automatically imported in member cluster test-cluster-east. A Service and an Endpoints with name default/antrea-mc-nginx will be created in test-cluster-east, as well as a ServcieImport CR with name default/nginx. Now, Pods in test-cluster-east can access the imported Service using its ClusterIP, and the requests will be routed to the backend nginx Pods in test-cluster-west. You can check the imported Service and ServiceImport with commands:

  1. $kubectl get serviceimport antrea-mc-nginx -n default
  2. NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
  3. antrea-mc-nginx ClusterIP   10.107.57.62 <none>        443/TCP   10s
  5. $kubectl get serviceimport nginx -n default
  6. NAME      TYPE           IP                AGE
  7. nginx     ClusterSetIP   ["10.19.57.62"]   10s

As part of the Service export/import process, in the leader cluster, two ResourceExport CRs will be created in the Multi-cluster Controller Namespace, for the exported Service and Endpoints respectively, as well as two ResourceImport CRs. You can check them in the leader cluster with commands:

  1. $kubectl get resourceexport -n antrea-multicluster
  2. NAME                                         CLUSTER ID          KIND          NAMESPACE     NAME      AGE
  3. test-cluster-west-default-nginx-endpoints    test-cluster-west   Endpoints     default       nginx     30s
  4. test-cluster-west-default-nginx-service      test-cluster-west   Service       default       nginx     30s
  6. $kubectl get resourceimport -n antrea-multicluster
  7. NAME                      KIND            NAMESPACE       NAME           AGE
  8. default-nginx-endpoints   Endpoints       default         nginx          99s
  9. default-nginx-service     ServiceImport   default         nginx          99s

When there is any new change on the exported Service, the imported multi-cluster Service resources will be updated accordingly. Multiple member clusters can export the same Service (with the same name and Namespace). In this case, the imported Service in a member cluster will include endpoints from all the export clusters, and the Service requests will be load-balanced to all these clusters. Even when the client Pod’s cluster also exported the Service, the Service requests may be routed to other clusters, and the endpoints from the local cluster do not take precedence. A Service cannot have conflicted definitions in different export clusters, otherwise only the first export will be replicated to other clusters; other exports as well as new updates to the Service will be ingored, until user fixes the conflicts. For example, after a member cluster exported a Service: default/nginx with TCP Port 80, other clusters can only export the same Service with the same Ports definition including Port names. At the moment, Antrea Multi-cluster supports only IPv4 multi-cluster Services.

By default, a multi-cluster Service will use the exported Services’ ClusterIPs (the original Service ClusterIPs in the export clusters) as Endpoints. Since Antrea v1.9.0, Antrea Multi-cluster also supports using the backend Pod IPs as the multi-cluster Service endpoints. You can change the value of configuration option endpointIPType in ConfigMap antrea-mc-controller-config from ClusterIP to PodIP to use Pod IPs as endpoints. All member clusters in a ClusterSet should use the same endpoint type. Existing ServiceExports should be re-exported after changing endpointIPType. ClusterIP type requires that Service CIDRs (ClusterIP ranges) must not overlap among member clusters, and always requires Multi-cluster Gateways to be configured. PodIP type requires Pod CIDRs not to overlap among clusters, and it also requires Multi-cluster Gateways when there is no direct Pod-to-Pod connectivity across clusters. Also refer to Multi-cluster Pod-to-Pod Connectivity for more information.

Multi-cluster Pod-to-Pod Connectivity

Since Antrea v1.9.0, Multi-cluster supports routing Pod traffic across clusters through Multi-cluster Gateways. Pod IPs can be reached in all member clusters within a ClusterSet. To enable this feature, the cluster’s Pod CIDRs must be set in ConfigMap antrea-mc-controller-config of each member cluster and multicluster.enablePodToPodConnectivity must be set to true in the antrea-agent configuration. Note, Pod CIDRs must not overlap among clusters to enable cross-cluster Pod-to-Pod connectivity.

  1. apiVersion: v1
  2. data:
  3.   controller_manager_config.yaml: |
  4.     apiVersion: multicluster.crd.antrea.io/v1alpha1
  5.     kind: MultiClusterConfig
  6.     ...
  7.     podCIDRs:
  8.       - "10.10.1.1/16"    
  9. kind: ConfigMap
  10. metadata:
  11.   labels:
  12.     app: antrea
  13.   name: antrea-mc-controller-config
  14.   namespace: kube-system
  1. antrea-controller.conf: |
  2.     featureGates:
  3. ...
  4.     Multicluster: true
  5. ...
  6.   multicluster:
  7.     enablePodToPodConnectivity: true

You can edit antrea-multicluster-member.yml, or use kubectl edit to change the ConfigMap:

  1. kubectl edit configmap -n kube-system antrea-mc-controller-config

Normally, podCIDRs should be the value of kube-controller-manager’s cluster-cidr option. If it’s left empty, the Pod-to-Pod connectivity feature will not be enabled. If you use kubectl edit to edit the ConfigMap, then you need to restart the antrea-mc-controller Pod to load the latest configuration.

Multi-cluster NetworkPolicy

Antrea-native policies can be enforced on cross-cluster traffic in a ClusterSet. To enable Multi-cluster NetworkPolicy features, check the Antrea Controller and Agent ConfigMaps and make sure that enableStretchedNetworkPolicy is set to true in addition to enabling the multicluster feature gate:

  1. antrea-controller.conf: |
  2.     featureGates:
  3. ...
  4.     Multicluster: true
  5. ...
  6.   multicluster:
  7.     enableStretchedNetworkPolicy: true # required by both egress and ingres rules
  1. antrea-agent.conf: |
  2.     featureGates:
  3. ...
  4.     Multicluster: true
  5. ...
  6.   multicluster:
  7.     enableGateway: true
  8.     enableStretchedNetworkPolicy: true # required by only ingress rules
  9.     namespace: ""

Egress Rule to Multi-cluster Service

Restricting Pod egress traffic to backends of a Multi-cluster Service (which can be on the same cluster of the source Pod or on a different cluster) is supported by Antrea-native policy’s toServices feature in egress rules. To define such a policy, simply put the exported Service name and Namespace in the toServices field of an Antrea-native policy, and set scope of the toServices peer to ClusterSet:

  1. apiVersion: crd.antrea.io/v1alpha1
  2. kind: ClusterNetworkPolicy
  3. metadata:
  4.   name: acnp-drop-tenant-to-secured-mc-service
  5. spec:
  6.   priority: 1
  7.   tier: securityops
  8.   appliedTo:
  9.     - podSelector:
  10.         matchLabels:
  11.           role: tenant
  12.   egress:
  13.     - action: Drop
  14.       toServices:
  15.         - name: secured-service   # an exported Multi-cluster Service
  16.           namespace: svcNamespace
  17.           scope: ClusterSet

The scope field of toServices rules is supported since Antrea v1.10. For earlier versions of Antrea, an equivalent rule can be written by not specifying scope and providing the imported Service name instead (i.e. antrea-mc-[svcName]).

Note that the scope of policy’s appliedTo field will still be restricted to the cluster where the policy is created in. To enforce such a policy for all role=tenant Pods in the entire ClusterSet, use the ClusterNetworkPolicy Replication feature described in the later section, and set the clusterNetworkPolicy field of the ResourceExport to the acnp-drop-tenant-to-secured-mc-service spec above. Such replication should only be performed by ClusterSet admins, who have clearance of creating ClusterNetworkPolicies in all clusters of a ClusterSet.

Ingress Rule

Antrea-native policies now support selecting ingress peers in the ClusterSet scope (since v1.10.0). Policy rules can be created to enforce security postures on ingress traffic from all member clusters in a ClusterSet:

  1. apiVersion: crd.antrea.io/v1alpha1
  2. kind: ClusterNetworkPolicy
  3. metadata:
  4.   name: drop-tenant-access-to-admin-namespace
  5. spec:
  6.   appliedTo:
  7.   - namespaceSelector:
  8.       matchLabels:
  9.         role: admin
  10.   priority: 1
  11.   tier: securityops
  12.   ingress:
  13.   - action: Deny
  14.     from:
  15.     # Select all Pods in role=tenant Namespaces in the ClusterSet
  16.     - scope: ClusterSet
  17.       namespaceSelector:
  18.         matchLabels:
  19.           role: tenant
  1. apiVersion: crd.antrea.io/v1alpha1
  2. kind: AntreaNetworkPolicy
  3. metadata:
  4.   name: db-svc-allow-ingress-from-client-only
  5.   namespace: prod-us-west
  6. spec:
  7.   appliedTo:
  8.   - podSelector:
  9.       matchLabels:
  10.         app: db
  11.   priority: 1
  12.   tier: application
  13.   ingress:
  14.   - action: Allow
  15.     from:
  16.     # Select all Pods in Namespace "prod-us-west" from all clusters in the ClusterSet (if the
  17.     # Namespace exists in that cluster) whose labels match app=client
  18.     - scope: ClusterSet
  19.       podSelector:
  20.         matchLabels:
  21.           app: client
  22.   - action: Deny

As shown in the examples above, setting scope to ClusterSet expands the scope of the podSelector or namespaceSelector of an ingress peer to the entire ClusterSet that the policy is created in. Similar to egress rules, the scope of an ingress rule’s appliedTo is still restricted to the local cluster.

To use the ingress cross-cluster NetworkPolicy feature, the enableStretchedNetworkPolicy option needs to be set to true in antrea-mc-controller-config, for each antrea-mc-controller running in the ClusterSet. Refer to the previous section on how to change the ConfigMap:

  1.   controller_manager_config.yaml: |
  2.     apiVersion: multicluster.crd.antrea.io/v1alpha1
  3.     kind: MultiClusterConfig
  4.     ...
  5.     enableStretchedNetworkPolicy: true

Note that currently ingress stretched NetworkPolicy only works with the Antrea encap traffic mode.

ClusterNetworkPolicy Replication

Since Antrea v1.6.0, Multi-cluster admins can specify certain ClusterNetworkPolicies to be replicated and enforced across the entire ClusterSet. This is especially useful for ClusterSet admins who want all clusters in the ClusterSet to be applied with a consistent security posture (for example, all Namespaces in all clusters can only communicate with Pods in their own Namespaces). For more information regarding Antrea ClusterNetworkPolicy (ACNP), please refer to this document.

To achieve such ACNP replication across clusters, admins can, in the leader cluster of a ClusterSet, create a ResourceExport CR of kind AntreaClusterNetworkPolicy which contains the ClusterNetworkPolicy spec they wish to be replicated. The ResourceExport should be created in the Namespace where the ClusterSet’s leader Multi-cluster Controller runs.

  1. apiVersion: multicluster.crd.antrea.io/v1alpha1
  2. kind: ResourceExport
  3. metadata:
  4.   name: strict-namespace-isolation-for-test-clusterset
  5.   namespace: antrea-multicluster # Namespace that Multi-cluster Controller is deployed
  6. spec:
  7.   kind: AntreaClusterNetworkPolicy
  8.   name: strict-namespace-isolation # In each importing cluster, an ACNP of name antrea-mc-strict-namespace-isolation will be created with the spec below
  9.   clusterNetworkPolicy:
  10.     priority: 1
  11.     tier: securityops
  12.     appliedTo:
  13.       - namespaceSelector: {} # Selects all Namespaces in the member cluster
  14.     ingress:
  15.       - action: Pass
  16.         from:
  17.           - namespaces:
  18.               match: Self # Skip drop rule for traffic from Pods in the same Namespace
  19.           - podSelector:
  20.               matchLabels:
  21.                 k8s-app: kube-dns # Skip drop rule for traffic from the core-dns components
  22.       - action: Drop
  23.         from:
  24.           - namespaceSelector: {} # Drop from Pods from all other Namespaces

The above sample spec will create an ACNP in each member cluster which implements strict Namespace isolation for that cluster.

Note that because the Tier that an ACNP refers to must exist before the ACNP is applied, an importing cluster may fail to create the ACNP to be replicated, if the Tier in the ResourceExport spec cannot be found in that particular cluster. If there are such failures, the ACNP creation status of failed member clusters will be reported back to the leader cluster as K8s Events, and can be checked by describing the ResourceImport of the original ResourceExport:

  1. $kubectl describe resourceimport -A
  2. Name:         strict-namespace-isolation-antreaclusternetworkpolicy
  3. Namespace:    antrea-multicluster
  4. API Version:  multicluster.crd.antrea.io/v1alpha1
  5. Kind:         ResourceImport
  6. Spec:
  7.   Clusternetworkpolicy:
  8.     Applied To:
  9.       Namespace Selector:
  10.     Ingress:
  11.       Action:          Pass
  12.       Enable Logging:  false
  13.       From:
  14.         Namespaces:
  15.           Match:  Self
  16.         Pod Selector:
  17.           Match Labels:
  18.             k8s-app:   kube-dns
  19.       Action:          Drop
  20.       Enable Logging:  false
  21.       From:
  22.         Namespace Selector:
  23.     Priority:  1
  24.     Tier:      random
  25.   Kind:        AntreaClusterNetworkPolicy
  26.   Name:        strict-namespace-isolation
  27.   ...
  28. Events:
  29.   Type    Reason               Age    From                       Message
  30.   ----    ------               ----   ----                       -------
  31.   Warning ACNPImportFailed     2m11s  resourceimport-controller  ACNP Tier random does not exist in the importing cluster test-cluster-west

In future releases, some additional tooling may become available to automate the creation of ResourceExports for ACNPs, and provide a user-friendly way to define Multi-cluster NetworkPolicies to be enforced in the ClusterSet.

Build Antrea Multi-cluster Controller Image

If you’d like to build Multi-cluster Controller Docker image locally, you can follow the following steps:

  1. Go to your local antrea source tree, run make build-antrea-mc-controller, and you will get a new image named antrea/antrea-mc-controller:latest locally.
  2. Run docker save antrea/antrea-mc-controller:latest > antrea-mcs.tar to save the image.
  3. Copy the image file antrea-mcs.tar to the Nodes of your local cluster.
  4. Run docker load < antrea-mcs.tar in each Node of your local cluster.

Known Issue

We recommend user to redeploy or update Antrea Multi-cluster Controller through kubectl apply. If you are using kubectl delete -f * and kubectl create -f * to redeploy Controller in the leader cluster, you might encounter a known issue in ResourceExport CRD cleanup. To avoid this issue, please delete any ResourceExport CRs in the leader cluster first, and make sure kubectl get resourceexport -A returns empty result before you can redeploy Multi-cluster Controller.

All ResourceExports can be deleted with the following command:

  1. kubectl get resourceexport -A -o json | jq -r '.items[]|[.metadata.namespace,.metadata.name]|join(" ")' | xargs -n2 bash -c 'kubectl delete -n $0 resourceexport/$1'