Deploying Antrea on AKS and AKS Engine

This document describes steps to deploy Antrea to an AKS cluster or an AKS Engine cluster.

Deploy Antrea to an AKS cluster

Antrea can be deployed to an AKS cluster either in networkPolicyOnly mode or in encap mode.

In networkPolicyOnly mode, Antrea enforces NetworkPolicies and implements other services for the AKS cluster, while the Azure CNI takes care of Pod IPAM and traffic routing across Nodes. For more information about networkPolicyOnly mode, refer to this design document.

In encap mode, Antrea is in charge of Pod IPAM and of all the networking functions on the Nodes. Using encap mode provides access to additional Antrea features, such as Multicast, as inter-Node Pod traffic is encapsulated, and is not handled directly by the Azure Virtual Network. Note that the caveats which apply when deploying Antrea in encap mode on EKS do not apply for AKS.

We recommend encap mode, as it will give you access to the most Antrea features.

AKS Prerequisites

Install the Azure Cloud CLI. Refer to Azure CLI installation guide

We recommend using the latest version available (use at least version 2.39.0).

Deploying Antrea in networkPolicyOnly mode

Creating the cluster

You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI.

  1. Create an AKS cluster

    1. export RESOURCE_GROUP_NAME=aks-antrea-cluster
    2. export CLUSTER_NAME=aks-antrea-cluster
    3. export LOCATION=westus
    5. az group create --name $RESOURCE_GROUP_NAME --location $LOCATION
    6. az aks create \
    7.     --resource-group $RESOURCE_GROUP_NAME \
    8.     --name $CLUSTER_NAME \
    9.     --node-count 2 \
    10.     --network-plugin azure

    Note Do not specify network-policy option.

  2. Get AKS cluster credentials

    1. az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME

  3. Access your cluster

    1. kubectl get nodes
    2. NAME                                STATUS   ROLES   AGE     VERSION
    3. aks-nodepool1-84330359-vmss000000   Ready    agent   6m21s   v1.16.10
    4. aks-nodepool1-84330359-vmss000001   Ready    agent   6m25s   v1.16.10

Deploying Antrea

  1. Prepare the cluster Nodes

    Deploy antrea-node-init DaemonSet to enable azure cni to operate in transparent mode.

    1. kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea-aks-node-init.yml

  2. Deploy Antrea

    To deploy a released version of Antrea, pick a deployment manifest from the list of releases. Note that AKS support was added in release 0.9.0, which means you cannot pick a release older than 0.9.0. For any given release <TAG> (e.g. v0.9.0), you can deploy Antrea as follows:

    1. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea-aks.yml

    To deploy the latest version of Antrea (built from the main branch), use the checked-in deployment yaml:

    1. kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea-aks.yml

    The command will deploy a single replica of Antrea controller to the AKS cluster and deploy Antrea agent to every Node. After a successful deployment you should be able to see these Pods running in your cluster:

    1. $ kubectl get pods --namespace kube-system  -l app=antrea
    2. NAME                                 READY   STATUS    RESTARTS   AGE
    3. antrea-agent-bpj72                   2/2     Running   0          40s
    4. antrea-agent-j2sjz                   2/2     Running   0          40s
    5. antrea-controller-6f7468cbff-5sk4t   1/1     Running   0          43s
    6. antrea-node-init-6twqg               1/1     Running   0          2m
    7. antrea-node-init-mqsqr               1/1     Running   0          2m

  3. Restart remaining Pods

    Once Antrea is up and running, restart all Pods in all Namespaces (kube-system, etc) so they can be managed by Antrea.

    1. kubectl delete pods -n kube-system $(kubectl get pods -n kube-system -o custom-columns=NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>' | awk '{ print $1 }')
    2. pod "coredns-544d979687-96xm9" deleted
    3. pod "coredns-544d979687-p7dfb" deleted
    4. pod "coredns-autoscaler-78959b4578-849k8" deleted
    5. pod "dashboard-metrics-scraper-5f44bbb8b5-5qkkx" deleted
    6. pod "kube-proxy-6qxdw" deleted
    7. pod "kube-proxy-h6d89" deleted
    8. pod "kubernetes-dashboard-785654f667-7twsm" deleted
    9. pod "metrics-server-85c57978c6-pwzcx" deleted
    10. pod "tunnelfront-649ff5fb55-5lxg7" deleted

Deploying Antrea in encap mode

AKS now officially supports Bring your own Container Network Interface (BYOCNI). Thanks to this, you can deploy Antrea on AKS in encap mode, and you will not lose access to any functionality. Check the AKS BYOCNI documentation for prerequisites, in particular for AKS version requirements.

Creating the cluster

You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI.

  1. Create an AKS cluster

    1. export RESOURCE_GROUP_NAME=aks-antrea-cluster
    2. export CLUSTER_NAME=aks-antrea-cluster
    3. export LOCATION=westus
    5. az group create --name $RESOURCE_GROUP_NAME --location $LOCATION
    6. az aks create \
    7.     --resource-group $RESOURCE_GROUP_NAME \
    8.     --name $CLUSTER_NAME \
    9.     --node-count 2 \
    10.     --network-plugin none

    Notice --network-plugin none, which tells AKS not to install any CNI plugin.

  2. Get AKS cluster credentials

    1. az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME

  3. Access your cluster

    1. kubectl get nodes
    2. NAME                                STATUS     ROLES   AGE   VERSION
    3. aks-nodepool1-40948307-vmss000000   NotReady   agent   18m   v1.27.7
    4. aks-nodepool1-40948307-vmss000001   NotReady   agent   17m   v1.27.7

    The Nodes are supposed to report a NotReady Status, since no CNI plugin is installed yet.

Deploying Antrea

You can use Helm to easily install Antrea (or any other supported installation method). Just make sure that you configure Antrea NodeIPAM:

  1. # you may not need this:
  2. helm repo add antrea https://charts.antrea.io
  3. helm repo update
  5. cat <<EOF >> values-aks.yml
  6. nodeIPAM:
  7.   enable: true
  8.   clusterCIDRs: ["10.10.0.0/16"]
  9. EOF
  11. helm install -n kube-system -f values-aks.yml antrea antrea/antrea

For more information about how to configure Antrea Node IPAM, please refer to Antrea Node IPAM guide.

After a while, make sure that all your Nodes report a Ready Status and that all your Pods are running correctly. Some Pods, and in particular the metrics-server Pods, may restart once after installing Antrea; this is not an issue.

After a successful installation, Pods should look like this:

  1. NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
  2. kube-system   antrea-agent-bpskv                    2/2     Running   0          7m34s
  3. kube-system   antrea-agent-pfqrn                    2/2     Running   0          7m34s
  4. kube-system   antrea-controller-555b8c799d-wk8zz    1/1     Running   0          7m34s
  5. kube-system   cloud-node-manager-2nszz              1/1     Running   0          31m
  6. kube-system   cloud-node-manager-wj68q              1/1     Running   0          31m
  7. kube-system   coredns-789789675-2nwd7               1/1     Running   0          6m48s
  8. kube-system   coredns-789789675-lbkfn               1/1     Running   0          31m
  9. kube-system   coredns-autoscaler-649b947bbd-j5wqc   1/1     Running   0          31m
  10. kube-system   csi-azuredisk-node-4bnnl              3/3     Running   0          31m
  11. kube-system   csi-azuredisk-node-52nwd              3/3     Running   0          31m
  12. kube-system   csi-azurefile-node-2h66l              3/3     Running   0          31m
  13. kube-system   csi-azurefile-node-dhrf2              3/3     Running   0          31m
  14. kube-system   konnectivity-agent-5fc7989878-6nhwl   1/1     Running   0          31m
  15. kube-system   konnectivity-agent-5fc7989878-t2n6h   1/1     Running   0          30m
  16. kube-system   kube-proxy-96c9p                      1/1     Running   0          31m
  17. kube-system   kube-proxy-x8g8s                      1/1     Running   0          31m
  18. kube-system   metrics-server-5955767688-2hjvn       2/2     Running   0          3m45s
  19. kube-system   metrics-server-5955767688-vmcq7       2/2     Running   0          3m45s

Deploy Antrea to an AKS Engine cluster

Antrea is an integrated CNI of AKS Engine, and can be installed in networkPolicyOnly mode or encap mode to an AKS Engine cluster as part of the AKS Engine cluster deployment. To learn basics of AKS Engine cluster deployment, please refer to AKS Engine Quickstart Guide.

Deploying Antrea in networkPolicyOnly mode

To configure Antrea to enforce NetworkPolicies for the AKS Engine cluster, "networkPolicy": "antrea" needs to be set in kubernetesConfig of the AKS Engine cluster definition (Azure CNI will be used as the networkPlugin):

  1.   "apiVersion": "vlabs",
  2.   "properties": {
  3.     "orchestratorProfile": {
  4.       "kubernetesConfig": {
  5.         "networkPolicy": "antrea"
  6.       }
  7.     }
  8.   }

You can use the deployment template examples/networkpolicy/kubernetes-antrea.json to deploy an AKS Engine cluster with Antrea in networkPolicyOnly mode:

  1. $ aks-engine deploy --dns-prefix <dns-prefix> \
  2.     --resource-group <reource-group> \
  3.     --location westus2 \
  4.     --api-model examples/networkpolicy/kubernetes-antrea.json \
  5.     --auto-suffix

Deploying Antrea in encap mode

To deploy Antrea in encap mode for an AKS Engine cluster, both "networkPlugin": "antrea" and "networkPolicy": "antrea" need to be set in kubernetesConfig of the AKS Engine cluster definition:

  1.   "apiVersion": "vlabs",
  2.   "properties": {
  3.     "orchestratorProfile": {
  4.       "kubernetesConfig": {
  5.         "networkPlugin": "antrea",
  6.         "networkPolicy": "antrea"
  7.       }
  8.     }
  9.   }

You can add "networkPlugin": "antrea" to the deployment template examples/networkpolicy/kubernetes-antrea.json, and use the template to deploy an AKS Engine cluster with Antrea in encap mode.