NodePortLocal (NPL)

Table of Contents

What is NodePortLocal?

NodePortLocal (NPL) is a feature that runs as part of the Antrea Agent, through which each port of a Service backend Pod can be reached from the external network using a port of the Node on which the Pod is running. NPL enables better integration with external Load Balancers which can take advantage of the feature: instead of relying on NodePort Services implemented by kube-proxy, external Load-Balancers can consume NPL port mappings published by the Antrea Agent (as K8s Pod annotations) and load-balance Service traffic directly to backend Pods.

Prerequisites

NodePortLocal was introduced in v0.13 as an alpha feature, and was graduated to beta in v1.4, at which time it was enabled by default. Prior to v1.4, a feature gate, NodePortLocal, must be enabled on the antrea-agent for the feature to work. Starting from Antrea v1.7, NPL is supported on the Windows antrea-agent.

Usage

In addition to enabling the NodePortLocal feature gate (if needed), you need to ensure that the nodePortLocal.enable flag is set to true in the Antrea Agent configuration. The nodePortLocal.portRange parameter can also be set to change the range from which Node ports will be allocated. Otherwise, the default range of 61000-62000 will be used by default. When using the NodePortLocal feature, your antrea-agent ConfigMap should look like this:

  1. kind: ConfigMap
  2. apiVersion: v1
  3. metadata:
  4.   name: antrea-config
  5.   namespace: kube-system
  6. data:
  7.   antrea-agent.conf: |
  8.     featureGates:
  9.       # True by default starting with Antrea v1.4
  10.       # NodePortLocal: true
  11.     nodePortLocal:
  12.       enable: true
  13.       # Uncomment if you need to change the port range.
  14.       # portRange: 61000-62000

Pods can be selected for NodePortLocal by tagging a Service with annotation: nodeportlocal.antrea.io/enabled: "true". Consequently, NodePortLocal is enabled for all the Pods which are selected by the Service through a selector, and the ports of these Pods will be reachable through Node ports allocated from the port range. The selected Pods will be annotated with the details about allocated Node port(s) for the Pod.

For example, given the following Service and Deployment definitions:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: nginx
  5.   annotations:
  6.     nodeportlocal.antrea.io/enabled: "true"
  7. spec:
  8.   ports:
  9.   - name: web
  10.     port: 80
  11.     protocol: TCP
  12.     targetPort: 80
  13.   selector:
  14.     app: nginx
  15. ---
  16. apiVersion: apps/v1
  17. kind: Deployment
  18. metadata:
  19.   name: nginx
  20. spec:
  21.   selector:
  22.     matchLabels:
  23.       app: nginx
  24.   replicas: 3
  25.   template:
  26.     metadata:
  27.       labels:
  28.         app: nginx
  29.     spec:
  30.       containers:
  31.       - name: nginx
  32.         image: nginx

If the NodePortLocal feature gate is enabled, then all the Pods in the Deployment will be annotated with the nodeportlocal.antrea.io annotation. The value of this annotation is a serialized JSON array. In our example, a given Pod in the nginx Deployment may look like this:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: nginx-6799fc88d8-9rx8z
  5.   labels:
  6.     app: nginx
  7.   annotations:
  8.     nodeportlocal.antrea.io: '[{"podPort":8080,"nodeIP":"10.10.10.10","nodePort":61002,"protocol":"tcp","protocols":["tcp"]}]'

This annotation indicates that port 8080 of the Pod can be reached through port 61002 of the Node with IP Address 10.10.10.10 for TCP traffic.

The nodeportlocal.antrea.io annotation is generated and managed by Antrea. It is not meant to be created or modified by users directly. A user-provided annotation is likely to be overwritten by Antrea, or may lead to unexpected behavior.

NodePortLocal can only be used with Services of type ClusterIP or LoadBalancer. The nodeportlocal.antrea.io annotation has no effect for Services of type NodePort or ExternalName. The annotation also has no effect for Services with an empty or missing Selector.

Starting from the Antrea v1.7 minor release, the protocols field in the annotation is deprecated. The array contains a single member, equal to the protocol field. The protocols field will be removed from Antrea for minor releases post March 2023, as per our deprecation policy.

Usage pre Antrea v1.7

Prior to the Antrea v1.7 minor release, the nodeportlocal.antrea.io annotation could contain multiple members in protocols. An example may look like this:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: nginx-6799fc88d8-9rx8z
  5.   labels:
  6.     app: nginx
  7.   annotations:
  8.     nodeportlocal.antrea.io: '[{"podPort":8080,"nodeIP":"10.10.10.10","nodePort":61002}, "protocols":["tcp","udp"]]'

This annotation indicates that port 8080 of the Pod can be reached through port 61002 of the Node with IP Address 10.10.10.10 for both TCP and UDP traffic.

Prior to v1.7, the implementation would always allocate the same nodePort value for all the protocols exposed for a given podPort. Starting with v1.7, there will be multiple annotations for the different protocols for a given podPort, and the allocated nodePort may be different for each one.

Usage pre Antrea v1.4

Prior to the Antrea v1.4 minor release, the nodePortLocal option group in the Antrea Agent configuration did not exist. To enable the NodePortLocal feature, one simply needed to enable the feature gate, and the port range could be configured using the (now deprecated) nplPortRange parameter.

Usage pre Antrea v1.2

Prior to the Antrea v1.2 minor release, the NodePortLocal feature suffered from a known issue. In order to use the feature, the correct list of ports exposed by each container had to be provided in the Pod specification (.spec.containers[*].Ports). The NodePortLocal implementation would then use this information to decide which ports to map for each Pod. In the above example, the Deployment definition would need to be changed to:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx
  5. spec:
  6.   selector:
  7.     matchLabels:
  8.       app: nginx
  9.   replicas: 3
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx
  17.         image: nginx
  18.         ports:
  19.         - containerPort: 80

This was error-prone because providing this list of ports is typically optional in K8s and omitting it does not prevent ports from being exposed, which means that many user may omit this information and expect NPL to work. Starting with Antrea v1.2, we instead rely on the service.spec.ports[*].targetPort information, for each NPL-enabled Service, to determine which ports need to be mapped.

Limitations

This feature is currently only supported for Nodes running Linux or Windows with IPv4 addresses. Only TCP & UDP Service ports are supported (not SCTP).

Integrations with External Load Balancers

AVI

When using AVI and the AVI Kubernetes Operator (AKO), the AKO serviceType configuration parameter can be set to NodePortLocal. After that, annotating Services manually with nodeportlocal.antrea.io is no longer required. AKO will automatically annotate Services of type LoadBalancer, along with backend ClusterIP Services used by Ingress resources (for which AVI is the Ingress class). For more information refer to the AKO documentation.