Apply Calico policy to services exposed externally as cluster IPs

Big picture

Control access to services exposed through clusterIPs that are advertised outside the cluster using BGP.

Value

Calico network policy uses standard Kubernetes Services that allow you to expose services within clusters to external clients in the following ways:

Concepts

Advertise cluster IPs outside the cluster

A cluster IP is a virtual IP address that represents a Kubernetes Service. Kube Proxy on each host translates the clusterIP into a pod IP for one of the pods backing the service, acting as a reverse proxy and load balancer.

Cluster IPs were originally designed for use within the Kubernetes cluster. Calico allows you to advertise Cluster IPs externally — so external clients can use them to access services hosted inside the cluster. This means that Calico ingress policy can be applied at one or both of the following locations:

  • Host interface, when the traffic destined for the clusterIP first ingresses the cluster
  • Pod interface of the backend pod

Traffic routing: local versus cluster modes

Calico implements Kubernetes service external traffic policy, which controls whether external traffic is routed to node-local or cluster-wide endpoints. The following table summarizes key differences between these settings. The default is cluster mode.

Service settingTraffic is load balanced…Pros and consRequired service type
externalTrafficPolicy: Cluster(default)Across all nodes in the clusterEqual distribution of traffic among all pods running a service.

Possible unnecessary network hops between nodes for ingress external traffic.When packets are rerouted to pods on another node, traffic is SNAT’d (source network address translation).

Destination pod can see the proxying node’s IP address rather than the actual client IP.		ClusterIP
externalTrafficPolicy: LocalAcross the nodes with the endpoints for the serviceAvoids extra hops so better for apps that ingress a lot external traffic.

Traffic is not SNAT’d so actual client IPs are preserved.

Traffic distributed among pods running a service may be imbalanced.		LoadBalancer (for cloud providers), or NodePort (for node’s static port)

Before you begin…

Configure Calico to advertise cluster IPs over BGP.

How to

Selecting which mode to use depends on your goals and resources. At an operational level, local mode simplifies policy, but load balancing may be uneven in certain scenarios. Cluster mode requires more work to manage clusterIPs, SNAT, and create policies that reference specific IP addresses, but you always get even load balancing.

Secure externally exposed cluster IPs, local mode

Using local mode, the original source address of external traffic is preserved, and you can define policy directly using standard Calico network policy.

  1. Create Calico NetworkPolicies or GlobalNetworkPolicies that select the same set of pods as your Kubernetes Service.
  2. Add rules to allow the external traffic.
  3. If desired, add rules to allow in-cluster traffic.

Secure externally exposed cluster IPs, cluster mode

In the following steps, we define GlobalNetworkPolicy and HostEndpoints.

Step 1: Verify Kubernetes Service manifest

Ensure that your Kubernetes Service manifest explicitly lists the clusterIP; do not allow Kubernetes to automatically assign the clusterIP because you need it for your policies in the following steps.

Step 2: Create global network policy at the host interface

In this step, you create a GlobalNetworkPolicy that selects all host endpoints. It controls access to the cluster IP, and prevents unauthorized clients from outside the cluster from accessing it. The hosts then forwards only authorized traffic.

Set policy to allow external traffic for cluster IPs

Add rules to allow the external traffic for each clusterIP. The following example allows connections to two cluster IPs. Make sure you add applyOnForward and preDNAT rules.

  1. apiVersion: projectcalico.org/v3
  2. kind: GlobalNetworkPolicy
  3. metadata:
  4.   name: allow-cluster-ips
  5. spec:
  6.   selector: k8s-role == 'node'
  7.   types:
  8.   - Ingress
  9.   applyOnForward: true
  10.   preDNAT: true
  11.   ingress:
  12.    # Allow 50.60.0.0/16 to access Cluster IP A
  13.   - action: Allow
  14.     source:
  15.       nets:
  16.       - 50.60.0.0/16
  17.     destination:
  18.       nets:
  19.       - 10.20.30.40/32  Cluster IP A
  20.    # Allow 70.80.90.0/24 to access Cluster IP B
  21.   - action: Allow
  22.     source:
  23.       nets:
  24.       - 70.80.90.0/24
  25.     destination:
  26.       nets:
  27.       - 10.20.30.41/32  Cluster IP B

Add a rule to allow traffic destined for the pod CIDR

Without this rule, normal pod-to-pod traffic is blocked because the policy applies to forwarded traffic.

  1. apiVersion: projectcalico.org/v3
  2. kind: GlobalNetworkPolicy
  3. metadata:
  4.   name: allow-to-pods
  5. spec:
  6.   selector: k8s-role == 'node'
  7.   types:
  8.   - Ingress
  9.   applyOnForward: true
  10.   preDNAT: true
  11.   ingress:
  12.    # Allow traffic forwarded to pods
  13.   - action: Allow
  14.     destination:
  15.       nets:
  16.       - 192.168.0.0/16  Pod CIDR

Add a rule to allow traffic destined for all host endpoints

Or, you can add rules that allow specific host traffic including Kubernetes and Calico. Without this rule, normal host traffic is blocked.

  1. apiVersion: projectcalico.org/v3
  2. kind: GlobalNetworkPolicy
  3. metadata:
  4.   name: allow-traffic-hostendpoints
  5. spec:
  6.   selector: k8s-role == 'node'
  7.   types:
  8.   - Ingress
  9.   # Allow traffic to the node (not nodePorts, TCP) (not nodePorts, TCP)
  10.   - action: Allow
  11.     protocol: TCP
  12.     destination:
  13.       selector: k8s-role == 'node'
  14.       notPorts: ["30000:32767"]  #nodePort range
  15.   # Allow traffic to the node (not nodePorts, TCP) (not nodePorts, UDP)
  16.   - action: Allow
  17.     protocol: UDP
  18.     destination:
  19.       selector: k8s-role == 'node'
  20.       notPorts: ["30000:32767"]  #nodePort range

Step 3: Create a global network policy that selects pods

In this step, you create a GlobalNetworkPolicy that selects the same set of pods as your Kubernetes Service. Add rules that allow host endpoints to access the service ports.

  1. apiVersion: projectcalico.org/v3
  2. kind: GlobalNetworkPolicy
  3. metadata:
  4.   name: allow-nodes-svc-a
  5. spec:
  6.   selector: k8s-svc == 'svc-a'
  7.   types:
  8.     - Ingress
  9.   ingress:
  10.     - action: Allow
  11.       protocol: TCP
  12.       source:
  13.         selector: k8s-role == 'node'
  14.       destination:
  15.         ports: [80, 443]
  16.     - action: Allow
  17.       protocol: UDP
  18.       source:
  19.         selector: k8s-role == 'node'
  20.       destination:
  21.         ports: [80, 443]

Step 4: (Optional) Create network polices or global network policies that allow in-cluster traffic to access the service

Step 5: Create HostEndpoints

Create HostEndpoints for the interface of each host that will receive traffic for the clusterIPs. Be sure to label them so they are selected by the policy in Step 2 (Add a rule to allow traffic destined for the pod CIDR), and the rules in Step 3.

In the previous example policies, the label k8s-role: node is used to identify these HostEndpoints.

Additional resources