Protect hosts tutorial

Imagine that the administrator of a Kubernetes cluster wants to secure it as much as possible against incoming traffic from outside the cluster. But suppose that the cluster provides various useful services that are exposed as Kubernetes NodePorts, i.e., as well-known TCP port numbers that appear to be available on any node in the cluster. The administrator does want to expose some of those NodePorts to traffic from outside.

In this example we will use pre-DNAT policy applied to the external interfaces of each cluster node:

  • to disallow incoming traffic from outside, in general

  • but then to allow incoming traffic to particular NodePorts.

We use pre-DNAT policy for these purposes, instead of normal host endpoint policy, because:

  1. We want the protection against general external traffic to apply regardless of where that traffic is destined for - for example, to a locally hosted pod, or to a pod on another node, or to a local server process running on the host itself. Pre-DNAT policy is enforced in all of those cases - as we want - whereas normal host endpoint policy is not enforced for traffic going to a local pod.

  2. We want to write this policy in terms of the advertised NodePorts, not in terms of whatever internal port numbers those may be transformed to. kube-proxy on the ingress node will use a DNAT to change a NodePort number and IP address to those of one of the pods that backs the relevant Service. Our policy therefore needs to take effect before that DNAT - and that means that it must be a pre-DNAT policy.

Protect hosts tutorial - 图1note

Note: This tutorial is intended to be used with named host endpoints, i.e. host endpoints with interfaceName set to a specific interface name. This tutorial does not work, as-is, with host endpoints with interfaceName: "*".

Here is the pre-DNAT policy that we need to disallow incoming external traffic in general:

  1. calicoctl apply -f - <<EOF
  2. apiVersion: projectcalico.org/v3
  3. kind: GlobalNetworkPolicy
  4. metadata:
  5.   name: allow-cluster-internal-ingress
  6. spec:
  7.   order: 10
  8.   preDNAT: true
  9.   applyOnForward: true
  10.   ingress:
  11.     - action: Allow
  12.       source:
  13.         nets: [10.240.0.0/16, 192.168.0.0/16]
  14.   selector: has(host-endpoint)
  15. ---
  16. apiVersion: projectcalico.org/v3
  17. kind: GlobalNetworkPolicy
  18. metadata:
  19.   name: drop-other-ingress
  20. spec:
  21.   order: 20
  22.   preDNAT: true
  23.   applyOnForward: true
  24.   ingress:
  25.     - action: Deny
  26.   selector: has(host-endpoint)
  27. EOF

Specifically, this policy allows traffic coming from IP addresses that are known to be cluster-internal, and denies traffic from any other sources. For the cluster-internal IP addresses in this example, we assume 10.240.0.0/16 for the nodes’ own IP addresses, and 192.168.0.0/16 for IP addresses that Kubernetes will assign to pods; obviously you should adjust for the CIDRs that are in use in your own cluster.

Protect hosts tutorial - 图2note

The drop-other-ingress policy has a higher order value than allow-cluster-internal-ingress, so that it applies after allow-cluster-internal-ingress. The explicit drop-other-ingress policy is needed because there is no automatic default-drop semantic for pre-DNAT policy. There is a default-drop semantic for normal host endpoint policy but—as noted above—normal host endpoint policy is not always enforced.

We also need policy to allow egress traffic through each node’s external interface. Otherwise, when we define host endpoints for those interfaces, no egress traffic will be allowed from local processes (except for traffic that is allowed by the failsafe rules. Because there is no default-deny rule for forwarded traffic, forwarded traffic will be allowed for host endpoints.

  1. calicoctl apply -f - <<EOF
  2. apiVersion: projectcalico.org/v3
  3. kind: GlobalNetworkPolicy
  4. metadata:
  5.   name: allow-outbound-external
  6. spec:
  7.   order: 10
  8.   egress:
  9.     - action: Allow
  10.   selector: has(host-endpoint)
  11. EOF

Protect hosts tutorial - 图3note

These egress rules are defined as normal host endpoint policies, not pre-DNAT, because pre-DNAT policy does not support egress rules. (Which is because pre-DNAT policies are enforced at a point in the Linux networking stack where it is not yet determined what a packet’s outgoing interface will be.) Because these are normal host endpoint policies which do not apply to forwarded traffic (applyOnForward is false), they are not enforced for traffic that is sent from a local pod. The policy above allows applications or server processes running on the nodes themselves (as opposed to in pods) to connect outbound to any destination. In case you have a use case for restricting to particular IP addresses, you can achieve that by adding a corresponding destination spec.

Now we can define a host endpoint for the outwards-facing interface of each node. The policies above all have a selector that makes them applicable to any endpoint with a host-endpoint label, so we should include that label in our definitions. For example, for eth0 on node1:

  1. calicoctl apply -f - <<EOF
  2. apiVersion: projectcalico.org/v3
  3. kind: HostEndpoint
  4. metadata:
  5.   name: node1-eth0
  6.   labels:
  7.     host-endpoint: ingress
  8. spec:
  9.   interfaceName: eth0
  10.   node: node1
  11. EOF

After defining host endpoints for each node, you should find that internal cluster communications are all still working as normal—for example, that you can successfully execute commands like calicoctl get hep and calicoctl get pol—but that it is impossible to connect into the cluster from outside (except for any failsafe rules.
For example, if the cluster includes a Kubernetes Service that is exposed as NodePort 31852, you should find, at this point, that that NodePort works from within the cluster, but not from outside.

To open a pinhole for that NodePort, for external access, you can configure a pre-DNAT policy like this:

  1. calicoctl apply -f - <<EOF
  2. apiVersion: projectcalico.org/v3
  3. kind: GlobalNetworkPolicy
  4. metadata:
  5.   name: allow-nodeport
  6. spec:
  7.   preDNAT: true
  8.   applyOnForward: true
  9.   order: 10
  10.   ingress:
  11.     - action: Allow
  12.       protocol: TCP
  13.       destination:
  14.         selector: has(host-endpoint)
  15.         ports: [31852]
  16.   selector: has(host-endpoint)
  17. EOF

If you wanted to make that NodePort accessible only through particular nodes, you could achieve that by giving those nodes a particular host-endpoint label:

  1. host-endpoint: <special-value>

and then using host-endpoint=='<special-value>' as the selector of the allow-nodeport policy, instead of has(host-endpoint).