Masquerading

IPv4 addresses used for pods are typically allocated from RFC1918 private address blocks and thus, not publicly routable. Cilium will automatically masquerade the source IP address of all traffic that is leaving the cluster to the IPv4 address of the node as the node’s IP address is already routable on the network.

../../../_images/masquerade.png

This behavior can be disabled with the option masquerade: false in which case no masquerading will be performed.

Configuration

Setting the routable CIDR

The default behavior is to exclude any destination within the IP allocation CIDR of the local node. If the pod IPs are routable across a wider network, that network can be specified with the option: native-routing-cidr: 10.0.0.0/8 in which case all destinations within that CIDR will not be masqueraded.

Setting the masquerading interface

See Implementation Modes for configuring the masquerading interfaces.

Implementation Modes

eBPF-based

The eBPF-based implementation is the most efficient implementation. It requires Linux kernel 4.19 and can be enabled with the bpf.masquerade=true helm option (enabled by default).

The current implementation depends on the BPF NodePort feature. The dependency will be removed in the future (GH-13732).

Masquerading can take place only on those devices which run the eBPF masquerading program. This means that a packet sent from a pod to an outside will be masqueraded (to an output device IPv4 address), if the output device runs the program. If not specified, the program will be automatically attached to the devices selected by the BPF NodePort device detection metchanism. To manually change this, use the devices helm option. Use cilium status to determine which devices the program is running on:

  1. kubectl exec -it -n kube-system cilium-xxxxx -- cilium status | grep Masquerading
  2. Masquerading:   BPF (ip-masq-agent)   [eth0, eth1]  10.0.0.0/16

From the output above, the program is running on the eth0 and eth1 devices.

The eBPF-based masquerading can masquerade packets of the following IPv4 L4 protocols:

  • TCP
  • UDP
  • ICMP (only Echo request and Echo reply)

By default, any packet from a pod destined to an IP address outside of the native-routing-cidr range is masqueraded. The exclusion CIDR is shown in the above output of cilium status (10.0.0.0.16). To allow more fine-grained control, Cilium implements ip-masq-agent in eBPF which can be enabled with the ipMasqAgent.enabled=true helm option.

The eBPF-based ip-masq-agent supports the nonMasqueradeCIDRs and masqLinkLocal options set in a configuration file. A packet sent from a pod to a destination which belongs to any CIDR from the nonMasqueradeCIDRs is not going to be masqueraded. If the configuration file is empty, the agent will provision the following non-masquerade CIDRs:

  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16
  • 100.64.0.0/10
  • 192.0.0.0/24
  • 192.0.2.0/24
  • 192.88.99.0/24
  • 198.18.0.0/15
  • 198.51.100.0/24
  • 203.0.113.0/24
  • 240.0.0.0/4

In addition, if the masqLinkLocal is not set or set to false, then 169.254.0.0/16 is appended to the non-masquerade CIDRs list.

The agent uses Fsnotify to track updates to the configuration file, so the original resyncInterval option is unnecessary.

The example below shows how to configure the agent via ConfigMap and to verify it:

  1. cat agent-config/config
  2. nonMasqueradeCIDRs:
  3. - 10.0.0.0/8
  4. - 172.16.0.0/12
  5. - 192.168.0.0/16
  6. masqLinkLocal: false
  8. kubectl create configmap ip-masq-agent --from-file=agent-config --namespace=kube-system
  10. # Wait ~60s until the ConfigMap is mounted into a cilium pod
  12. kubectl -n kube-system exec -ti cilium-xxxxx -- cilium bpf ipmasq list
  13. IP PREFIX/ADDRESS
  14. 10.0.0.0/8
  15. 169.254.0.0/16
  16. 172.16.0.0/12
  17. 192.168.0.0/16

iptables-based

This is the legacy implementation that will work on all kernel versions.

The default behavior will masquerade all traffic leaving on a non-Cilium network device. This typically leads to the correct behavior. In order to limit the network interface on which masquerading should be performed, the option egress-masquerade-interfaces: eth0 can be used.

Note

It is possible to specify an interface prefix as well, by specifying eth+, all interfaces matching the prefix eth will be used for masquerading.