VXLAN Tunnel Endpoint (VTEP) Integration (beta)

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

The VTEP integration allows third party VTEP devices to send and receive traffic to and from Cilium-managed pods directly using VXLAN. This allows for example external load balancers like BIG-IP to load balance traffic to Cilium-managed pods using VXLAN.

This document explains how to enable VTEP support and configure Cilium with VTEP endpoint IPs, CIDRs, and MAC addresses.

Note

This guide assumes that Cilium has been correctly installed in your Kubernetes cluster. Please see Quick Installation for more information. If unsure, run cilium status and validate that Cilium is up and running. This guide also assumes VTEP devices has been configured with VTEP endpoint IP, VTEP CIDRs, VTEP MAC addresses (VTEP MAC). The VXLAN network identifier (VNI) must be configured as VNI 2, which represents traffic from the VTEP as the world identity. See Special Identities for more details.

Warning

This feature is in beta, and currently, it is partially incompatible the L7 policy. When a pod with an egress L7 policy sends a request to VTEP devices, the VTEP redirection is bypassed. The improvement is tracked in GitHub issue 19699.

Enable VXLAN Tunnel Endpoint (VTEP) integration

This feature requires a Linux 5.2 kernel or later, and is disabled by default. When enabling the VTEP integration, you must also specify the IPs, CIDR ranges and MACs for each VTEP device as part of the configuration.

HelmConfigMap

If you installed Cilium via helm install, you may enable the VTEP support with the following command:

  1. helm upgrade cilium cilium/cilium --version 1.12.0    --namespace kube-system    --reuse-values    --set vtep.enabled="true"    --set vtep.endpoint="10.169.72.236 10.169.72.238"    --set vtep.cidr="10.1.1.0/24   10.1.2.0/24"    --set vtep.mask="255.255.255.0"    --set vtep.mac="82:36:4c:98:2e:56 82:36:4c:98:2e:58"

VTEP support can be enabled by setting the following options in the cilium-config ConfigMap:

  1. enable-vtep:   "true"
  2. vtep-endpoint: "10.169.72.236    10.169.72.238"
  3. vtep-cidr:     "10.1.1.0/24   10.1.2.0/24"
  4. vtep-mask:     "255.255.255.0"
  5. vtep-mac:      "82:36:4c:98:2e:56 82:36:4c:98:2e:58"

Restart Cilium daemonset:

  1. kubectl -n $CILIUM_NAMESPACE rollout restart ds/cilium

How to test VXLAN Tunnel Endpoint (VTEP) Integration

Start up a Linux VM with node network connectivity to Cilium node. To configure the Linux VM, you will need to be root user or run the commands below using sudo.

  1.  Test VTEP Integration
  3.  Node IP: 10.169.72.233
  4. +--------------------------+            VM IP: 10.169.72.236
  5. |                          |            +------------------+
  6. | CiliumNode               |            |  Linux VM        |
  7. |                          |            |                  |
  8. |  +---------+             |            |                  |
  9. |  | busybox |             |            |                  |
  10. |  |         |           ens192<------>ens192              |
  11. |  +--eth0---+             |            |                  |
  12. |      |                   |            +-----vxlan2-------+
  13. |      |                   |
  14. |   lxcxxx                 |
  15. |      |                   |
  16. +------+-----cilium_vxlan--+
  1. # Create a vxlan device and set the MAC address.
  2. ip link add vxlan2 type vxlan id 2 dstport 8472 local 10.169.72.236 dev ens192
  3. ip link set dev vxlan2 address 82:36:4c:98:2e:56
  4. ip link set vxlan2 up
  5. # Configure the VTEP with IP 10.1.1.236 to handle CIDR 10.1.1.0/24.
  6. ip addr add 10.1.1.236/24 dev vxlan2
  7. # Assume Cilium podCIDR network is 10.0.0.0/16, add route to 10.0.0.0/16
  8. ip route add 10.0.0.0/16 dev vxlan2  proto kernel  scope link  src 10.1.1.236
  9. # Allow Linux VM send ARP broadcast request to Cilium node for busybox pod ARP resolution
  10. # through vxlan2 device, note this depend on if Cilium is able to proxy busybox pod ARP
  11. # see https://github.com/cilium/cilium/issues/16890
  12. bridge fdb append 00:00:00:00:00:00 dst 10.169.72.233 dev vxlan2
  13. # Another way is to manually add busybox pod ARP address with Cilium node 10.169.72.233
  14. # cilium_vxlan interface MAC address like below
  15. bridge fdb append  <cilium_vxlan MAC> dst 10.169.72.233 dev vxlan2
  16. arp -i vxlan2 -s <busybox pod IP> <cilium_vxlan MAC>

If you are managing multiple VTEPs, follow the above process for each instance. Once the VTEPs are configured, you can configure Cilium to use the MAC, IP and CIDR ranges that you have configured on the VTEPs. Follow the instructions to VXLAN Tunnel Endpoint (VTEP) Integration (beta).

To test the VTEP network connectivity:

  1. # ping Cilium-managed busybox pod IP 10.0.1.1 for example from Linux VM
  2. ping 10.0.1.1

Limitations

  • This feature does not work with traffic from the host network namespace (including pods with hostNetwork=true).

  • This feature does not work with ipsec encryption between Cilium managed pod and VTEPs.