IPsec Transparent Encryption

This guide explains how to configure Cilium to use IPsec based transparent encryption using Kubernetes secrets to distribute the IPsec keys. After this configuration is complete all traffic between Cilium-managed endpoints, as well as Cilium-managed host traffic, will be encrypted using IPsec. This guide uses Kubernetes secrets to distribute keys. Alternatively, keys may be manually distributed, but that is not shown here.

Packets are not encrypted when they are destined to the same node from which they were sent. This behavior is intended. Encryption would provide no benefits in that case, given that the raw traffic can be observed on the node anyway.

Transparent encryption is not currently supported when chaining Cilium on top of other CNI plugins. For more information, see GitHub issue 15596.

Generate & Import the PSK

First, create a Kubernetes secret for the IPsec configuration to be stored. The example below demonstrates generation of the necessary IPsec configuration which will be distributed as a Kubernetes secret called cilium-ipsec-keys. A Kubernetes secret should consist of one key-value pair where the key is the name of the file to be mounted as a volume in cilium-agent pods, and the value is an IPSec configuration in the following format:

  1. key-id encryption-algorithms PSK-in-hex-format key-size

Note

Secret resources need to be deployed in the same namespace as Cilium! In our example, we use kube-system.

In the example below, GMC-128-AES is used. However, any of the algorithms supported by Linux may be used. To generate the secret, you may use the following command:

  1. $ kubectl create -n kube-system secret generic cilium-ipsec-keys \
  2.     --from-literal=keys="3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128"

The secret can be seen with kubectl -n kube-system get secrets and will be listed as cilium-ipsec-keys.

  1. $ kubectl -n kube-system get secrets cilium-ipsec-keys
  2. NAME                TYPE     DATA   AGE
  3. cilium-ipsec-keys   Opaque   1      176m

Enable Encryption in Cilium

Cilium CLI

Helm

If you are deploying Cilium with the Cilium CLI, pass the following options:

  1. cilium install --encryption ipsec

If you are deploying Cilium with Helm by following Installation using Helm, pass the following options:

  1. helm install cilium cilium/cilium --version 1.10.2 \
  2.   --namespace kube-system \
  3.   --set encryption.enabled=true \
  4.   --set encryption.nodeEncryption=false \
  5.   --set encryption.type=ipsec

encryption.enabled enables encryption of the traffic between Cilium-managed pods and encryption.nodeEncryption controls whether host traffic is encrypted. encryption.type specifies the encryption method and can be omitted as it defaults to ipsec.

Attention

When using Cilium in any direct routing configuration, ensure that the native routing CIDR is set properly. This is done using --native-routing-cidr=CIDR with the CLI or --set nativeRoutingCIDR=CIDR with Helm.

At this point the Cilium managed nodes will be using IPsec for all traffic. For further information on Cilium’s transparent encryption, see eBPF Datapath.

Encryption interface

An additional argument can be used to identify the network-facing interface. If direct routing is used and no interface is specified, the default route link is chosen by inspecting the routing tables. This will work in many cases, but depending on routing rules, users may need to specify the encryption interface as follows:

Cilium CLI

Helm

  1. cilium install --encryption ipsec --config encryption-interface=ethX
  1. --set encryption.ipsec.interface=ethX

Node-to-node encryption (beta)

In order to enable node-to-node encryption, add:

Cilium CLI

Helm

  1. cilium install --encryption ipsec --node-encryption
  1. --set encryption.enabled=true \
  2. --set encryption.nodeEncryption=true \

Note

Node-to-node encryption is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

Node-to-node encryption is tested and supported with direct routing modes. Using with encapsulation/tunneling is not currently tested or supported.

Support with tunneling mode is tracked with GitHub issue 13663.

Validate the Setup

Run a bash shell in one of the Cilium pods with kubectl -n kube-system exec -ti ds/cilium -- bash and execute the following commands:

  1. Install tcpdump

    1. $ apt-get update
    2. $ apt-get -y install tcpdump

  2. Check that traffic is encrypted. In the example below, this can be verified by the fact that packets carry the IP Encapsulating Security Payload (ESP). In the example below, eth0 is the interface used for pod-to-pod communication. Replace this interface with cilium_vxlan if tunneling is enabled.

    1. tcpdump -n -i eth0 esp
    2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    3. listening on cilium_vxlan, link-type EN10MB (Ethernet), capture size 262144 bytes
    4. 15:16:21.626416 IP 10.60.1.1 > 10.60.0.1: ESP(spi=0x00000001,seq=0x57e2), length 180
    5. 15:16:21.626473 IP 10.60.1.1 > 10.60.0.1: ESP(spi=0x00000001,seq=0x57e3), length 180
    6. 15:16:21.627167 IP 10.60.0.1 > 10.60.1.1: ESP(spi=0x00000001,seq=0x579d), length 100
    7. 15:16:21.627296 IP 10.60.0.1 > 10.60.1.1: ESP(spi=0x00000001,seq=0x579e), length 100
    8. 15:16:21.627523 IP 10.60.0.1 > 10.60.1.1: ESP(spi=0x00000001,seq=0x579f), length 180
    9. 15:16:21.627699 IP 10.60.1.1 > 10.60.0.1: ESP(spi=0x00000001,seq=0x57e4), length 100
    10. 15:16:21.628408 IP 10.60.1.1 > 10.60.0.1: ESP(spi=0x00000001,seq=0x57e5), length 100

Key Rotation

To replace cilium-ipsec-keys secret with a new key:

  1. KEYID=$(kubectl get secret -n kube-system cilium-ipsec-keys -o yaml | awk '/^\s*keys:/ {print $2}' | base64 -d | awk '{print $1}')
  2. if [[ $KEYID -gt 15 ]]; then KEYID=0; fi
  3. data=$(echo "{\"stringData\":{\"keys\":\"$((($KEYID+1))) "rfc4106\(gcm\(aes\)\)" $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 64)) 128\"}}")
  4. kubectl patch secret -n kube-system cilium-ipsec-keys -p="${data}" -v=1

Then restart Cilium agents to transition to the new key with kubectl delete pod -n kube-system -l k8s-app=cilium. During transition the new and old keys will be in use. The Cilium agent keeps per endpoint data on which key is used by each endpoint and will use the correct key if either side has not yet been updated. In this way encryption will work as new keys are rolled out.

The KEYID environment variable in the above example stores the current key ID used by Cilium. The key variable is a uint8 with value between 0-16 and should be monotonically increasing every re-key with a rollover from 16 to 0. The Cilium agent will default to KEYID of zero if its not specified in the secret.

Troubleshooting

  • If the cilium Pods fail to start after enabling encryption, double-check if the IPSec Secret and Cilium are deployed in the same namespace together.

  • Make sure that the Cilium pods have kvstore connectivity:

    1. cilium status
    2. KVStore:                Ok   etcd: 1/1 connected: http://127.0.0.1:31079 - 3.3.2 (Leader)
    3. [...]

  • Check for level=warning and level=error messages in the Cilium log files

    • If there is a warning message similar to Device eth0 does not exist, use --set encryption.ipsec.interface=ethX to set the encryption interface.

  • Run a bash in a Cilium Pod and validate the following:

    • Routing rules matching on fwmark:

      1. $ ip rule list
      2. 1:      from all fwmark 0xd00/0xf00 lookup 200
      3. 1:      from all fwmark 0xe00/0xf00 lookup 200
      4. [...]

    • Content of routing tables

      1. $ ip route list table 200
      2. local 10.60.0.0/24 dev cilium_vxlan proto 50 scope host
      3. 10.60.1.0/24 via 10.60.0.1 dev cilium_host

      In case of IPAM ENI mode, check if routing rules exist for the the IP address of cilium_host interface..

      1. $ ip addr show cilium_host
      2. 5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
      3. link/ether 96:b1:5c:82:75:a3 brd ff:ff:ff:ff:ff:ff
      4. inet 192.168.174.161/32 scope link cilium_host
      5.    valid_lft forever preferred_lft forever
      6. inet6 fe80::94b1:5cff:fe82:75a3/64 scope link
      7.    valid_lft forever preferred_lft forever
      9. $ ip rule | grep 192.168.174.161
      10. 111:   from 192.168.174.161 to 192.168.0.0/16 lookup 11

    • XFRM policy:

      1. $ ip xfrm p
      2. src 10.60.1.1/24 dst 10.60.0.1/24
      3.         dir fwd priority 0
      4.         mark 0xd00/0xf00
      5.         tmpl src 10.60.1.1 dst 10.60.0.1
      6.                 proto esp spi 0x00000001 reqid 1 mode tunnel
      7. src 10.60.1.1/24 dst 10.60.0.1/24
      8.         dir in priority 0
      9.         mark 0xd00/0xf00
      10.         tmpl src 10.60.1.1 dst 10.60.0.1
      11.                 proto esp spi 0x00000001 reqid 1 mode tunnel
      12. src 10.60.0.1/24 dst 10.60.1.1/24
      13.         dir out priority 0
      14.         mark 0xe00/0xf00
      15.         tmpl src 10.60.0.1 dst 10.60.1.1
      16.                 proto esp spi 0x00000001 reqid 1 mode tunnel

    • XFRM stats with state:

  1. > Check if the packets count increases as you send traffic.
  2. > 
  3. > Following is the output from the source node.
  4. > 
  5. > ```
  6. > $ ip -s xfrm s
  7. > src 10.60.0.1 dst 10.60.1.1
  8. >         proto esp spi 0x00000001 reqid 1 mode tunnel
  9. >         replay-window 0
  10. >         auth-trunc hmac(sha256) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546 96
  11. >         enc cbc(aes) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546
  12. >         anti-replay context: seq 0x0, oseq 0xe0c0, bitmap 0x00000000
  13. >         sel src 0.0.0.0/0 dst 0.0.0.0/0
  14. >         lifetime config:
  15. >           limit: soft (INF)(bytes), hard (INF)(bytes)
  16. >           limit: soft (INF)(packets), hard (INF)(packets)
  17. >           expire add: soft 0(sec), hard 0(sec)
  18. >           expire use: soft 0(sec), hard 0(sec)
  19. >         lifetime current:
  20. >           9507(bytes), 137(packets)
  21. >           add 2021-02-10 08:20:09 use 2021-02-10 08:30:12
  22. >         stats:
  23. >           replay-window 0 replay 0 failed 0
  24. > ```
  25. > 
  26. > Following is the output from the destination node.
  27. > 
  28. > ```
  29. > $ ip -s xfrm s
  30. > src 10.60.1.1 dst 10.60.0.1
  31. >         proto esp spi 0x00000001 reqid 1 mode tunnel
  32. >         replay-window 0
  33. >         auth-trunc hmac(sha256) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546 96
  34. >         enc cbc(aes) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546
  35. >         anti-replay context: seq 0x0, oseq 0xe0c0, bitmap 0x00000000
  36. >         sel src 0.0.0.0/0 dst 0.0.0.0/0
  37. >         lifetime config:
  38. >           limit: soft (INF)(bytes), hard (INF)(bytes)
  39. >           limit: soft (INF)(packets), hard (INF)(packets)
  40. >           expire add: soft 0(sec), hard 0(sec)
  41. >           expire use: soft 0(sec), hard 0(sec)
  42. >         lifetime current:
  43. >           9507(bytes), 137(packets)
  44. >           add 2021-02-10 08:20:09 use 2021-02-10 08:30:12
  45. >         stats:
  46. >           replay-window 0 replay 0 failed 0
  47. > ```
  49. -   BPF program to decrypt traffic:
  51. > Check if the BPF program to decrypt traffic is attached to all network facing interfaces, or matching the configuration of `--encrypt-interface` (if specified).
  52. > 
  53. > ```
  54. > $ tc filter show dev eth0 ingress
  55. > filter protocol all pref 1 bpf chain 0
  56. > filter protocol all pref 1 bpf chain 0 handle 0x1 bpf_network.o:[from-network] direct-action not_in_hw id 1145 tag 51b408acf94aa23f jited
  57. > ```

Disabling Encryption

To disable the encryption, regenerate the YAML with the option encryption.enabled=false