Use IPVS kube-proxy

Big picture

Use IPVS kube-proxy mode for load balancing traffic across pods.

Value

No matter where you are on your journey with container networking, iptables will serve you well. However, if you are scaling above 1,000 services, it’s worth looking at potential performance improvements using kube-proxy IPVS mode.

Concepts

Kubernetes kube-proxy

Kube-proxy process handles everything related to Services on each node. It ensures that connections to the service cluster IP and port go to a pod that backs the service. If backed by more than one pod, kube-proxy load-balances traffic across pods.

Kube-proxy runs in three modes: userspace, iptables, and ipvs. (Userspace is old, slow and not recommended.) Here’s a quick summary of iptables and ipvs modes.

kube-proxy modeDesigned to be…Linux kernel hooksConnection processing overhead…
iptablesAn efficient firewallNAT pre-routing using sequential rulesGrows proportional to cluster size
ipvsA load balancer with scheduling options like round-robin, shortest-expected delay, least connections, etc.Optimized lookup routineStays constant, independent of cluster size

If you are wondering about the performance differences between iptables and ipvs, the answers are definitely not straightforward. For a comparison between iptables (including Calico’s own use of iptables) and ipvs modes, see Comparing kube-proxy modes: iptables or IPVS?.

IPVS mode and NodePort ranges

Kube-proxy IPVS mode supports NodePort services and cluster IPs. Calico also uses NodePorts for routing traffic to the cluster, including the same default Kubernetes NodePort range (30000:32767). If you change your default NodePort range in Kubernetes, you must also change it on Calico to maintain ipvs coverage.

iptables: when to change mark bits

To police traffic in IPVS mode, Calico uses additional iptables mark bits to store an ID for each local Calico endpoint. If you are planning to run more than 1,022 pods per host with IPVS enabled, you may need to adjust the mark bit size using the IptablesMarkMask parameter in Calico FelixConfiguration.

Calico auto detects ipvs mode

When Calico detects that kube-proxy is running in IPVS mode (during or after installation), IPVS support is automatically activated. Detection happens when calico-node starts up, so if you change kube-proxy’s mode in a running cluster, you will need to restart your calico-node instances.

Before you begin…

Required

  • kube-proxy is configured to use IPVS mode
  • Services for ipvs mode are type, NodePort

How to

As previously discussed, there is nothing you need to do in Calico to use IPVS mode; if enabled, the mode is automatically detected. However, if your default Kubernetes NodePort range changes, use the following instructions to update Calico nodeport ranges to stay in sync. Detection happens when calico-node starts up, so if you change kube-proxy’s mode in a running cluster, you will need to restart your calico-node instances.

Change Calico default nodeport range

In the FelixConfiguration resource, change the configuration parameter for the default node port range (KubeNodePortRange,) in Calico to match your new default range in Kubernetes. For help, see FelixConfiguration.