Calico Open Source 3.27 release notes

Learn about the new features, bug fixes, and other updates in this release of Calico.

New features and enhancements

HPC-based Windows install and upgrade

We’ve added support for Operator-based install and upgrade for Windows nodes using HPC (HostProcess/Privileged containers), Calico can now bootstrap Windows worker nodes just like Linux worker nodes. Platform engineers can scale Windows worker nodes with an auto-scaling node pool without bootstrapping each node manually, and upgrade Calico deployments without manually upgrading nodes one at a time.

For more information, see the Calico for Windows Operator guide.

Support for Azure CNI with overlay networking for AKS

We’ve add supported for Azure CNI with overlay networking for AKS clusters. This option augments support for the Azure CNI with no overlay where a VNet IP address is assigned to every pod, which can lead to IP address exhaustion.

To review Azure CNI options, see Microsoft Azure Kubernetes Service (AKS).

Calico with VPP data plane is generally available

The VPP data plane has been validated in Calico and is ready for general use.

For more information, see Get started with VPP networking.

Improved Felix performance for extremely large clusters

This release adds improvements to how Felix processes active policy rules for clusters with extremely high numbers of endpoints and policy rules. Previously, calculating selector-based rules in these circumstances could take a long time or end in failure. Now Felix can perform these calculations quickly while reducing CPU usage by orders of magnitude.

For more information, see Selector performance in EntityRules.

Enhancements

  • Use upstream hcsshim package instead of fork and update it to latest stable version (v0.11.4). calico #8192 (@coutinhop)
  • Enhance BGPFilter to filter route advertisement to BGP peers based on route’s interface. Calico nodes stop advertising tunnel routes to iBGP peers as the routes are programmed by Felix. If you are doing iBGP peering with external peers this should not affect since the workload’s routes are still advertised. Felix expect either .cali or .calico suffix for tunnel names like Wireguard. calico #8175 (@mazdakn)
  • We no longer provide Calico packages for OpenStack on Ubuntu Bionic or older, or test ourselves on Bionic, as it has become difficult to build a working Felix package for Bionic, due to glibc and libpcap library API version issues, and Bionic is now more than 5 years old and beyond its Canonical LTS period. Our current recommendation is to use Calico with OpenStack Yoga on Ubuntu Focal or later. calico #8157 (@nelljerram)
  • Enhance BGPFilter to filter route advertisement to BGP peers based on route’s origin. calico #8149 (@mazdakn)
  • Update Typha Deployment tolerations to helm charts so that it can be scheduled on any node. calico #7979 (@ffilippopoulos)
  • kube controllers run as a non-root user in s390x builds by default calico #7955 (@liudalibj)
  • calico-node -bpf dump policy now suppresses printing assembly by default. You need to use —asm flag [or -a] to see the assembly as well. calico #7954 (@StevenTigera)
  • Add a timeout to Typha’s TLS handshake. calico #7909 (@rodrigorfk)
  • Add pre-deletion hook to helm chart to facilitate helm uninstall calico #7859 (@caseydavenport)
  • Don’t write AS number to node if running with CALICO_NETWORKING_BACKEND=none. calico #7821 (@tobiasgiese)
  • Add pod security standard annotation to the tigera-operator namespace for Kubernetes 1.25+ calico #7732 (@si458)
  • allows reference to existing secrets for imagePullSecrets without passing the secret itself to helm. this enables management of secrets by an external system like sealedsecrets and prevents the secret data from being stored in helm. It works by allowing use of the installation’s imagePullSecret field directly, whereas previously only the toplevel imagePullSecrets field was used. calico #7657 (@ozdanborne)
  • eBPF: Allow user to specify a regexp of interfaces that we want to turn GRO off at. calico #7633 (@StevenTigera)
  • Calico’s integration code for OpenStack has been updated for OpenStack Yoga. calico #7625 (@nelljerram)
  • Add support for OpenStack on Ubuntu Jammy calico #7622 (@danudey)
  • BPF mode: add felix configuration parameter to allow NOTRACK to be disabled for certain external interfaces interfaces. Useful for workaround conflicts with non-Calico networked containers etc. calico #7595 (@StevenTigera)
  • Expose the configuration for the CSI plugin in the operator helm chart calico #7480 (@nimimeht)
  • Update upstream CNI plugins and Flannel downloads to latest golang patches calico #8303 (@matthewdupre)
  • Remove the policy index from the correct jump map calico #8282 (@sridhartigera)
  • Avoid conntrack cleanup of IPv6 host networked workloads. (#8272) calico #8273 (@sridhartigera)
  • eBPF: ClusterIP reflects InternalTrafficPolicy=Local calico #8262 (@tomastigera)
  • Add openstack publishing promotion [v3.27][calico #8261](https://github.com/projectcalico/calico/pull/8261) (@danudey)
  • Bump k8s dependency to v1.27.8 calico #8246 (@hjiawei)
  • Node learns about its ipv6 address in Kubernetes even if BGP is turned off and CNI is not Calico.
  • eBPF: kube-proxy ServiceInternalTrafficPolicy is now GA and setting the gate would generate a warning message. calico #8242 (@tomastigera)
  • eBPF: fixed policy cleanup after felix restart if a device is not present anymore calico #8231 (@fasaxc)

Deprecated and removed features

  • In Calico 3.27, the Felix configuration parameter BPFConnectTimeLoadBalancingEnabled is deprecated and is planned to be removed in a future release. As an alternative, you can use the bpfConnectTimeLoadBalancing and BPFHostNetworkedNATWithoutCTLB configuration parameters.

Technology preview features

IPv6 support for eBPF

Calico now supports IPv6 in the eBPF data plane.

For more information, see Enable the eBPF data plane.

Bug fixes

  • Prevent IPAM garbage collection from freezing when under high load calico #8154 (@JuoCode)
  • Correct policy for OpenStack sec group with no remote_ip_prefix calico #8026 (@nelljerram)
  • Fixed AWS ec2 detection not working with imdsv2 on Calico for Windows. calico #7970 (@davidgiga1993)
  • Fix panic when running ‘calicoctl get nodes’ when ASNumber was not present in the default BGPConfiguration. calico #7858 (@coutinhop)
  • Fix a few instances where KUBECONFIG was not respected calico #7796 (@skmatti)
  • Fix helm chart rendering multiple image pull secrets incorrectly calico #7752 (@oxr463)
  • Fix YAML injection vulnerabilities due to unsafe templating calico #7642 (@skmatti)
  • Fix ‘error while loading shared libraries: libresolv.so.2: cannot open shared object file’ on csi-node-driver-registrar. calico #7586 (@coutinhop)
  • Fix a divide-by-zero panic in Typha if it received a SIGTERM when it had no active connections. Since Typha exits in either case, the impact was limited to a scary panic log. calico #7585 (@fasaxc)
  • Fix a divide-by-zero panic in Typha if it received a SIGTERM when it had no active connections. Since Typha exits in either case, the impact was limited to a scary panic log. calico #7575 (@fasaxc)
  • When running Calico in policy-only mode, do not write the IP annotations to the node. calico #7550 (@skmatti)
  • eBPF: fixed host access to self and a service that redirects to self without CTLB calico #8189 (@tomastigera)
  • Fix incorrect conversion to 16-bit offset in the BPF assembler. Fail if the value would wrap. calico #8176 (@fasaxc)
  • eBPF: icmp errors must be always treated as related, not only after a CT miss. calico #8153 (@tomastigera)
  • eBPF: Config option added for host networked NAT. Change in the configs related to connect time load balancing. calico #8139 (@sridhartigera)
  • BPF mode: fix that netlink IP sets were programmed even in BPF mode until the first policy/endpoint deletion event. calico #8101 (@fasaxc)
  • eBPF: fixes felix panic upon restart in debug mode when there are existing policy counters calico #7797 (@tomastigera)
  • eBPF: fix applyOnforward=false in global policies calico #7707 (@tomastigera)
  • eBPF: handles IPv4 packets with IP options. XDP only does so in kernels 5.18+ it drops them in older kernels. calico #7701 (@tomastigera)
  • eBPF: only report errors setting /proc/sys/net/../accept_local etc. if the device is present calico #7631 (@tomastigera)

Known issues

  • Calico panics if kube-proxy or other components are using native nftables rules instead of the iptables-nft compatibility shim.

    Until Calico supports native nftables mode, we recommend that you continue to use the iptables-nft compatibility layer for all components. (The compatibility layer was the only option before Kubernetes v1.29 added alpha-level nftables support.)

    Do not run Calico in “legacy” iptables mode on a system that is also using nftables. Although this combination does not panic or fail (at least on kernels that support both), the interaction between iptables “legacy” mode and nftables is confusing: both iptables and nftables rules can be executed on the same packet, leading to policy verdicts being “overturned”.

Release details

Calico Open Source 3.27.0 general availability release

December 18, 2023

Calico Open Source release 3.27.0 is now generally available.

Calico Open Source 3.27.2 bug fix release

February 17, 2024

Enhancements

  • ebpf: alternative cgroup2 mount path can be specified by setting CALICO_CGROUP_PATH env var for node. calico #8512 (@tomastigera)
  • Host MTU auto-detection now ignores interfaces that are down. calico #8499 (@fasaxc)
  • Bump iptables version of calico-node to 1.8.8 calico #8485 (@coutinhop)
  • Fix crypto UT after upgrading to golang v1.21.6 calico #8478 (@hjiawei)
  • ebpf: setting BPFExcludeIPsFromNAT allows node-local dns cache to work calico #8473 (@tomastigera)
  • Update runc calico #8471 (@Behnam-Shobiri)
  • Enable CGO for arm64 when building felix and node calico #8470 (@hjiawei)
  • ebpf: align defaultEndpointToHostAction with iptables do not apply normal -hep policy to wep calico #8459 (@tomastigera)
  • Felix now avoids accessing non-Calico IP sets. This reduces the scope for IP set compatibility errors when another app has created an IP set that Calico’s version of IP set can’t parse. calico #8412 (@mazdakn)
  • Remove spammy BIRD log about ignoring DOWN interfaces calico #8401 (@lbogdan)
  • Skip apiserver etcd3 healthcheck and disable profiling calico #8396 (@hjiawei)

Bug fixes

  • Added retry mechanism to Windows version retrieval in install-cni to address possible panics when the OS is not ready. calico #8466 (@coutinhop)
  • ebpf: fixes possible holes in the list NAT backends if there is a terminating pod. calico #8460 (@tomastigera)
  • ebpf: fixed pods in nat-outgoing should not SNAT when accessing local host calico #8458 (@tomastigera)
  • ebpf: fixed leakage of nodeport healthcheck servers calico #8457 (@tomastigera)
  • ebpf: don’t stumble on unknown prog types passed as int in json calico #8456 (@tomastigera)
  • Only program failsafe rules for IP version of the CIDR calico #8455 (@tomastigera)
  • Fix confd issues when running on Windows operator installations (using HPC). calico #8422 (@coutinhop)

How to update

To update an existing installation of Calico Open Source 3.27, see the upgrade guides.

Calico Open Source 3.27.3 bug fix release

April 2, 2024

Bug fixes

  • Fix CVE-2023-48795: bump crypto package to 0.19.0 calico #8639 (@cyclinder)
  • ebpf: Update map definitions in programs used in iptables mode to let libbpf v1.0+ load them successfully. calico #8624 (@mazdakn)
  • Route reflector nodes now properly advertise Service LoadBalancer IP addresses even if there is no local endpoint on the node. calico #8593 (@AMacedoP)
  • ebpf: host can access self via a service without CTLB calico #8588 (@tomastigera)
  • Add running of token refresher to Calico for Windows. calico #8571 (@coutinhop)
  • Fix tigera-operator Helm chart missing RBAC for uninstall job calico #8547 (@sridhartigera)
  • Fix libpcap.so.0.8 missing on arm64 calico #8533 (@hjiawei)
  • Fix missing libresolv.so in node-driver-registrar calico #8516 (@hjiawei)

Other changes

  • Update Flannel to v1.2.0-flannel2-go1.21.8 calico #8661] (@coutinhop)

How to update

To update an existing installation of Calico Open Source 3.27, see the upgrade guides.

Calico Open Source 3.27.4 bug fix release

July 8, 2024

Enhancements

  • Update go version to v1.22.5, kubernetes version to v1.27.15, UBI version to v8.10 calico #8937 (@coutinhop)
  • ebpf: support for service loop prevention calico #8893 (@sridhartigera)
  • ebpf: cleanup BPF special devices when BPF is turned off calico #8891 (@tomastigera)

Bug fixes

  • Fix calico-felix rpm libpcap.so linking dependency. calico #8986 (@hjiawei)
  • Upgrade bpftool to v7.4 to fix the issue of loading XDP programs in iptables data plane that happens in few distributions. calico #8940 (@mazdakn)
  • ebpf: fix* let the node handle packet when we are not sure about the destination calico #8926 (@tomastigera)
  • ebpf: fixed cleaning of programs and map when switching from ebpf to iptables mode. calico #8888 (@tomastigera)
  • Fix errors on kernels that don’t support the XFRM netlink family. calico #8872 (@fasaxc)
  • ebpf: Update map definition in sockops program to let libbpf v1.0+ load them successfully. calico #8702 (@mazdakn)
  • ebpf: fix natOutgoing SNAT for icmp6 calico #8686 (@tomastigera)
  • Improve cni-plugin binary install verification. calico #8841 (@ialidzhikov)

How to update

To update an existing installation of Calico Open Source 3.27, see the upgrade guides.