Calico Open Source 3.28 release notes

Learn about the new features, bug fixes, and other updates in this release of Calico.

New features and enhancements

Configure multiple IP pools during installation

You can now configure multiple IP pools when you install Calico on your clusters. Specifying multiple IP pools during installation is important for clusters where IP address assignment is tied to the topology of the cluster. This can be helpful, for example, when IP pools are assigned to particular nodes, racks, or availability zones in order to satisfy internal policy or firewalling restrictions.

For more information, see Create multiple IP pools.

Delay pod Ready status until after network policies are rendered

This update lets you prevent new pods from moving to a ready status until after their network policies have finished rendering. In some high-churn environments, pods may be deployed and marked as ready before network policies have been rendered, causing problems when traffic to those pods is denied. The new policy_setup_timeout_seconds configuration option allows you build in a delay to give those polices time to finish processing.

You may find this option useful if you’re running traditional applications that assume resiliency in the architecture.

For more information, see Enabling policy setup timeout.

IPv6 support for eBPF mode is generally available

Calico Open Source 3.28 introduces full support for IPv6 for clusters using the eBPF data plane.

For more information, see Enable the eBPF dataplane and Configure dual stack or IPv6 only.

Enhancements

  • We made it easier to migrate your cluster from iptables to eBPF data plan modes. Previously, having a mix of nodes in iptables and eBPF modes could cause a breakdown in cluster networking. Now, the operator makes the transition with minimal disruption.
  • We improved how Felix calculates rules in situations where there are high numbers (up to 200) of shared policies per endpoint. Depending on the number of shared policies, this can reduce the number of rules per pod by several hundred. These improvements apply only when Calico is running in iptables mode.
  • Re-enable VXLAN checksum offload for kernels > 5.7 (takes effect when nodes are restarted). calico #8774 (@tomastigera)
  • Calico is now built with Go 1.22.3 against Kubernetes v1.28.7. Moving to Go 1.22 fixed a couple of latent bugs, detected by the new for loop semantics. calico #8717 (@fasaxc)
  • Calico now builds against Kubernetes v1.28.9 calico #8733 (@fasaxc)
  • Update flannel version to v0.24.3. calico #8595 (@laibe)
  • Bump iptables version of calico-node to 1.8.8 calico #8416 (@cyclinder)
  • Bump github.com/containerd/containerd from 1.6.23 to 1.6.26 calico #8355 (@dependabot[bot])
  • Bump github.com/opencontainers/runc from 1.1.6 to 1.1.12 calico #8468 (@dependabot[bot])
  • Update upstream CNI plugins and Flannel downloads to latest golang patches calico #8307 (@matthewdupre)
  • The calico/node-driver-registrar image now has labels for description/maintainers/etc as required by OpenShift certification. calico #8730 (@fasaxc)
  • Migrate to UBI based go-build calico #8103 (@hjiawei)
  • Adds options to Felix and the CNI to delay pods going ready until their dataplane programming is complete. calico #8469 (@aaaaaaaalex)
  • Add global +x permissions to endpoint-status dir (#8633) calico #8641 (@aaaaaaaalex)
  • Typha’s typha_breadcrumb_size Prometheus stat now decays to zero if there are no breadcrumbs at all. Previously it would show the last value, or NaN, which were misleading. calico #8614 (@fasaxc)
  • Update the Grafana dashboard for Typha. Tested with Grafana v10.4.0. calico #8613 (@frozenprocess)
  • Remove unnecessary FIPS code calico #8538 (@rene-dekker)
  • Move key-cert-provisioner to the monorepo calico #8475 (@rene-dekker)
  • Host MTU auto-detection now ignores interfaces that are down. calico #8496 (@fasaxc)
  • Improve IPAM block garbage collection behavior for IP pools with small blocks. calico #8454 (@caseydavenport)
  • Clean up: VXLAN ARP and FDB programming is moved to a new sub-component. This should make it easier to maintain. calico #8449 (@fasaxc)
  • Felix now avoids accessing non-Calico IP sets. This reduces the scope for IP set compatibility errors when another app has created an IP set that Calico’s version of IP set can’t parse. calico #8387 (@mazdakn)
  • Move certificates permissions out of the else-block. calico #8369 (@rene-dekker)
  • Docker images now use COPY instead of ADD as recommended by CIS. Typha no longer relies on the tini init daemon, it handles the common signals internally (and it does not spawn any subprocesses so there is no need for a reaper). calico #8289 (@fasaxc)
  • Only program failsafe rules for IP version of the CIDR calico #8286 (@tomastigera)
  • Felix now breaks up “policy jump rules” into new iptables “policy group chains” by selector. If two endpoints share a common sequence of policies they will share the same group chain, which reduces the number of rules that need to be programmed. calico #8098 (@fasaxc)
  • Felix and Typha now support enabling the Go standard library’s debug server via the DebugHost/DebugPort configuration options. This allows process profiling data to be collected more easily. calico #8091 (@fasaxc)
  • Disable IPIP tunnel checksum offload on kernels ≤ v5.7 calico #8031 (@cyclinder)
  • Improve BIRD liveness probe so that it confirms BIRD is responsive over its socket calico #7556 (@caseydavenport)
  • Run calico/apiserver as non-root by default calico #8576 (@hjiawei)
  • Move calicoctl binary to standard executable search path calico #8364 (@hjiawei)
  • ebpf: wg6 traffic is allowed even if blocked by policy calico #8755 (@tomastigera)
  • ebpf: clean up stale icmp6 conntrack entries calico #8754 (@tomastigera)
  • ebpf: fixed fd leak calico #8750 (@tomastigera)
  • ebpf: When a pod connects via a service to self, ingress traffic is policed as if it’s source is the pod and not the host after MASQ calico #8719 (@tomastigera)
  • ebpf: fixed source IP used by host when CTLB is disabled and loopback device has non-local IP set. calico #8718 (@tomastigera)
  • ebpf: fix map creation during upgrade. calico #8690 (@sridhartigera)
  • ebpf: fix natOutgoing SNAT for icmp6 calico #8688 (@tomastigera)
  • ebpf: fixed source IP used by host when CTLB is disabled and loopback device has non-local IP set. calico #8618 (@tomastigera)
  • ebpf: Update map definitions in programs used in iptables mode to let libbpf v1.0+ load them successfully. calico #8610 (@mazdakn)
  • ebpf: XDP v6 requires Linux kernel 5.18+ (Ubuntu >=22.04) calico #8587 (@sridhartigera)
  • ebpf: host can access self via a service without CTLB calico #8564 (@tomastigera)
  • ebpf: Support dual stack. calico #8509 (@sridhartigera)
  • ebpf: projectcalico.org/natExcludeService=true makes kube-proxy to ignore the service. That allows using node local dns cache. calico #8484 (@tomastigera)
  • ebpf: fixes arm64 build for use with eBPF* Felix is able to enable ebpf (again) calico #8467 (@hjiawei)
  • Fix that Felix could briefly report “ready” in the middle of initialisation, before going “non-ready” again until the dataplane was in-sync. In eBPF mode, Felix will now report non-Ready if it fails to program some BPF programs. Previously, this would only be reported through logging. calico #8506 (@fasaxc)
  • ebpf: fixes possible holes in the list NAT backends if there is a terminating pod. calico #8438 (@tomastigera)
  • ebpf: fixed cleaning of programs and map when switching from ebpf to iptables mode. calico #8415 (@tomastigera)
  • ebpf: align defaultEndpointToHostAction with iptables* do not apply normal -hep policy to wep calico #8388 (@tomastigera)
  • ebpf: fixed pods in nat-outgoing should not SNAT when accessing local host calico #8380 (@tomastigera)
  • ebpf: setting BPFExcludeIPsFromNAT allows node-local dns cache to work calico #8338 (@tomastigera)
  • ebpf: fixed leakage of nodeport healthcheck servers calico #8313 (@tomastigera)
  • ebpf: don’t stumble on unknown prog types passed as int in json calico #8295 (@tomastigera)
  • ebpf: ClusterIP reflects InternalTrafficPolicy=Local calico #8259 (@tomastigera)
  • ebpf: fixed policy cleanup after felix restart if a device is not present anymore. calico #8235 (@fasaxc)
  • eBPF: Support many more active policy rules per endpoint+direction. The BPF policy compiler now supports splitting policy programs if they get larger than the kernel would allow. The exact number of policy rules per endpoint depends on the details of the rules but for some real-world examples we see an increase from approximately 2k rules to approx 15k rules per endpoint direction. calico #8230 (@fasaxc)
  • ebpf: kube-proxy ServiceInternalTrafficPolicy is now GA and setting the gate would generate a warning message. calico #8213 (@tomastigera)
  • ebpf: BPFKubeProxyEndpointSlicesEnabled config option is deprecated, has no effect and will be removed. calico #8160 (@tomastigera)
  • ebpf: Config option added for host networked NAT. Change in the configs related to connect time load balancing. calico #8139 (@sridhartigera)
  • ebpf: alternative cgroup2 mount path can be specified by setting CALICO_CGROUP_PATH evn var for node. calico #8085 (@amrut-asm)
  • ebpf: When a pod connects via a service to self, ingress traffic is policed as if it’s source is the pod and not the host after MASQ calico #6949 (@tomastigera)
  • ebpf: Use a label to clean up conntrack to terminating UDP backends calico #8480 (@tomastigera)
  • You can now specify kubernetesServiceEndpoint in the helm chart to support windows or eBFP. calico #8443 (@davhdavh)
  • Helm chart now supports specifying priorityClassName in values.yaml calico #8427 (@elsnepal)
  • Support affinity in tigera-operator chart calico #8095 (@gyuho)
  • Ability to set FelixConfiguration via helm chart calico #8559 (@ti-afra)
  • Added retry mechanism to Windows version retrieval in install-cni to address possible panics when the OS is not ready. calico #8462 (@coutinhop)

Bug fixes

  • Fix bug that inhibited garbage collection of Namespaces and ServiceAccounts with OwnerReferences. calico #8586 (@caseydavenport)
  • Fix that projectcalico.org/v3 resources with OwnerReferences were unable to be garbage collected due to non-unique UIDs. calico #8586 (@caseydavenport)
  • apiserver defaults logrus level based on -v argument calico #8699 (@caseydavenport)
  • Fix missing log line numbers in cni-installer log output calico #8698 (@caseydavenport)
  • Fix bug where key usage was not consistent calico #8581 (@rene-dekker)
  • calicoctl node run no longer executes the Kubernetes token watcher, which can only run inside a Kubernetes pod. calico #8483 (@fasaxc)
  • Fix missing permissions when uninstalling tigera-operator. calico #8413 (@KonstantinVishnivetskii)
  • Route reflector nodes now properly advertise Service LoadBalancer IP addresses even if there is no local endpoint on the node. calico #8358 (@AMacedoP)
  • Fix source IP spoofing annotation being ignored in etcd datastore mode calico #8347 (@fm9282)
  • Node learns about it’s ipv6 address in kubernetes even if BGP is turned off and CNI is not calico. calico #8209 (@tomastigera)
  • Fix that cross-subnet routes were not moved when the VXLAN parent device was changed. calico #8279 (@fasaxc)
  • Add running of token refresher to Calico for Windows. calico #8563 (@coutinhop)
  • Fix confd issues when running on Windows operator installations (using HPC). calico #8421 (@coutinhop)
  • Fixed AutoCreateServiceAccountTokenSecret param handling in install-calico-windows.ps1 calico #8365 (@coutinhop)
  • Use KUBECONFIG env variable to build cluster config calico #8549 (@skmatti)

Updating

Release notes - 图1info

Calico 3.28 contains breaking changes for installations that use the Calico API server.

  • Breaking change: Upgrading Calico will alter the UID of all projectcalico.org/v3 resources. If you’re using the Calico API server, you must restart any controllers, including kube-controller-manager, that manage these resources after the upgrade. This change addresses an issue where duplicate UIDs on different API resources could disrupt Kubernetes garbage collection. calico #8586 (@caseydavenport)
  • This release re-enables VXLAN checksum offload for kernels > 5.7 (this takes effect when nodes are restarted). The offload was disabled in the past due to kernel bugs but has substantial performance impact. We’ve been unable to find an environment where this needs to be disabled, so are re-enabling the offload with the intent of proving it works or finding a setup where it doesn’t. If you encounter problems after upgrading and restarting nodes, please let us know and set ChecksumOffloadBroken=true in the FelixConfiguration’s featureDetectOverride field to restore the previous behavior. calico #8774 (@tomastigera)

Release details

Calico Open Source 3.28.0 general availability release

May 10, 2024

Calico Open Source release 3.28.0 is now generally available.

Updating

To update a previous version of Calico, see our upgrade guides.

Calico Open Source 3.28.1 bug fix update

July 30, 2024

Enhancements

  • Add IPReservation and BGPFilter to etcd datastore migration. #9056 (@caseydavenport)
  • Update k8s to v1.28.12. #9040 (@coutinhop)
  • Disable checksum offloading for VXLAN devices. #9037
  • Update go version to v1.22.5, node-driver-registrar to v2.11.1. #9030 (@coutinhop)
  • ebpf: don’t drop, but reject unknown midflow tcp packets with rst. #8970 (@sridhartigera)
  • ebpf: set bpfin/out.cali MTU to the smallest of all host ifaces including overlay. That means if jumbo frames are used, this device also uses them. #8954 (@tomastigera)
  • Upgrade bpftool to v7.4 to fix the issue of loading XDP programs in iptables data plane that happens in few distributions. #8939 (@mazdakn)
  • ebpf: Forwarding services via vxlan tunnel uses different source ports for different flows to better utilize bonded devices and CPUs on the receiving side. #8898 (@tomastigera)
  • ebpf: support for service loop prevention. #8892 (@sridhartigera)
  • ebpf: cleanup BPF special devices when BPF is turned off. #8887 (@tomastigera)
  • Improve cni-plugin binary install verification. #8844 (@ialidzhikov)
  • ebpf: Update map definition in sockops program to let libbpf v1.0+ load them successfully. #8834 (@mazdakn)
  • The calico-node, calico-kube-controllers and calico-typha pods now run with securityContext.seccompProfile.type=RuntimeDefault. #6524 (@dimityrmirchev)

Bug fixes

  • Fix ‘undefined symbol: xtables_strdup’ error when running ‘iptables-legacy-save’ in the calico-node image. #9023 (@coutinhop)
  • Fixed continuous addition/deletion of service routes in eBPF mode. #8988 (@sridhartigera)
  • ebpf: fix - let the node handle packet when we are not sure about the destination. #8925 (@tomastigera)
  • ebpf: If a bond master device is part of the bpfDataIfacePattern regexp, calico attaches to it and not to the slaves. #8914 (@sridhartigera)
  • ebpf: fixed forwarding, NATing and checksuming of related ICMP traffic (icmp errors). #8902 (@tomastigera)
  • Fix errors when running on a kernel that doesn’t support the XFRM netlink family. #8871 (@fasaxc)

How to update

To update an existing installation of Calico Open Source 3.28, see the upgrade guides.

Calico Open Source 3.28.2 bug fix update

September 19, 2024

Bug fixes

  • Don’t run pprof on prometheus metrics port calico #9224 (@coutinhop)
  • ebpf: Fix for Istio ambient mode - traffic that arrives from host should go back through host and not skip iptables calico #9199 (@tomastigera)
  • [etcd mode] Fix issue where Calico nodes failed to decommission if calico-kube-controllers was running on the terminated node. calico #9197 (@caseydavenport)
  • ebpf: Attach XDP to bond slave devices. calico #9143 (@sridhartigera)
  • BGP: Prevent the advertisement of local kernel routes learned from eBPF interfaces (bpf*.cali) to peers. calico #9127 (@mazdakn)
  • Fix Felix panic when using non-default BPF map sizes. Size was not updated in all places resulting in failure to attach programs. calico #9118 (@sridhartigera)
  • Fix interaction between kube-proxy and Calico’s SNAT rules that could cause corrupted VXLAN packets when checksum offload was enabled. Move Calico’s rules after kube-proxy’s to make sure kube-proxy’s mark bit is cleared if both would have done SNAT. calico #9102 (@tomastigera)
  • Fix missing resources in calioctl command help text calico #9095 (@caseydavenport)
  • ebpf: Fix parsing host IP update and re-attach program on all interfaces when there is a host IP update; fix frequently attaching BPF programs when pods annotations/labels change and eventually failing due ro running out of tc priority. calico #9094 (@sridhartigera)
  • Fix Felix panicing when trying to resync a temporary IP set. Temporary IP sets are created in certain scenarios after previous failures. calico #9078 (@fasaxc)
  • Helm: Fix error parsing kubernetesServiceEndpoint.host and kubernetesServiceEndpoint.port as an integer calico #9068 (@MichalFupso)

To update an existing installation of Calico Open Source 3.28, see the upgrade guides.