v1.28.X

Upgrade Notice

Before upgrading from earlier releases, be sure to read the Kubernetes Urgent Upgrade Notes.

Release v1.28.15+k3s1

This release updates Kubernetes to v1.28.15, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.14+k3s1:

• Add int test for flannel-ipv6masq (#10906)
• Bump Wharfie to v0.6.7 (#10977)
• Add user path to runtimes search (#11005)
• Add e2e test for advanced fields in services (#11020)
• Launch private registry with init (#11045)
• Backports for 2024-10 (#11063)
• Allow additional Rootless CopyUpDirs through K3S_ROOTLESS_COPYUPDIRS (#11042)
• Bump containerd to v1.7.22 (#11075)
• Add the nvidia runtime cdi (#11095)
• Simplify svclb ds (#11085)
• Revert “Make svclb as simple as possible” (#11115)
• Fixes “file exists” error from CNI bins when upgrading k3s (#11128)
• Update to Kubernetes v1.28.15-k3s1 and Go 1.22.8 (#11161)

Release v1.28.14+k3s1

This release updates Kubernetes to v1.28.14, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.13+k3s1:

• Testing Backports for 2024-09 (#10804)
  • Update to newer OS images for install testing
  • Fix caching name for e2e vagrant box
  • Fix deploy latest commit on E2E tests
  • DRY E2E Upgrade test setup
  • Cover edge case when on new minor release for E2E upgrade test
• Update CNI plugins version (#10820)
• Backports for 2024-09 (#10845)
• Fix hosts.toml header var (#10874)
• Update to v1.28.14-k3s1 and Go 1.22.6 (#10884)
• Update Kubernetes to v1.28.14-k3s2 (#10907)

Release v1.28.13+k3s1

This release updates Kubernetes to v1.28.13, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.12+k3s1:

• Fixing setproctitle function (#10624)
• Bump docker/docker to v24.0.10-0.20240723193628-852759a7df45 (#10651)
• Backports for 2024-08 release cycle (#10666)
  • Use pagination when listing large numbers of resources
  • Fix multiple issues with servicelb
  • Remove deprecated use of wait. functions
  • Wire lasso metrics up to metrics endpoint
• Backports for August 2024 (#10673)
• Bump containerd to v1.7.20 (#10662)
• Add tolerations support for DaemonSet pods (#10705)
  • New Feature: Users can now define Kubernetes tolerations for ServiceLB DaemonSet directly in the svccontroller.k3s.cattle.io/tolerations annotation on services.
• Update to v1.28.13-k3s1 and Go 1.22.5 (#10719)

Release v1.28.12+k3s1

This release updates Kubernetes to v1.28.12, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.11+k3s2:

• Backports for 2024-07 release cycle (#10499)
  • Bump k3s-root to v0.14.0
  • Bump github.com/hashicorp/go-retryablehttp from 0.7.4 to 0.7.7
  • Bump Local Path Provisioner version
  • Ensure remotedialer kubelet connections use kubelet bind address
  • Chore: Bump Trivy version
  • Add etcd s3 config secret implementation
• July Test Backports (#10509)
• Update to v1.28.12-k3s1 and Go 1.22.5 (#10541)
• Fix issues loading data-dir value from env vars or dropping config files (#10598)

Release v1.28.11+k3s2

This release updates Kubernetes to v1.28.11, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.11+k3s1:

• Update flannel to v0.25.4 and fixed issue with IPv6 mask (#10428)

Release v1.28.11+k3s1

This release updates Kubernetes to v1.28.11, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.10+k3s1:

• Replace deprecated ruby function (#10090)
• Fix bug when using tailscale config by file (#10144)
• Bump flannel version to v0.25.2 (#10221)
• Update kube-router version to v2.1.2 (#10182)
• Improve tailscale test & add extra log in e2e tests (#10213)
• Backports for 2024-06 release cycle (#10258)
  • Add WithSkipMissing to not fail import on missing blobs
  • Use fixed stream server bind address for cri-dockerd
  • Switch stargz over to cri registry config_path
  • Bump to containerd v1.7.17, etcd v3.5.13
  • Bump spegel version
  • Fix issue with externalTrafficPolicy: Local for single-stack services on dual-stack nodes
  • ServiceLB now sets the priorityClassName on svclb pods to system-node-critical by default. This can be overridden on a per-service basis via the svccontroller.k3s.cattle.io/priorityclassname annotation.
  • Bump minio-go to v7.0.70
  • Bump kine to v0.11.9 to fix pagination
  • Update valid resolv conf
  • Add missing kernel config check
  • Symlinked sub-directories are now respected when scanning Auto-Deploying Manifests (AddOns)
  • Fix bug: allow helm controller set owner reference
  • Bump klipper-helm image for tls secret support
  • Fix issue with k3s-etcd informers not starting
  • --Enable-pprof can now be set on agents to enable the debug/pprof endpoints. When set, agents will listen on the supervisor port.
  • --Supervisor-metrics can now be set on servers to enable serving internal metrics on the supervisor endpoint; when set agents will listen on the supervisor port.
  • Fix netpol crash when node remains tainted uninitialized
  • The embedded load-balancer will now fall back to trying all servers with health-checks ignored, if all servers have been marked unavailable due to failed health checks.
• More backports for 2024-06 release cycle (#10289)
• Add snapshot retention etcd-s3-folder fix (#10315)
• Add test for isValidResolvConf (#10302) (#10331)
• Fix race condition panic in loadbalancer.nextServer (#10323)
• Fix typo, use rancher/permissions (#10299)
• Update Kubernetes to v1.28.11 (#10347)
• Fix agent supervisor port using apiserver port instead (#10355)
• Fix issue that allowed multiple simultaneous snapshots to be allowed (#10377)

Release v1.28.10+k3s1

This release updates Kubernetes to v1.28.10, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.9+k3s1:

• Bump E2E opensuse leap to 15.6, fix btrfs test (#10095)
• Windows changes (#10114)
• Update to v1.28.10-k3s1 (#10098)

Release v1.28.9+k3s1

This release updates Kubernetes to v1.28.9, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.8+k3s1:

• Add a new error when kine is with disable apiserver or disable etcd (#9804)
• Remove old pinned dependencies (#9827)
• Transition from deprecated pointer library to ptr (#9824)
• Golang caching and E2E ubuntu 23.10 (#9821)
• Add tls for kine (#9849)
• Bump spegel to v0.0.20-k3s1 (#9880)
• Backports for 2024-04 release cycle (#9911)
  • Send error response if member list cannot be retrieved
  • The k3s stub cloud provider now respects the kubelet’s requested provider-id, instance type, and topology labels
  • Fix error when image has already been pulled
  • Add /etc/passwd and /etc/group to k3s docker image
  • Fix etcd snapshot reconcile for agentless servers
  • Add health-check support to loadbalancer
  • Add certificate expiry check, events, and metrics
  • Add workaround for containerd hosts.toml bug when passing config for default registry endpoint
  • Add supervisor cert/key to rotate list
  • The embedded containerd has been bumped to v1.7.15
  • The embedded cri-dockerd has been bumped to v0.3.12
  • The k3s etcd-snapshot command has been reworked for improved consistency. All snapshots operations are now performed by the server process, with the CLI acting as a client to initiate and report results. As a side effect, the CLI is now less noisy when managing snapshots.
  • Improve etcd load-balancer startup behavior
  • Actually fix agent certificate rotation
  • Traefik has been bumped to v2.10.7.
  • Traefik pod annotations are now set properly in the default chart values.
  • The system-default-registry value now supports RFC2732 IPv6 literals.
  • The local-path provisioner now defaults to creating local volumes, instead of hostPath.
• Allow LPP to read helper logs (#9938)
• Update kube-router to v2.1.0 (#9942)
• Update to v1.28.9-k3s1 and Go 1.21.9 (#9959)
• Fix on-demand snapshots timing out; not honoring folder (#9994)
• Make /db/info available anonymously from localhost (#10002)

Release v1.28.8+k3s1

This release updates Kubernetes to v1.28.8, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.7+k3s1:

• Add an integration test for flannel-backend=none (#9608)
• Install and Unit test backports (#9641)
• Update klipper-lb image version (#9605)
• Chore(deps): Remediating CVE-2023-45142 CVE-2023-48795 (#9647)
• Adjust first node-ip based on configured clusterCIDR (#9631)
• Improve tailscale e2e test (#9653)
• Backports for 2024-03 release cycle (#9669)
  • Fix: use correct wasm shims names
  • The embedded flannel cni-plugin binary is now built and versioned separate from the rest of the cni plugins and the embedded flannel controller.
  • Bump spegel to v0.0.18-k3s3
  • Adds wildcard registry support
  • Fixes issue with excessive CPU utilization while waiting for containerd to start
  • Add env var to allow spegel mirroring of latest tag
  • Tweak netpol node wait logs
  • Fix coredns NodeHosts on dual-stack clusters
  • Bump helm-controller/klipper-helm versions
  • Fix snapshot prune
  • Fix issue with etcd node name missing hostname
  • Rootless mode should also bind service nodePort to host for LoadBalancer type, matching UX of rootful mode.
  • To enable raw output for the check-config subcommand, you may now set NO_COLOR=1
  • Fix additional corner cases in registries handling
  • Bump metrics-server to v0.7.0
  • K3s will now warn and suppress duplicate entries in the mirror endpoint list for a registry. Containerd does not support listing the same endpoint multiple times as a mirror for a single upstream registry.
• Docker and E2E Test Backports (#9707)
• Fix wildcard entry upstream fallback (#9733)
• Update to v1.28.8-k3s1 and Go 1.21.8 (#9746)

Release v1.28.7+k3s1

This release updates Kubernetes to v1.28.7, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.6+k3s2:

• Chore: bump Local Path Provisioner version (#9426)
• Bump cri-dockerd to fix compat with Docker Engine 25 (#9293)
• Auto Dependency Bump (#9419)
• Runtimes refactor using exec.LookPath (#9431)
  • Directories containing runtimes need to be included in the $PATH environment variable for effective runtime detection.
• Changed how lastHeartBeatTime works in the etcd condition (#9424)
• Bump Flannel v0.24.2 + remove multiclustercidr (#9401)
• Allow executors to define containerd and docker behavior (#9254)
• Update Kube-router to v2.0.1 (#9404)
• Backports for 2024-02 release cycle (#9462)
• Enable longer http timeout requests (#9444)
• Test_UnitApplyContainerdQoSClassConfigFileIfPresent (#9440)
• Support PR testing installs (#9469)
• Update Kubernetes to v1.28.7 (#9492)
• Fix drone publish for arm (#9508)
• Remove failing Drone step (#9516)
• Restore original order of agent startup functions (#9545)
• Fix netpol startup when flannel is disabled (#9578)

Release v1.28.6+k3s2

This release updates Kubernetes to v1.28.6, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Important Notes

Addresses the runc CVE: CVE-2024-21626 by updating runc to v1.1.12.

Changes since v1.28.5+k3s1:

• Add a retry around updating a secrets-encrypt node annotations (#9125)
• Wait for taint to be gone in the node before starting the netpol controller (#9175)
• Etcd condition (#9181)
• Backports for 2024-01 (#9203)
• Pin opa version for missing dependency chain (#9216)
• Added support for env *_PROXY variables for agent loadbalancer (#9206)
• Etcd node is nil (#9228)
• Update to v1.28.6 and Go 1.20.13 (#9260)
• Use ipFamilyPolicy: RequireDualStack for dual-stack kube-dns (#9269)
• Backports for 2024-01 k3s2 (#9336)
  • Bump runc to v1.1.12 and helm-controller to v0.15.7
  • Fix handling of bare hostname or IP as endpoint address in registries.yaml
• Bump helm-controller to fix issue with ChartContent (#9346)

Release v1.28.5+k3s1

This release updates Kubernetes to v1.28.5, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.4+k3s1:

• Remove s390x steps temporarily since runners are disabled (#8983)
• Remove s390x from manifest (#8998)
• Fix overlapping address range (#8913)
• Modify CONTRIBUTING.md guide (#8954)
• Nov 2023 stable channel update (#9022)
• Default runtime and runtime classes for wasm/nvidia/crun (#8936)
  • Added runtime classes for wasm/nvidia/crun
  • Added default runtime flag for containerd
• Bump containerd/runc to v1.7.10-k3s1/v1.1.10 (#8962)
• Allow setting default-runtime on servers (#9027)
• Bump containerd to v1.7.11 (#9040)
• Update to v1.28.5-k3s1 (#9081)

Release v1.28.4+k3s2

This release updates Kubernetes to v1.28.4, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.3+k3s2:

• Update channels latest to v1.27.7+k3s2 (#8799)
• Add etcd status condition (#8724)
  • Now the user can see the etcd status from each node in a simple way
• ADR for etcd status (#8355)
• Wasm shims detection (#8751)
  • Automatic discovery of WebAssembly runtimes
• Add warning for removal of multiclustercidr flag (#8758)
• Improve dualStack log (#8798)
• Optimize: Simplify and clean up Dockerfile (#8244)
• Add: timezone info in image (#8764)
  • • New timezone info in Docker image allows the use of spec.timeZone in CronJobs
• Bump kine to fix nats, postgres, and watch issues (#8778)
  • Bumped kine to v0.11.0 to resolve issues with postgres and NATS, fix performance of watch channels under heavy load, and improve compatibility with the reference implementation.
• QoS-class resource configuration (#8726)
  • Containerd may now be configured to use rdt or blockio configuration by defining rdt_config.yaml or blockio_config.yaml files.
• Add agent flag disable-apiserver-lb (#8717)
  • Add agent flag disable-apiserver-lb, agent will not start load balance proxy.
• Force umount for NFS mount (like with longhorn) (#8521)
• General updates to README (#8786)
• Fix wrong warning from restorecon in install script (#8871)
• Fix issue with snapshot metadata configmap (#8835)
  • Omit snapshot list configmap entries for snapshots without extra metadata
• Skip initial datastore reconcile during cluster-reset (#8861)
• Tweaked order of ingress IPs in ServiceLB (#8711)
  • Improved ingress IP ordering from ServiceLB
• Disable helm CRD installation for disable-helm-controller (#8702)
• More improves for K3s patch release docs (#8800)
• Update install.sh sha256sum (#8885)
• Add jitter to client config retry to avoid hammering servers when they are starting up (#8863)
• Handle nil pointer when runtime core is not ready in etcd (#8886)
• Bump dynamiclistener; reduce snapshot controller log spew (#8894)
  • Bumped dynamiclistener to address a race condition that could cause a server to fail to sync its certificates into the Kubernetes secret
  • Reduced etcd snapshot log spam during initial cluster startup
• Remove depends_on for e2e step; fix cert rotate e2e (#8906)
• Fix etcd snapshot S3 issues (#8926)
  • Don’t apply S3 retention if S3 client failed to initialize
  • Don’t request metadata when listing S3 snapshots
  • Print key instead of file path in snapshot metadata log message
• Update to v1.28.4 and Go to v1.20.11 (#8920)
• Remove s390x steps temporarily since runners are disabled (#8983)
• Remove s390x from manifest (#8998)

Release v1.28.3+k3s2

This release updates Kubernetes to v1.28.3, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.3+k3s1:

• Restore selinux context systemd unit file (#8593)
• Update channel to v1.27.7+k3s1 (#8753)
• Bump Sonobuoy version (#8710)
• Bump Trivy version (#8739)
• Fix: Access outer scope .SystemdCgroup (#8761)
  • Fixed failing to start with nvidia-container-runtime
• Upgrade traefik chart to v25.0.0 (#8771)
• Update traefik to fix registry value (#8792)
• Don’t use iptables-save/iptables-restore if it will corrupt rules (#8795)

Release v1.28.3+k3s1

This release updates Kubernetes to v1.28.3, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.2+k3s1:

• Fix error reporting (#8250)
• Add context to flannel errors (#8284)
• Update channel, September patch release (#8397)
• Add missing link to drone in documentation (#8295)
• Include the interface name in the error message (#8346)
• Add extraArgs to vpn provider (#8354)
  • Allow to pass extra args to the vpn provider
• Disable HTTP on main etcd client port (#8402)
  • Embedded etcd no longer serves http requests on the client port, only grpc. This addresses a performance issue that could cause watch stream starvation under load. For more information, see https://github.com/etcd-io/etcd/issues/15402
• Server token rotation (#8215)
• Fix issues with etcd member removal after reset (#8392)
  • Fixed an issue that could cause k3s to attempt to remove members from the etcd cluster immediately following a cluster-reset/restore, if they were queued for removal at the time the snapshot was taken.
• Fix gofmt error (#8439)
• Added advertise address integration test (#8344)
• Added cluster reset from non bootstrap nodes on snapshot restore e2e test (#8292)
• Fix .github regex to skip drone runs on gh action bumps (#8433)
• Added error when cluster reset while using server flag (#8385)
  • The user will receive a error when —cluster-reset with the —server flag
• Update kube-router (#8423)
  • Update kube-router to v2.0.0-rc7 to fix performance issues
• Add SHA256 signatures of the install script (#8312)
  • • Add SHA256 signatures of the install script.
• Add —image-service-endpoint flag (#8279)
  • Add --image-service-endpoint flag to specify an external image service socket.
• Don’t ignore assets in home dir if system assets exist (#8458)
• Pass SystemdCgroup setting through to nvidia runtime options (#8470)
  • Fixed issue that would cause pods using nvidia container runtime to be killed after a few seconds, when using newer versions of nvidia-container-toolkit.
• Improve release docs - updated (#8414)
• Take IPFamily precedence based on order (#8460)
• Fix spellcheck problem (#8507)
• Network defaults are duplicated, remove one (#8523)
• Fix slemicro check for selinux (#8526)
• Update install.sh.sha256sum (#8566)
• System agent push tags fix (#8568)
• Fixed tailscale node IP dualstack mode in case of IPv4 only node (#8524)
• Server Token Rotation (#8265)
  • Users can now rotate the server token using k3s token rotate -t <OLD_TOKEN> --new-token <NEW_TOKEN>. After command succeeds, all server nodes must be restarted with the new token.
• E2E Domain Drone Cleanup (#8579)
• Bump containerd to v1.7.7-k3s1 (#8604)
• Bump busybox to v1.36.1 (#8602)
• Migrate to using custom resource to store etcd snapshot metadata (#8064)
• Switch build target from main.go to a package. (#8342)
• Use IPv6 in case is the first configured IP with dualstack (#8581)
• Bump traefik, golang.org/x/net, google.golang.org/grpc (#8624)
• Update kube-router package in build script (#8630)
• Add etcd-only/control-plane-only server test and fix control-plane-only server crash (#8638)
• Use version.Program not K3s in token rotate logs (#8653)
• [Windows Port (#7259)
• Fix CloudDualStackNodeIPs feature-gate inconsistency (#8667)
• Re-enable etcd endpoint auto-sync (#8675)
• Manually requeue configmap reconcile when no nodes have reconciled snapshots (#8683)
• Update to v1.28.3 and Go to v1.20.10 (#8682)
• Fix s3 snapshot restore (#8729)

Release v1.28.2+k3s1

This release updates Kubernetes to v1.28.2, and fixes a number of issues.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.28.1+k3s1:

• Update channel for version v1.28 (#8305)
• Bump kine to v0.10.3 (#8323)
• Update to v1.28.2 and go v1.20.8 (#8364)
  • Bump embedded containerd to v1.7.6
  • Bump embedded stargz-snapshotter plugin to latest
  • Fixed intermittent drone CI failures due to race conditions in test environment setup scripts
  • Fixed CI failures due to changes to api discovery changes in Kubernetes 1.28

Release v1.28.1+k3s1

This release is K3S’s first in the v1.28 line. This release updates Kubernetes to v1.28.1.

Important

This release includes remediation for CVE-2023-32187, a potential Denial of Service attack vector on K3s servers. See https://github.com/k3s-io/k3s/security/advisories/GHSA-m4hf-6vgr-75r2 for more information, including documentation on changes in behavior that harden clusters against this vulnerability.

Critical Regression

Kubernetes v1.28 contains a critical regression (kubernetes/kubernetes#120247) that causes init containers to run at the same time as app containers following a restart of the node. This issue will be fixed in v1.28.2. We do not recommend using K3s v1.28 at this time if your application depends on init containers.

For more details on what’s new, see the Kubernetes release notes.

Changes since v1.27.5+k3s1:

• Update to v1.28.1 (#8239)
• CLI Removal for v1.28.0 (#8203)
• Secrets Encryption V3 (#8111)
• Add new CLI flag to disable TLS SAN CN filtering (#8252)
  • Added a new --tls-san-security option.
• Add RWMutex to address controller (#8268)