Release notes for kOps 1.24 series

Significant changes

Hetzner Cloud support

Adds experimental Hetzner Cloud support. To get started check the docs

Karpenter support

By enabling the Karpenter feature flag, users can now create InstanceGroups managed by Karpenter:

spec:
  manager: Karpenter

You can also start a Karpenter-only cluster with kops create cluster --instance-manager=karpenter ...

kOps will directly manage the Karpenter Provisioner resources. Read more about how Karpenter works on kOps in the Karpenter docs.

Other significant changes

• Adds support for Ubuntu 22.04

• Adds support for Rocky Linux 8

• Adds support for overriding the Kubernetes version when upgrading a cluster by using the --kubernetes-version flag.

• The minimum version for the Terraform AWS Provider has been bumped to 4.0.0 to address the deprecation of the aws_s3_bucket_object resource and its replacement with the aws_s3_object resource. Such resources will be destroyed and recreated without downtime when applying the changes.

• ARM64 support for nvidia device driver. Nvidia nodes on ARM64 requires Ubuntu 22.04 AMIs.

Breaking changes

• The nfs-common/nfs-utils package is no longer installed by default. Use the packages option at instance group level to add it back.

Control plane taints and labels

As of Kubernetes version 1.24, the control plane (formerly master) nodes no longer have the deprecated node-role.kubernetes.io/master label. The deprecated node-role.kubernetes.io/master taint has been replaced by node-role.kubernetes.io/control-plane. If you run your own workload on the control plane, you have to adjust your Pod spec to accommodate for this change.

The following shows a node affinity/node label selector and tolerations that works with both new and old control plane nodes:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
        - matchExpressions:
          - key: node-role.kubernetes.io/master
            operator: Exists
  tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
  - key: node-role.kubernetes.io/master
    operator: Exists

Removing the kubernetes.io/role label

The deprecated kubernetes.io/role label has been removed for all roles as of Kubernetes version 1.24. Use node-role.kubernetes.io/<role> label instead.

Cert Manager removes old API versions

Cert Manager upgraded from 1.6 to 1.8. This has backwards-breaking changes. See upgrading from 1.6 to 1.7 and [1.1.7 to 1.8.

In particular, if you are using the snapshot-controller addon, upgrade your cluster to kOps 1.23 before upgrading to kOps 1.24 to ensure the certificate has the correct API version.

Other breaking changes

• Support for Kubernetes version 1.18 has been removed.

• Support for Aliyun/Alibaba Cloud has been removed.

• Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq

• Cluster Autoscaler can no longer run on spot instances. This is to avoid cluster autoscaler not being scheduled if the spot instances terminate.

Required actions

Deprecations

• Support for Kubernetes version 1.19 is deprecated and will be removed in kOps 1.25.

• Support for Kubernetes version 1.20 is deprecated and will be removed in kOps 1.26.

• All legacy addons are deprecated in favor of managed addons, including the metrics server addon and the autoscaler addon.

• Due to lack of maintainers, the CloudFormation support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this target.

• Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq

• The minimum supported version of the AWS EBS CSI Driver is now v1.3.1.

Other changes of note

Full change list since 1.23.0 release

1.24.0-alpha.1 to 1.24.0-alpha.2

• Update release notes and minimum k8s version @hakman #12929
• kops auth-plugin: need to clear any existing password / key @justinsb #12921
• Add integration test for k8s 1.24 @olemarkus #12930
• Only shellcheck files @olemarkus #12931
• Do not set insecure-port as of k8s 1.20 @olemarkus #12926
• tests: Improve logging on test failure @justinsb #12933
• nodeup: store the CloudProvider in the context @justinsb #12923
• bazel: always build with pure (CGO_ENABLED=0) @justinsb #12934
• nodeup: print more info on hash mismatches @justinsb #12935
• PKI library: Add initial support for EC keys @justinsb #12936
• Recognize debian bullseye as having “broken” resolv.conf @justinsb #12937
• Remove code for now-unsupported Kubernetes 1.18 @johngmyers #12939
• Add missing k8s 1.18 relnote @johngmyers #12938
• Remove obsolete, redundant secrets.md @johngmyers #12942
• Drop support for Weave as of k8s 1.23 @johngmyers #12941
• Remove support for Aliyun/Alibaba Cloud @johngmyers #12944
• Document CoreDNS configuration settings @recollir #12914
• Update name of kubernetes-ca keypair in documentation @johngmyers #12943
• Revert “Recognize debian bullseye as having “broken” resolv.conf” @olemarkus #12947
• Set the default LT version to the new LT version @olemarkus #12932
• Make service topology for cilium configurable @olemarkus #12918
• gce: ServiceAccount task @justinsb #12950
• Update Calico and Canal to v3.21.2 @hakman #12951
• Update Go to v1.17.5 @hakman #12954
• Skip IPv6 LB test in the k/s e2e @hakman #12953
• GCE: Task for StorageBucket IAM @justinsb #12958
• GCE: Project IAM Binding task @justinsb #12959
• add verify-golangci-lint.sh script @rlankfo #12892
• Hubble relay should not tolerate anything @olemarkus #12963
• Do not explicitly skip Dashboard tests @hakman #12962
• Do not skip NodePort tests for Calico @hakman #12960
• Remove verify-staticcheck @rifelpet #12965
• wait for instances to drain from classic LB @heybronson #12902
• Support Karpenter @olemarkus #12906
• Update containerd to v1.6.0-beta.4 @hakman #12968
• Update controller-runtime to v0.11.0 @hakman #12967
• Add missing permissions @olemarkus #12977
• Do not skip HPA tests @hakman #12972
• Do not skip RuntimeClass tests @hakman #12974
• gce: Use ServiceAccount task when building model @justinsb #12978
• Quote values and remove limits in karpenter provisioners @olemarkus #12979
• Promote alpha with December releases @olemarkus #12984
• gce: map multiple serviceaccounts @justinsb,@hakman #12982
• Defend against nil containerd @justinsb #12990
• Remove unused TemplateResource interface @justinsb #12989
• Avoid double-encoding templates @justinsb #12991
• Refactor nodeup script to avoid action-at-a-distance @justinsb #12993
• gce: use per InstanceGroup serviceaccounts @justinsb #12988
• dep: update github.com/pkg/sftp @justinsb #12996
• Create helper functions for parsing public keys @justinsb #12999
• Use terraform literals in GCP service account references @rifelpet #12995
• kops-controller: use controller-runtime manager @justinsb #12997
• gce: clean up networking objects by reference @justinsb #12987
• componentconfig: expose advertise-address flag for kube-apiserver @justinsb #12998
• Do not allow docker on k8s 1.24+ @olemarkus #12927
• Ignore images hosted in private ECR repositories as containerd cannot pull these @olemarkus #13000
• Skip RuntimeClass tests for older Kubernetes versions @hakman #13003
• Various nill pointer fixes for karpenter @olemarkus #12973
• Set Resource Based Naming on managed subnets @johngmyers #12864
• Add kubetest2-kops flags for overriding instance group fields @rifelpet #13005
• Support creating dualstack internal NLBs @johngmyers #13006
• Skip SCTP check for all versions of k8s 1.23/1.24 @olemarkus #13008
• Use spread constraints rather than affinity to spread pods @olemarkus #12961
• Bump karpenter to 0.5.3 and RBN support @olemarkus #13002
• Validate IGs more strictly after defaults have applied @olemarkus #12660
• Karpenter template fix @olemarkus #13009
• staticcheck cleanup: fixup nodeup/pkg/model @justinsb #13013
• nodeup bash script: use explicit return code @justinsb #13012
• Prevent creation of unsupported etcd clusters @olemarkus #13011
• Create cgroups for kube and runtime if configured @olemarkus #12917
• Do not install ClusterRole and binding used by in-tree volume provider if CSI is used @olemarkus #13010
• kubetest2 - Use the same binary path and env when fetching IGs @rifelpet #13018
• Use fi.Keyset instead of passing tasks around @justinsb #12992
• add instance connection draining for NLBs @heybronson #12966
• Use kubelet —non-masquerade-cidr only for Docker with kubenet @hakman #13007
• Fix dangling ENIs from AWS VPC CNI @olemarkus #13021
• Update k8s dependencies to v1.23.1 @hakman #13022
• Improve HA for various addons @olemarkus #13027
• Add a CLI flag for creating one karpenter-managed IG for worker nodes instead of ASG-managed ones @olemarkus #12975
• Allow IPv6-only subnets @johngmyers #13026
• Support specifying instance requirements per IG @olemarkus #13019
• Remove TerraformJSON feature flag @rifelpet #13029
• LBC has to run on the control plane, so set replicas accordingly @olemarkus #13033
• Fix various typos related to karpenter @olemarkus #13035
• Kube components log to stdout @olemarkus #13038
• Identify pending instances @olemarkus #13040
• Add managed-by label to static kube-proxy pods @olemarkus #13039
• Prefix karpenter logging-config name @olemarkus #13037
• gce: don’t set per-IG permissions when using shared account @justinsb #13043
• Add documentation on karpenter @olemarkus #13036
• external CCM for GCE @jiahuif #13017
• Migrate to GCE CCM in k8s 1.24 @johngmyers #13045
• Fix OpenStack SecurityGroupRule/LB When CIDR is IPv6 @iGene #13032
• update deps @zetaab #13047
• Bump Cluster Autoscaler and update manifest @olemarkus #13050
• Use instance requirements with Karpenter @olemarkus #13031
• force update dependencies @zetaab #13055
• Enhance AddHostPathMapping to support a fluent style @justinsb #13062
• addons: support for kopeio-networking addon @justinsb #12727
• Use latest GCP CCM for k8s 1.24 @johngmyers #13066
• Add action for automatically tagging releases @johngmyers #12805
• Bump external-snapshotted to v5.0.0 @olemarkus #13067
• Release 1.24.0-alpha.2 @johngmyers #13069

1.24.0-alpha.2 to 1.24.0-alpha.3

• Release notes for 1.24.0-alpha.2 @johngmyers #13070
• Update release process for automatic tagging @johngmyers #13075
• Remove temporary restrictions on automatically tagging releases @johngmyers #13071
• add flatcar note related to additionalUserData @shubhindia #13061
• Drain OpenStack loadbalancers @zetaab #12983
• Extend terraform support for IPv6 @rifelpet #13028
• Update containerd to v1.6.0-beta.5 @hakman #13084
• Release notes for 1.22.3 @johngmyers #13085
• Spotinst: Update spotinst/ocean-controller to v1.0.81 @liranp #13086
• Support price and priority cluster-autoscaler expanders @danports #13081
• Update containerd to v1.6.0-rc.0 @hakman #13098
• decrease the openstack monitoring default timeout @zetaab #13097
• Don’t try to add node name to instances without node object @olemarkus #13106
• fix ipv4+ipv6 sec groups/listeners in OpenStack @zetaab #13093
• Do not create an IAM role for dns-controller on gossip clusters @olemarkus #13110
• Add ipv6 to relnotes @olemarkus #13088
• Use IPv6-only subnets for worker nodes in private IPv6 topology @johngmyers #13030
• Remove networking flags as of k8s 1.24 @olemarkus #13120
• Create helper function for ec2 create/tag-on-create IAM permissions @olemarkus #13104
• Add DescribeRegions to nodeup privs @olemarkus #13114
• Remove featureflag for creating IPv6 clusters @hakman #12788
• Preload channel versions from namespaces @olemarkus #13049
• Don’t set unsupported configs by default @olemarkus #13111
• Update pause image to v3.6 @hakman #13125
• Clean up kubelet networking flags for dockershim @hakman #13128
• January bump of channels @olemarkus #13130
• expose external ccm metrics for OpenStack @zetaab #13131
• Update to aws-sdk-go to v1.42.37 @jinhong- #13132
• Fix recommended kops versions in channels @olemarkus #13134
• Tag on create for remaining CCM privileges @olemarkus #12911
• Bump metrics-server to 0.6.0 and enable HA mode @olemarkus #13135
• OpenStack - Add loadbalancer pool monitor to API LB @zetaab #13096
• Bump CCM images @olemarkus #13143
• Bump karpenter to 0.5.6 @olemarkus #13151
• Promote alpha AMIs to stable @yurrriq #13152
• Bump 1.23 version in alpha channel @olemarkus #13153
• Add missing v prefix to default upgrade test version @olemarkus #13155
• Bump cert-manager and related godep to 1.6.2 @olemarkus #13154
• add node-drain-timeout flag to rolling-update @heybronson #13103
• Bump etcd-manager to v3.0.20220128 @olemarkus #13158
• Replace deprecated aws.BackgroundContext with context.Background @justinsb #13162
• Fix nil pointer when IAM not populated @justinsb #13167
• JWKS / IRSA: Expose public ACLs to terraform @justinsb #13166
• [DigitalOcean] update ccm version to 0.1.36 @srikiz #13175
• Bump Ubuntu AMI in alpha @olemarkus #13177
• Use etcd-manager pre-release until final release has been cut @olemarkus #13183
• Bump karpenter to 0.6.0 @olemarkus #13185
• More descriptive error message when public key file can’t be opened @nckturner #13186
• update GCE default images @zetaab #13181
• Fix etcd-manager for ipv6 @olemarkus #13191
• Update Calico and Canal to v3.21.4 @hakman #13189
• Update to etcd-manager v3.0.20220203 @justinsb #13196
• Pull k8s-custom-iptables from k8s.gcr.io @justinsb #13194
• Add support for AB tests starting out with released kops version @olemarkus #13174
• Update containerd to v1.6.0-rc.2 @hakman #13198
• tests: ensure that we use ACLs with memfs @justinsb #13165
• Karpenter fixes @olemarkus #13207
• Always enable Leader Election for cloud-controller-manager @jiahuif #13187
• Use short commit sha for default stage location instead of git-describe @olemarkus #13208
• use 1.23.1 ccm for openstack @zetaab #13136
• Document download of test versions @olemarkus #13209
• Remove snapshot controller dependency on ebs csi driver @olemarkus #13213
• fix KCM LogLevel setting not honored @jiahuif #13218
• Fix CSI migration feature gates @olemarkus #13203
• CCM: use flagbuilder instead of manually building argv @jiahuif #13219
• Update containerd to v1.6.0-rc.3 @hakman #13224
• Promote alpha to stable @MoShitrit #13227
• always enable Leader Election for openstack CCM @jiahuif #13220
• Update aws node termination handler to 1.14.0 @ryan-dyer-sp,@ryan-dyer #13092
• [Issue-12293] Fix json output to keep it consistent for single or multiple objects @srikiz #13188
• Fix irsa for k8s < 1.20 @olemarkus #13212
• enable pruning for CCM @jiahuif #13235
• Add support for graceful node shutdown @olemarkus #12994
• allow specify GCP project via env. @jiahuif #13237
• KCM should not run with leader migraton when aws ccm is enabled @olemarkus #13241
• Do not enable graceful shutdown if k8s version < 1.21 @olemarkus #13242
• Update metrics-server e2e test for 0.6.0 @olemarkus #13243
• Install runc from opencontainers/runc @hakman #13240
• Fix nilpointer when graceful shutdown is not configured @olemarkus #13246
• Install contained from the release package @hakman #13248
• CCM: allow setting Controllers for cloudControllerManagerConfig @jiahuif #13252
• CCM: add livenessProbe for GCP CCM @jiahuif #13253
• E2E HA Upgrade/Rollback for Leader Migration @jiahuif #13251
• Bump AWS CNI to 1.10.2 @MoShitrit #13228
• Update supported distros for IPv6 @hakman #13256
• Karpenter on kOps will now use approperiate max pods @olemarkus #13178
• Allow PrefixList for sshAccess and kubernetesApiAccess @hierynomus #13113
• service account workaround for gce @jiahuif #13261
• GCP API health checks @zetaab #13199
• Update containerd to v1.6.0 @hakman #13262
• re-organize Leader Migration test with exec tester @jiahuif #13265
• Update LBC to 2.4.0 @olemarkus #13267
• Enable RBN with AWS CCM 1.22.0-alpha.1 @johngmyers #13268
• Disable some flags in kube-apiserver when logging-format is not text @h3poteto #13264
• kops: Leader Migration testing: run with pure kubetest2 @jiahuif #13276
• Bump k8s versions in alpha with Feb 2022 releases @MoShitrit #13275
• Validate taints in IG spec @olemarkus #13266
• test: use T.TempDir to create temporary test directory @Juneezee #13283
• Do not create a cert-manager namespace @olemarkus #13284
• Add missing permissions to aws lbc for irsa @olemarkus #13280
• [DigitalOcean] Implement new VPC if network-cidr flag is specified @srikiz #13060
• Use current tree in presubmit upgrade jobs if version B is latest @olemarkus #13290
• Release notes for 1.22.4 @justinsb #13294
• alpha channel: recommend kOps 1.22.4 @justinsb #13296
• docs: add hubble ui helm chart deployment @eddycharly #13299
• cleanup GCP Cluster Service Accounts @zetaab #13201
• docs for release process shouldn’t assume remotes @justinsb #13295
• Release notes for 1.23.0-beta.2 @hakman #13303
• Add support to install EKS Pod Identity Webhook @h3poteto,@olemarkus #13176
• Update kubetest2 deps @olemarkus #13314
• use own function to define CSI image version @zetaab #13311
• Add support for ed25519 keys in AWS @aclevername #13304
• Bump AWS SDK to v1.43.11 @olemarkus #13322
• Make cloudProvider a struct in v1alpha3 API @johngmyers #13059
• Update containerd to v1.6.1 @hakman #13325
• Fix GCE service account creation @zetaab #13310
• Use proper image and add health check @olemarkus #13328
• Update stable and alpha channels @olemarkus #13334
• Release notes for 1.21.5 @hakman #13336
• Add e2e for pod identity webhook @olemarkus #13335
• Add webhook notes + some docs changes @olemarkus #13338
• Only delete node object on GCE @olemarkus #13289
• Release notes for 1.23.0 @hakman #13340
• Bump AWS CCM to 1.22.0-alpha.2 @olemarkus #13342
• Bump CCM 1.22 image. Use the 1.23 image for 1.24 due to latest being broken @olemarkus #13357
• Update channels @hakman #13356
• Recommend enabling IRSA for new clusters @olemarkus #12976
• Post 1.23.0 release doc updates @johngmyers #13359
• Add user to container securityContext and remove command @olemarkus #13343
• [Digital Ocean] e2e tests - Fix seeding for generating random zones @srikiz #13362
• wait for all targetGroups to drain @heybronson #13363
• Support GPU in OpenStack @zetaab #13330
• Add missing permissions to aws lbc for IP targeting @olemarkus #13369
• If kubetest2 fails cluster validation, we run down before exiting @olemarkus #13373
• If image is empty, have kops upgrade fill it in @olemarkus #13374
• Update channels @hakman #13379
• Update HPA docs @ddelange #13367
• Clean up nodeup targets @olemarkus #13370
• Upgrade aws-iam-authenticator to v0.5.5 @glebiller #13381
• Add protocol explicitly to services @olemarkus #13383
• Allow duplicate taint keys @olemarkus #13366
• [Digital Ocean] Remove sfo2 region from the list of supported DO regions @srikiz #13382
• Fix long role names @olemarkus #13364
• Migrate to registry.k8s.io @hakman #13380
• Remove oss-upload target since aliyun support has been removed @olemarkus #13389
• dev: create scripts to make it easier to run e2e tests @justinsb #13161
• Remove pr target @olemarkus #13392

1.24.0-alpha.3 to 1.24.0-alpha.4

• Release notes for 1.24.0-alpha.3 @heybronson #13375
• Spotinst: Ocean as a template for VNGs @liranp #13234
• [Digital Ocean] Use available regions as per the latest DO documentation @srikiz #13394
• Build kops images with ko @olemarkus #13341
• Clean up kops build targets @olemarkus #13390
• update k8s dependencies @heybronson #13397
• Update golangci-lint to v1.45.0 @hakman #13403
• Use k8s.gcr.io for k8s side-loaded images @hakman #13402
• Use ko-builds for dev-upload @olemarkus #13401
• Clean up channels and protokube targets @olemarkus #13404
• Don’t use bazel for dev-upload @olemarkus #13407
• Clean up the rest of the dev-upload targets @olemarkus #13408
• AWS-NODE-TERMINATION: Add possibility to set a tune image version @anthonyhaussman #13405
• Update netflify deps @hakman #13412
• Promote alpha to stable @MoShitrit #13415
• Correctly detect GovCloud regions @mixja #13410
• dev-upload from upload directory, not bazelupload @olemarkus #13419
• fix member update when draining @zetaab #13414
• Update the security docs @olemarkus #13421
• Remove id var, which is never used beyond that if test @olemarkus #13420
• Pin actions to a full length commit SHA @naveensrinivasan #13395
• Digest images during remapping @olemarkus #13422
• Do not return a ‘-1’ exit if no keys found and json/yaml output @hierynomus #13378
• Use non-bazel builds in kubetest2 @olemarkus #13409
• Run CSI controller in masters @zetaab #13426
• Push ko-built images to staging @olemarkus #13428
• Included githubactions in dependabot config @naveensrinivasan #13423
• Bump actions/setup-go from 2.2.0 to 3 @dependabot[bot],@hakman #13429
• Use golang 1.18 for building @olemarkus #13430
• Use golang 1.18.0 explicitly @hakman #13432
• Install gcloud instead of gsutil @hakman #13434
• Use the correct upload folder for the latest.txt file @olemarkus #13435
• Fix upload dir in cloudbuild-artifacts target @olemarkus #13437
• Skip upstream e2e log dump @olemarkus #13438
• Remove direct dependency on klog v1 @olemarkus #13447
• Bump semver for kubetest2 @olemarkus #13446
• fix dns controller crashing for DO @zak905 #13443
• Add possibility to set PodAnnotations into NodeLocalDNS @anthonyhaussman #13396
• Disable CGO and enable trimpath @olemarkus #13451
• Possibility to add additional routes in route tables of subnets @guillomep #13318
• Update containerd to v1.6.2 @hakman #13455
• Don’t run the CSI snapshot plugin if snapshot controller is not installed @olemarkus #13453
• Add back hash for containerd v1.6.1 @hakman #13462
• Use Cilium 1.11 as default @olemarkus #12919
• Bump Cert Manager to 1.8 @olemarkus #13464
• Pick the right OS server group when creating cloud groups @ederst #13461
• Don’t hash parts of the cluster name for e2e @olemarkus #13354
• Enable etcd corruption check as mitigatio of 3.5 corruption issue @olemarkus #13454
• Move Azure settings to cloudProvider.azure @johngmyers #13065
• Add PDBs to addons where this was missing @olemarkus #13475
• Bump NTH 1.16 and add excludeFromLoadBalancers option @DingGGu #13467
• Move Openstack settings to cloudProvider.openstack @johngmyers #13326
• Bump viper and cobra @olemarkus #13482
• Bump cert-manager deps @olemarkus #13481
• Remove explicit dependency on yamlv2 @olemarkus #13483
• Use latest stable release by default for scenario test @olemarkus #13476
• Remove GOPATH dependency for apimachinery @hakman #13472
• Document NodeLocalDNS forwardToKubeDNS breaking change @jorge07 #13448
• Fix typos in docs/getting_started/arguments @scottchiang #13485
• Update kubetest2 @olemarkus #13486
• Bump Ubuntu AMIs in alpha and stable @olemarkus #13487
• Bump AWS CNI to version 1.10.3 @MoShitrit #13488
• Use aws_s3_object instead of deprecated aws_s3_bucket_object @hakman #13491
• Docs getting started aws @anthonytwh #13489
• Update Calico and Canal to v3.21.5 @hakman #13497
• Update to etcd-manager 3.0.20220417 @justinsb #13499
• Revert “Enable etcd corruption check” @hakman #13495
• etcd 3 5 3 @justinsb #13501
• Remove bazel targets and tools @olemarkus #13484
• Bump CCM 1.22 and 1.23 images to stable versions @olemarkus #13506
• Use Cluster Autoscaler 1.23 for k8s 1.24 @olemarkus #13510
• Adopt control-plane taint and remove master role labels @olemarkus #13452
• Update aws-sdk-go to v1.43.41 @hakman #13515
• Revert to using 1.23.0-alpha.0 for AWS CCM @hakman #13514
• add cluster autoscaler pod annotations @heybronson #13511
• Trim GCE firewall rule names to their max length @rifelpet #13513
• kubetest2-kops - Create ephemeral SSH keys @rifelpet #13522
• Skip topology hints tests in k8s 1.23 @rifelpet #13524
• Update remaining addon manifests for the control-plane node role @rifelpet #13521
• Release notes for 1.22.5 @olemarkus #13528
• Release notes for 1.23.1 @olemarkus #13531
• Allow cluster autoscaler to read EC2 instance types to build catalog dynamically @seh #13532
• Use expected pointer type in type assertion when iterating over GS ACLs @tesspib #13534
• Shell out to ssh-keygen for creating ed25519 keys @rifelpet #13538
• Update control plane toleration for external-dns @rifelpet #13539
• Pin AWS CCM image tag for k8s 1.25 @rifelpet #13543
• fix pod annotations in addon yamls @heybronson #13536
• Skip SCTP tests in k8s 1.25 as well @rifelpet #13545
• Bump k8s deps to 1.24.0-rc.0 @olemarkus #13548
• Trim GCE Subnet and Disk names @rifelpet #13546
• Update codegen to v1.24.0-rc.0 @hakman #13549
• Update Go to v1.18.1 @hakman #13550
• Upgrade flannel to 0.17.0 @rifelpet #13552
• Always set cluster-id flag for Protokube @hakman #13555
• Creaate kubeconfig for the root user on Ubuntu @hakman #13553
• Bump alpha channel with april releases @olemarkus #13556
• Enable internal load balancers when running on GCP @nat-henderson #13445
• Add back support for Ubuntu 18.04 @hakman #13557
• Add support for Rocky Linux 8 @hakman #13559
• Re-add net.bridge settings for flannel @rifelpet #13564
• Update Canal’s Flannel to v0.15.1 @tesspib #13562
• Bump AWS CNI to version 1.11.0 @MoShitrit #13493
• Fix OIDC Provider cleanup @rifelpet #13571
• Include sysctls in toolbox dump @rifelpet #13570
• Add NodeTerminationHandler CPU request settings @glebiller #13566
• Add docs for Rocky Linux 8 @hakman #13561
• Don’t digest side-loaded images @hakman #13576
• Update containerd to v1.6.3 @hakman #13578
• Upgrade EBS CSI Driver to v1.6.0 @rifelpet #13565
• Bump nvidia device plugin to 0.11 and nvidia driver to 510 @olemarkus #13580
• Make kOps version show server version too @olemarkus #12327
• Spotinst: Add support for EBS root volume encryption @liranp #13585
• promote alpha to stable (k8s versions bump) @MoShitrit #13587
• Remove remaining bazel references @rifelpet #13586
• Revert containerd v1.6.3 upgrade @rifelpet #13582
• Update etcd to v3.5.4 @hakman #13588
• Bump cilium to 1.11.4 @olemarkus #13581
• Update golangci-lint to v1.45.2 @hakman #13584
• Don’t install nfs-common/nfs-utils by default @hakman #13577
• doc(mfa): add another workaround using aws-vault @techieforfun #13591
• add priorities to CSI openstack @zetaab #13594
• Fix unexpected type for object metadata when using gossip DNS @hakman #13592
• fix immutable retry logic when updating member @zetaab #13595
• Add support for installing additional packages @hakman #13590
• Update containerd to v1.6.4 @hakman #13596
• Update etcd-manager to v3.0.20220503 @hakman #13598
• Bump k8s.io deps to 1.24.0 @olemarkus #13601
• update deps @zetaab #13602
• update OpenStack CSI & cloud-controller manifest @zetaab #13603
• Add hashes for containerd and Docker in order to fix CVE-2022-23648 @drequena #13606
• Add IRSA for kube-router @rifelpet #13609
• Avoid “/etc/resolv.conf” file loopback for Flatcar Container Linux distribution @seh #13617
• Skip in-tree volume limits test @rifelpet #13621
• Add initial support for Hetzner Cloud @hakman #13572
• remove bind address for occm @zetaab #13623
• Add docs for the packages option @hakman #13622
• Update Calico and Canal to v3.23.0 @hakman #13625
• Use latest staging AWS CCM for k8s 1.24+ @hakman #13626
• Allow returning multiple addresses for API @hakman #13624
• [DigitalOcean]: make droplet image constant when building droplet in for fitask @zak905 #13628
• Upgrade Karpenter to 0.10.0 @rifelpet #13547
• Add load balancer support for Hetzner @hakman #13629

1.24.0-alpha.4 to 1.24.0-alpha.5

• Release notes for 1.23.2 @olemarkus #13632
• Release notes for 1.24.0-alpha.4 @hakman #13634
• Use fixed staging AWS CCM for k8s 1.24+ @hakman #13637
• Update runc to v1.1.2 @hakman #13638
• add annotation support to aws-ebs-csi pods @avish42 #13600
• Bump cluster autoscaler image for 1.24 @olemarkus #13642
• Ignore digest for kube-apiserver-healthcheck model test @hakman #13639
• Update documentation for manageStorageClasses @minkimipt #13641
• fix typo in docs/instance-groups.md @marquezika #13635
• Fix code block in manageStorageClasses docs @minkimipt #13645
• [Digital Ocean] Add CSI driver for DO block storage @srikiz #13643
• Update AWS CCM to 13.05.2022 releases @hakman #13644
• Skip nfs tests in all scenarios @hakman #13648
• Tweak integration tests to be simpler @justinsb #13647
• Use Calico v3.21 with older versions of k8s @hakman #13649
• Use -ginkgo.junit-report instead of -ginkgo.reportFile @hakman #13650
• Allow editing clusters with Hetzner @hakman #13654
• Bump actions/setup-go from 3.0.0 to 3.1.0 @dependabot[bot] #13653
• Promote May AMIs in channels @olemarkus #13658
• Allow the AWS Load Balancer Controller to use WAFs @seh #13636
• Update controller-runtime to v0.12.0 @hakman #13655
• Set nodeSelector and command to null @zetaab #13656
• Add a nameservers parameter for cert-manager. @jim-barber-he #13567
• Allow overriding the kubernetes version when upgrading the cluster @hakman #13652
• Bump EBS CSI driver to 1.6.1 @olemarkus #13664
• Bump Load Balancer Controller to 2.4.1 @olemarkus #13665
• Bump Cilium to 1.11.5 @olemarkus #13666
• Add missing backslash to aws.md @piec #13669
• Bump NTH to 1.16.4 and add support for scheduled instance change events @olemarkus #13662
• Bump EBS CSI driver to 1.6.2 @olemarkus #13670
• Bump metrics-server to 0.6.1 @olemarkus #13674
• Bump external-dns to 0.11.0 @olemarkus #13673

1.24.0-alpha.5 to 1.24.0-beta.1

• Release notes for 1.24.0-alpha.5 @olemarkus #13676
• Bump coredns to 1.8.6 @olemarkus #13677
• Add Support for OVN Load Balancer for OpenStack @ching-kuo,@zetaab #13678
• Add release note and getting started entries for Hetzner @hakman #13680
• Fix kops update for OpenStack with LB @ching-kuo #13682
• Update Calico and Canal to v3.23.1 @hakman #13672
• bump alpha k8s versions with May releases @MoShitrit #13683
• Update troubleshoot.md @simonccc #13685
• Add support for configuring which metrics cilium will export @olemarkus #13684
• Remove unused DNS logic from Protokube @hakman #13689
• Fix Protokube gossip flag @hakman #13692
• Bump actions/setup-go from 3.1.0 to 3.2.0 @dependabot[bot] #13694
• Refactor cloud providers and remove unused code from Protokube @hakman #13691
• Use dependabot for Go deps @hakman #13696
• Fix links to go docs @diversario #13705
• Update gophercloud to v0.25.0 @ching-kuo #13710
• Use build tags for Protokube @hakman #13706
• Clarify difference between terraform and kOps state stores @hakman #13709
• Bump node termination handler to 1.16.5 @olemarkus #13711
• promote alpha to stable (k8s releases) @MoShitrit #13713
• Migrate EBS CSI images back to registry.k8s.io @rifelpet #13718
• Bump github.com/prometheus/client_golang from 1.12.1 to 1.12.2 @dependabot[bot] #13697
• update openstack ccm + csi @zetaab #13716
• Bump github.com/spotinst/spotinst-sdk-go from 1.118.0 to 1.120.0 @dependabot[bot] #13699
• Bump aws cni to 1.11.2 @MoShitrit #13726
• Add CSI driver for Hetzner @hakman #13728
• Use kubectl replace instead of apply when updating addons @olemarkus #13731

1.24.0-beta.1 to 1.24.0-beta.2

• Channels to have exit status 1 on apply failure @olemarkus #13749
• Add support for setting mode field on file assets @yurrriq #13715
• Revert “Use kubectl replace instead of apply when updating addons” @hakman #13761
• Fix API group name for ingresses in DNS Controller @julienperignon #13750
• Update containerd and Docker versions @hakman #13741
• Update AWS CCM images for k8s 1.20-1.22 @hakman #13748
• Update runc to v1.1.3 @hakman #13763
• Fix namespace for cert manager webhook config @olemarkus #13773
• Add back the metrics-server 443 port with a new name @olemarkus #13779
• Avoid spurious changes with ed25519 keys @hakman #13774
• Fix broken node selector for node termination handler @olemarkus #13781

1.24.0-beta.2 to 1.24.0-beta.3

• Replace flexdriver with busybox @zetaab #13809
• Update etcd-manager to v3.0.20220617 @hakman #13824
• Do not run cluster autoscaler on spot instances @olemarkus #13846
• Fix GCE resource tracking @hakman #13857
• Adding GuestAccelerators to InstanceTemplate @jonasasx #13707
• Limit GCE tag for role to 63 chars @hakman #13866
• Replace manifests after apply @olemarkus #13819
• Don’t try to manage the kube-system namespace @hakman #13764
• Release 1.24.0-beta.3 @olemarkus #13880

1.24.0-beta.3 to 1.24.0

• Clean-up firewall rules that contain targets with the cluster name hash @hakman #13869
• Log errors from detachInstance @olemarkus #13896
• gce: set ProvisioningModel on InstanceTemplate @justinsb #13902
• gce: Move out of beta, drop feature flag @justinsb,@hakman #13903
• Fix cleanup of firewall rules that contain targets with the cluster name hash @hakman #13907
• Only fail an addon update if the final apply fails. Install PKI anyway @olemarkus #13897
• Update dependencies for 1.24 @hakman #13919
• gce: Refactor ClusterPrefixedName and ClusterSuffixedName to not return error @hakman #13920
• Make it possible to enable the shield addon for LBC @olemarkus #13929
• Make IRSA webhook configure apps to use regional STS and set the default region on them @olemarkus #13926
• Limit GCE names to 63 chars for various resources @hakman #13873
• Mount /etc/hosts from host for CoreDNS @hakman #13922
• Limit GCE router name to 63 chars @hakman #13932
• Update Cilium to 1.11.6 @ReillyBrogan #13917
• Remove the v1alpha3 API version @johngmyers #13935
• Fix unsetting ASG max price @olemarkus #13852
• Revert “Add back the metrics-server 443 port with a new name” @olemarkus #13940
• aws: introduce maximum instance lifetime in cluster @sterchelen #13892
• Be more specific when filtering OS instance ports @ederst #13861
• Use csi-snapshotter for OS only when the controller is enabled @ederst #13890
• Bump EBS CSI driver to 1.8.0 @hakman #13939

1.24.0 to 1.24.1

• Use Calico v3.23 for Kubernetes 1.22+ @hakman #13901
• Fix AWS IAM Authenticator nodeSelector in k8s 1.24 @rifelpet #13965
• Skip deregistering the instance during rolling update for Spotinst @hakman #13970
• Upgrade aws-iam-authenticator to v0.5.9 @rifelpet #13979
• Use only IPv4 for Hetzner servers @hakman #13982
• Add option to set etcd-manager backup interval @hakman #13975
• Update etcd-manager to v3.0.20220717 @hakman #13990
• Update Go to v1.18.4 @hakman #13994
• Add option to set number of replicas for pod-identity-webhook @hakman #13986
• Upgrade DO CSI controller to 4.2.0 @rifelpet #14005
• Update k8s.io/client-go to match k8s.io/api @hakman #14003
• Switch to latest MacOS version for CI @hakman #14015
• Revert to using instance private DNS name to lookup hostname @hakman #14024
• Add server group management for Hetzner @hakman #14018
• Update dependencies for kOps 1.24 @hakman #13989
• Update Calico to v3.23.2 @hakman #13908
• Update etcd-manager to v3.0.20220727 @hakman #14038
• Check keyset existence before attempting to distrust @yurrriq #14041
• Fix SIGSEGV when deleting a Hetzner instance @hakman #14046
• Remove namespaces from cluster-scoped resources in CNI manifests @rifelpet #14053
• Enable rolling updates for Hetzner @hakman #14034
• Use cabundle for etcd CA files to fix key rotation in HA clusters @olemarkus #14054

1.24.1 to 1.24.2

• Add config drive as a source for OpenStack instance metadata @ederst #13845
• Allow configuring OpenStack CCM networking options @ederst #14017
• aws-ebs-csi-driver: remove preStop hook @sterchelen #14081
• Add option to configure runc version for containerd @hakman #14090
• Bump nvidia device plugin to 0.12.0 @ddelange #13745
• Add hashes for containerd v1.6.7 @hakman #14093
• Add deployment-specific selectors to nth pdb @olemarkus #14113
• Disable some flags in kube-controller-manager and kube-scheduler when logging-format is not text @h3poteto #14115
• Limit GCE network names to 63 chars @hakman #14134
• Bump the CCM images @olemarkus #14130
• Update runc to v1.1.4 @hakman #14188
• OIDC: Tolerate extra service-account key set items @seh #14175
• Bump cilium to 1.11.8 @olemarkus #14137
• Bump cert-manager to 1.8.2 @olemarkus #14212
• Calico: Work around host port/conntrack problem @seh #14205

1.24.2 to 1.24.3

• aws-node-termination-handler to match node using providerID instead of AWS DNS name @anthonyhaussman #14244
• AWS LBC needs ec2:DescribeVpcPeeringConnections for IPv6 @johngmyers #14255
• Ignore the _rundir that kubetest2 now creates @olemarkus #13914
• Fix openstack tag limitation @akkina2107 #13853
• Warm pool-enabled ASGs scaled to zero will no longer panic @olemarkus #14251
• Bump AWS CNI to 1.11.3 @MoShitrit #14107

1.24.3 to 1.24.4

• Avoid spurious changes with NLB due to access log config @hakman #14319
• Add —network-id alias for —vpc flag @hakman #14326
• Avoid spurious changes with bastion hosts due to user data @hakman #14318
• cluster-autoscaler : Add iam permission autoscaling:DescribeScalingActivities needed since 1.24 version @noony #14317
• Disable rp_filter on cilium hosts @olemarkus #14369
• gce: memberlist needs TCP also @justinsb #14364
• update node-problem-detector to version 0.8.12 @jjinno #14382
• update k8s cloudprovider openstack images @zetaab #14401
• Fix default launch spec detection for spotinst @2solt #14389

1.24.4 to 1.24.5

• Update containerd to v1.6.9 @hakman #14458
• Update Calico and Canal to v3.24.3 @hakman #14466
• Set customizable affinity and tolerations for coredns-autoscaler to match main coredns deployment @MoShitrit #14503
• AWS CNI template updates @MoShitrit #14513
• Update Go to v1.18.8 @hakman #14555
• Update Calico and Canal to latest versions @hakman #14558
• Update containerd to v1.6.10 @hakman #14550
• use sprig join for template functions @heybronson #14564
• aws: Fix SIGSEGV when using instance selector @hakman #14576
• Add generics alternatives for fi.Bool/Float/Int/String*() @hakman #14595
• Remove CloudFormation tests @johngmyers #14602
• Fix pdb for identity webhook @olemarkus #14442
• aws: Add ec2:DescribeAvailabilityZones to the AWS CCM permissions list @lobziik #14650
• aws: Limit the number of target groups updated per operation @hakman #14648
• gce: Allow Cilium to connect to its etcd cluster @hakman #14655