Release notes for kOps 1.21 series

Significant changes

Service Account Issuer Discovery and AWS IAM Roles for Service Accounts (IRSA)

kOps now supports publishing an OIDC-compatible discovery document to an S3 bucket and configuring AWS to use it for IAM Roles for Service Accounts (IRSA).

See the Service Account Issuer Discovery documentation for more information.

Dedicated API Server nodes.

kOps now supports extending the control plane with dedicated apiserver nodes. These nodes run in dedicated instance groups that can be scaled horizontally.

In 1.21, this feature is behind a feature flag as node role name, labels, taints, and domains can change based on feedback from the community.

Warm Pool (AWS only)

A Warm Pool contains pre-initialized EC2 instances that can join the cluster significantly faster than regular instances. These instances run the kOps configuration process, pull known container images, and then shut down. When the ASG needs to scale out it will pull instances from the warm pool if any are available.

See the warm pool documentation for more information.

Other significant changes

• Protokube now runs as a systemd process rather than a docker container.

• Support for AWS launch configurations has been removed in favour of launch templates.

• kOps can now use Node Termination Handler’s Queue Processor mode, which offers more functionality than the IMDS Processor mode. See the addons page for more information.

• New addon for the CSI snapshot-controller.

Breaking changes

• Support for Kubernetes versions 1.13 and 1.14 has been removed.

Required Actions

• The ClusterRoleBinding for AWS EBS CSI DaemonSet has changed name. If you installed this addon before kOps 1.21, you need run kubectl delete crb ebs-csi-node-binding.

• To support Node Termination Handler’s Queue Process mode, AWS cluster deletion now requires the kops CLI have sqs:ListQueues and events:ListRules permissions regardless of whether or not the addon is used.

Deprecations

• Support for Kubernetes versions 1.15 and 1.16 is deprecated and will be removed in kOps 1.22.

• Support for Kubernetes version 1.17 is deprecated and will be removed in kOps 1.23.

• Support for CentOS 7 is deprecated and will be removed in future versions of kOps.

• Support for CentOS 8 is deprecated and will be removed in future versions of kOps.

• Support for Debian 9 (Stretch) is deprecated and will be removed in future versions of kOps.

• Support for RHEL 7 is deprecated and will be removed in future versions of kOps.

• Support for Ubuntu 18.04 (Bionic) is deprecated and will be removed in future versions of kOps.

• The legacy location for downloads s3://https://kubeupv2.s3.amazonaws.com/kops/ has been deprecated and will not be used as of kOps 1.22. The new canonical downloads location is https://artifacts.k8s.io/binaries/kops/.

• The manifest based metrics server addon has been deprecated in favour of a configurable addon.

• The manifest based cluster autoscaler addon has been deprecated in favour of a configurable addon.

• The node-role.kubernetes.io/master and kubernetes.io/role labels are deprecated and might be removed from control plane nodes in kOps 1.23.

• Due to lack of maintainers, the Aliyun/Alibaba Cloud support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this cloud provider.

Full change list since 1.20.0 release

1.20.0-alpha.2 to 1.21.0-alpha.1

• Release notes for 1.20.0-alpha.2 @hakman #10768
• Add troubleshooting of corrupted api server leases @olemarkus #10764
• Boot nodes without state store access @justinsb #10469
• Update GCE zones @bharath-123 #10771
• Kubetest2 - Use a shell lexer for passing extra args to create cluster @rifelpet #10772
• Use the kubeApiServerConfig clientCAFile field @slu2011 #10707
• Kubetest2 - Fix splitting of —create-args @rifelpet #10775
• Logging: don’t suggest we are pre-creating DNS records unless we are @justinsb #10782
• Add missing versions to channels @olemarkus #10781
• fix: asset task copy docker image @johanneswuerbach #10767
• Add support for creating world-readable managedFiles @olemarkus #10778
• Update kubectl documentation with new flags @rpadovani,@hakman #10779
• Add overrides testing in lifecycle integration tests @rifelpet #10752
• Add AWS LoadBalancerController @olemarkus #10489
• Update Calico to v3.17.2 @hakman #10787
• Enable CSIMigrationAWS if CSI EBS driver is installed @olemarkus #10791
• Fill Role names in kops-controller-config instead of instance profile names when it is specified @h3poteto #10728
• Storage: Allow disabling of kOps’s management of StorageClasses @seh #10733
• kubetest2 - Dump all pod logs in addition to host logs @rifelpet #10799
• Update Docker to v19.03.15 @hakman #10802
• Fix LaunchSpec TF output @hakman #10806
• Make protokube CP label setting consistent with kops-controller @olemarkus #10780
• Add deprecation notice for launch templates. @bharath-123 #10809
• add azure support for internal loadbalancer to k8s api @collin-woodruff-t1cg #10744
• Allow managed images for Azure instance groups @NickSchleicher #10797
• kubenet containerd: match upstream @justinsb #10759
• kubetest2: Add —host argument @justinsb #10814
• iptables: Use the lock when checking for existing rules @justinsb #10812
• Spotinst: Replace corev1.Taint to fix HCL2 serialization @liranp #10819
• Spotinst: Bump the Ocean Controller to 1.0.72 @liranp #10820
• Allow to control which subnets and IPs get used for the API loadbalancer @codablock #10741
• kubetest2: Call Test, not Execute @justinsb #10824
• Fix kdi ‘must specify’ error @olemarkus #10825
• Update aws-sdk-go @rifelpet #10830
• Use correct tag when creating node labels from azure cloud tags @NickSchleicher #10619
• Precreate the kops-controller DNS name @rifelpet #10833
• containerd installation: always configure, even if we don’t install @justinsb #10813
• Release binaries for protokube and channels @hakman #10840
• Release 1.21.0-alpha.1 @hakman #10841

1.21.0-alpha.1 to 1.21.0-alpha.2

• Release notes for 1.21.0-alpha.1 @hakman #10844
• Update mock to v1.21.0-alpha.1 @hakman #10845
• Kubetest2 - terraform support @rifelpet #10697
• Actually enable systemd cgroup for containerd @codablock #10846
• Update Go to v1.15.8 @hakman #10853
• Add liveness probe for calico-kube-controllers @hakman #10856
• Fix OpenStack delete functions @ottosulin #10849
• Add support for CAS 1.20 + support for disabling CAS for a given IG @olemarkus #10857
• Bump aws node termination handler to 1.12.0 @bharath-123 #10863
• Kubetest2 - add ginkgo node debug logs @rifelpet #10866
• K8s Version Updates February 2021 @MoShitrit #10865
• Add note about remote identities @olemarkus #10868
• Bump metrics-server to 0.4.2 @olemarkus #10858
• kubetest2 - support terraform with kops create cluster @rifelpet #10867
• Add validation for instanceType and ami architecture @bharath-123 #10747
• Upgrade k8s 1.20 to latest patch version @MoShitrit #10875
• Update AWS CNI to latest patch version @MoShitrit #10876
• Fixes for 1.21 e2e tests @olemarkus #10879
• Release notes for 1.19.1 @justinsb #10883
• Improve machine type and image validation @hakman #10884
• fix loadBalancerID null pointer @collin-woodruff-t1cg #10886
• Update Openstack Cloud Go module to v1.20.1 @bmelbourne #10896
• Enforce 1.14 deprecation @olemarkus #10897
• add usage of subnet and routetable shared resources in azure @ngalantowicz #10900
• Update Calico to v3.18.0 @hakman #10904
• Adding Elastic IP Allocations to NLB API @timothyclarke #10872
• Release notes for 1.20.0-beta.1 @hakman #10909
• Update Google Cloud Go module to v0.77.0 @bmelbourne #10894
• Add Tagging to Instance Profiles and OIDC Providers @rifelpet #10832
• AWS LB controller is as of 1.20, not 1.19 @olemarkus #10919
• Spotinst: Prevent instance groups with the same suffix from being deleted @liranp #10918
• add support for azure public loadbalancer @collin-woodruff-t1cg #10915
• Fix nil pointer deference for image ID with spotinst @hakman #10924
• Update SSH documentation for ubuntu @jpugliesi #10931
• Fix no-schedule issue @christian-schlichtherle #10928
• Update Controller Runtime Go module to v0.8.2 @bmelbourne #10914
• Sort external policies when checking for changes @hakman #10940
• Instruct GH to collapse BUILD.bazel diffs by default @rifelpet #10912
• Further improve cloudLabel validation @olemarkus #10910
• Add a standardised set of labels on all resources @olemarkus #10796
• Bump external-dns to 0.7.6 @olemarkus #10946
• Update etcd-manager to 3.0.20210228 @justinsb #10949
• gce doesn’t suffix the IG names with ClusterName @olemarkus #10944
• Add AWS Transit Gateway support @rifelpet #10948
• Fix node label conversion in Azure @kenji-cloudnatix #10935
• Spotinst: Bump the Ocean Controller to 1.0.73 @liranp #10960
• Spotinst: Don’t skip LB attachments when SpotinstHybrid is enabled @liranp #10961
• Add explicit RBAC permissions for finalizers subresources @olemarkus #10966
• Fix typos in docs/getting_started @roim #10921
• Add support for CPU Credits on AWS t2 and t3 instance families @rifelpet #10934
• Add support for enable-cadvisor-json-endpoints with Kubelet @adrianmoisey #10957
• Exclude CP nodes from load balancers @olemarkus #10945
• Update k8s.io Go modules to v0.20.4 @bmelbourne #10965
• Update Go to v1.16 @bmelbourne #10892
• Add a note about informal office hours @olemarkus #10650
• Removing duplicate local and output values in terraform(#10786) @mmerrill3 #10978
• Add CloudLabels as —extra-tags to aws-ebs-csi driver @codablock #10976
• Use internal api url for jwks @olemarkus #10888
• Disable Calico Prometheus metrics by default @hakman #10982
• Add etcd-manager discoveryPollInterval option @ottosulin #10975
• Remove manually added labels from addons @hakman #10987
• Fix kops-controller rbac due to leader election change @olemarkus #10988
• Various cleanups around apply_cluster and awsmodel @olemarkus #10579
• Fix very minor formatting typos in docs/manifests_and_customizing_via_api @vitaliyf #10990
• Run protokube as a systemd service @bharath-123,@hakman #10574
• kubetest2 - don’t overwrite create args that use equals signs @rifelpet #10994
• Remove support for launch configurations @bharath-123 #10937
• Use exponential backoff for DNS updates @hakman #10996
• Storage: Amend default choice for StorageClass management to honor a specified OpenStack-related value @seh #11002
• Add to 1.21 release notes @bharath-123 #11004
• Kubetest2 - Add support for publishing the kops version marker @rifelpet #11006
• Kubetest2 - Fix kops’ —kubernetes-version with k8s version markers @rifelpet #11007
• Don’t build kops during periodic upgrade tests @rifelpet #11005
• Remove extraneous field from integration test @rifelpet #11010
• Remove trailing newline from kubernetes version marker @rifelpet #11011
• aws: Graceful handling of EC2 detach errors @hwoarang #10740
• Kubetest2 - use same kops binary for all commands in upgrade scenario @rifelpet #11017
• Update Calico to v3.18.1 @hakman #11018
• Increase route53 retry count from 3 to 5 @rifelpet #11020
• Spotinst: Add support for block device mappings in Ocean Launch Spec @liranp #11009
• Allow cilium 1.10 @olemarkus #11026
• Fix rendering of multiple Docker insecure registries @hakman #11027
• azure: fix null pointer when updating in place cluster @collin-woodruff-t1cg #11015
• Release notes for 1.20.0-beta.2 @hakman #11034
• Update k8s dependencies to v1.21.0-beta.1 @hakman #11013
• Trim space on kops version markers @rifelpet #11037
• Honor OS update policy at InstanceGroup level too @seh #10913
• Update Go to v1.16.2 @hakman #11039
• Create an environment file for kops-configuration systemd process @bharath-123 #11042
• Improve instance type validation error message @bharath-123 #11043
• Revert upgrade script to build kops @rifelpet #11044
• cluster validation - allow flapping of validation errors @rifelpet #11049
• Update Terraform to v0.14.8 @bmelbourne #11051
• Cleanup some nodeup & protokube logging @rifelpet #11052
• Update Go modules to latest versions @bmelbourne #11047
• Add channels entries for image architecture @hakman #11046
• fix CNI bin path in troubleshoot.md @adrianmester #11061
• Kubetest2 - Add GCE default SSH key values from prow jobs @rifelpet #11065
• correct a word for readme @yojay11717 #11066
• Update Bazel to v3.5.0 @hakman #11041
• Install bazelisk before pushing images @hakman #11067
• Kubetest2 - Add boskos for GCE support @rifelpet #11070
• Download kubectl to /opt/kops/bin on Flatcar OS @rifelpet #11054
• Kubetest2 - initialize boskos heartbeat channel @rifelpet #11073
• Instance roles for service accounts (IRSA) contd @rifelpet,@olemarkus #10756
• Kubetest2 - add more validation time for —target terraform @rifelpet #11077
• Fix GCE channels version constraints @rifelpet #11076
• Update k8s versions with March 2021 releases @MoShitrit #11075
• Upgrade AWS CNI to version 1.7.10 @MoShitrit #11078
• Improve error messages around PublicJWKS @justinsb #11085
• Don’t add control-plane DNS permissions with UseServiceAccountIAM @justinsb #11086
• Ensure a publicdatastore exists for jwks and that it can only be s3 @olemarkus #11081
• Apiserver nodes @olemarkus #10722
• fix(docs): cpuCFSQuotaPeriod needs a feature gate @danmx #11071
• Update Ubuntu 20.04 to latest AMI @bmelbourne #11083
• Re-add integration tests for jwks @justinsb #11087
• Replace go-bindata with go:embed @rifelpet #11089
• Dns controller fixes @olemarkus #11069
• Remove unused RoleLabelName16 @justinsb #11097
• Add additional IOPS validation for AWS EBS gp3 volumes @lichuan0620 #10843
• Update google SDK libraries @justinsb #11096
• Add values page @justinsb #11094
• Deeper validation in dns controller tests @justinsb #11095
• Ensure protokube can connect to kube-apiserver before starting the sync loop @olemarkus #11093
• Remove dbus dependency @bharath-123 #11082
• Have nodeup retry kops-controller bootstrapping sooner if DNS isn’t setup @rifelpet #11101
• Update AWS zones used by e2e tests @rifelpet #11103
• Add docs about dedicated apiserver ndoes @olemarkus #11090
• Put awslbcontroller on the control-plane @olemarkus #11091
• Release 1.21.0 alpha.2 @hakman #11109

1.21.0-alpha.2 to 1.21.0-alpha.3

• Release notes for 1.21.0-alpha.2 @hakman #11111
• Update release process docs @hakman #11112
• Use “tag on create” for EIPs, NLBs, and TargetGroups @rifelpet #11107
• Load env vars from file for kops-configuration service @hakman #11114
• Update containerd to v1.3.10/v1.4.4 @bmelbourne #11084
• [DigitalOcean] Fix DO Tag issue @srikiz #11102
• Kubetest2 - Setup SSH keys for GCE @rifelpet #11123
• Validate that kube-apiserver has the necessary authz modes set @olemarkus #11127
• Remove instance-selector label @bharath-123 #11048
• Kubetest2 - fix temp directory created for GCE SSH keys @rifelpet #11133
• replace hard coded aws region checks with aws sdk calls @guydog28 #11119
• kubetest2 - Specify GCE network name @rifelpet #11139
• Update protokube systemd unit docs link @rifelpet #11138
• Add scaleDownDelayAfterAdd to clusterAutoscaler spec @jurriaanpro #11140
• Update cluster_spec.md @carnivorelogic #11142
• minor protokube code clean up @bharath-123 #11143
• Pass ctx to drain helper @olemarkus #11146
• Change registrable domains to placeholders @lukehinds #11147
• Add tags to instance profile and OIDC provider terraform resources @rifelpet #11149
• Clarify release notes around exporting kubeconfig @justinsb #11154
• Expand flag help on —user flags @justinsb #11153
• Update Getting Started AWS guide @allir #11150
• fix the mistake link in addons.md @maoyangLiu #11151
• cloudbuild: capture some hashes @justinsb #11159
• Only update kops-controller pods on deletion @olemarkus #10871
• Side load images also on apiserver @olemarkus #11156
• Add an option to skip NTP installation @kenji-cloudnatix #11160
• kubetest2 - Pass GOPATH when building kops @rifelpet #11167
• Filter kOps NatGateways from route table @zetaab #11169
• Bump k8s deps to 1.21-rc.0 @olemarkus #11168
• Allow setting dedicated apiserver node count from create cluster cmd @olemarkus #11152
• Update Go to v1.16.3 @bmelbourne #11174
• Add integration test for aws lb controller @olemarkus #11175
• Enable use of irsa for aws load balancer controller @olemarkus #11088
• Increase timeout and update images for postsubmit job @rifelpet #11177
• Update Go modules to latest versions @bmelbourne #11176
• Kubetest2 - Add flag to expose cluster validation wait time @rifelpet #11178
• Spotinst: Use BDM to configure the root volume size at VNG level @liranp #11179
• Spotinst: Configure headroom resources only at the VNG level @liranp #11181
• Update k8s dependencies to v1.21.0 @hakman #11188
• Release notes for 1.19.2 @justinsb #11193
• Update node local dns cache @zetaab #11057
• Update cilium.md @recollir #11189
• Release notes for 1.20.0 @justinsb #11196
• Docs: Remove ‘prerelease’ warning from 1.20 @justinsb #11198
• Kubetest2 - Create project-specific state store buckets in GCP @rifelpet #11200
• Update release compatibility matrix @johngmyers #11201
• Update integration tests to k8s v1.21.0 @bmelbourne #11206
• Kubetest2 - Set KOPS_BASE_URL to —build’s stage location @rifelpet #11210
• Update Docker to v20.10.5 @bmelbourne #11195
• Rename the service account key @johngmyers #11207
• Update go deps @zetaab #11208
• Kubetest2 - detect errors creating GCS bucket @rifelpet #11212
• Kubetest2 - Ensure the bucket path is the final gsutil arg @rifelpet #11215
• Update IG tutorial for per-AZ node groups @rifelpet #11218
• Use “string” for architecture type in ChannelRecommendedImage @hakman #11220
• Always secure api -> kubelet communication @olemarkus #11185
• Fix etcd volume validation logic @hakman #11225
• Replace k8s.io/utils/mount with k8s.io/mount-utils @hakman #11229
• Release 1.21.0-alpha.3 @hakman #11231

1.21.0-alpha.3 to 1.21.0-beta.1

• fix a typo @yojay11717 #11232
• Release notes for 1.21.0-alpha.3 @hakman #11233
• Remove validations for EBS from cluster validation @h3poteto #11228
• Add support for Docker v20.10.6 @hakman #11236
• Don’t start kubelet if instance is entering the warm pool @olemarkus #11216
• Correct typos @Akiros001 #11238
• Logging cleanup @rifelpet #11080
• Update kops_create_secret_dockerconfig.md @integrii,@hakman #11186
• Remove BLM banner @hakman #10672
• Run tests only in zones with increased limits @hakman #11240
• Give kOps CLI knowledge about ASG warm pools @olemarkus #11227
• Fix golint issue caused by typo @fenggw-fnst #11239
• Remove unused constants @johngmyers #11241
• Bump k8s versions with April 2021 releases in Alpha channel @MoShitrit #11245
• Update kOps recommended versions and images @hakman #11247
• Kubetest2 - Cleanup leaked resources from previous clusters @rifelpet #11250
• Run tests in all regions with increased limits @hakman #11249
• Don’t set NeedUpdate on first addon install @olemarkus #11257
• Make it possible to detect field changes when mixedInstancePolicy is removed @h3poteto #11255
• Update rolling update documentation @johngmyers #11263
• Pre-pull cilium and kube-proxy in warming mode @olemarkus #11258
• [cilium] Add support for choosing resources @dntosas #11248
• Add install section to kubelet unit @olemarkus #11264
• Update terraform and cloudformation lint versions @rifelpet #11266
• Fix cilium template scoping typo @javipolo #11270
• Add Azure image to alpha/stable channel @kenji-cloudnatix #11271
• Exclude nodes from load balancers upon cordoning @johngmyers #11273
• Make it possible to enable/configure warm pool @olemarkus #11235
• If one tries to use eip with a public ip that doesn’t exist, fail @olemarkus #11276
• Spotinst: Update spotinst/ocean-controller to v1.0.74 @liranp #11286
• Add NTH Queue Processor Mode @haugenj #10995
• Apiserver fixes @olemarkus #11293
• Spotinst: Prevent nil pointer dereference @liranp #11289
• fix: create.go doesnt add —name flag to the prompt: kops update cluster @ebarped #11296
• Make warm pool no ASG found error retryable @olemarkus #11285
• Document the newly required SQS permissions for NTH @rifelpet #11300
• fix permissions required for NTH Queue Processor @haugenj #11303
• bump NTH to 1.13.0 @haugenj #11301
• Add GCE Router task @kenji-cloudnatix #11184
• Add ability to set a default Issuer in certManager addon @javipolo #11281
• Make nodeup able to complete the warming life cycle hook @olemarkus #11259
• update deps @zetaab #11306
• Filter servers using cluster name in tags @zetaab #11305
• Add warm pool docs and release notes @olemarkus #11307
• Use the full operator instead of the generic one @olemarkus #11312
• Improve warm pool documentation @johngmyers #11313
• Disallow negative warmpool sizes @johngmyers #11317
• Promote channel alpha to stable @hakman #11318
• [metrics-server] Bump manifest to latest stable @dntosas #11319
• Allow disabling warm pool by setting WarmPool.MaxSize to 0 @johngmyers #11316
• Fix typo @johngmyers #11321
• [csi/aws] Bump templates + add support for warm pools @dntosas #11304
• Add a lifecycle test for GCE @kenji-cloudnatix #11291
• Add cluster-level warmPool settings @johngmyers #11322
• Fix arguments to csi-provisioner after bump to v2.2.0 @codablock #11326
• kubetest2: Infer the provider and zones from the kops cluster @justinsb,@rifelpet #10847
• Add support for configuring Cilium enable-host-reachable-services. @bjhaid #11333
• Fix lifecycle hook naming @olemarkus #11335
• Recognize Ubuntu 21.04 @hakman #11327
• Add enable-host-reachable-services to 1.8 and generic cilium. @bjhaid #11337
• Don’t try to delete warm pool when creating the cluster @olemarkus #11331
• Update Calico to v3.18.2 @hakman #11339
• Fix SQS resource flapping @olemarkus #11336
• Update controller-runtime to v0.9.0-beta.0 @hakman #11342
• Set SAN for addon CAs @olemarkus #11328
• Update kubetest2 dependency and fix install method for upgrade scenario @rifelpet #11338
• Bump cilium to 1.9.6 @olemarkus #11344
• Fix upgrade scenario kubetest2 install @rifelpet #11350
• Fix kubetest2 panic inheriting env vars @rifelpet #11351
• Mount /run inside etcd-manager pods for systemd mounts @hakman #11352
• Update deps @zetaab #11357
• Ignore detached nodes when doing validate cluster @rajatjindal #11349
• Move firewall, iam, network and sshkey to awsmodel @hakman #11358
• [addons/nth] Add capability to define resources @dntosas #11360
• Split oidc_provider @olemarkus #11359
• Expose hubble agent when hubble is enabled @olemarkus #11314
• Configure aws oidc provider @olemarkus #11361
• Use VFS as service account issuer if configured @olemarkus #11362
• Allow cert-manager to be provisioned externally @codablock #11354
• Mark control-plane node for update when etcd volume size changes @hakman #11365
• Mark control-plane node for update when etcd manager config changes @hakman #11369
• user-configurable IAM roles for ServiceAccounts @olemarkus #11016
• add permission to create sa tokens @zetaab #11373
• Add more support for cilium 1.10 @olemarkus #11374
• Update Calico to v3.19.0 @hakman #11372
• Refactor terraform writing @johngmyers #11371
• Remove unused k8s version parsing @rifelpet #11375
• Fix upgrade of service-account key @johngmyers #11376
• Don’t try to mount hubble TLS on the agent if we don’t use hubble @olemarkus #11378
• Kubetest2 - Update k8s upgrade test + add kops upgrade test @rifelpet #11382
• Kubetest2 - Fix GNU mktemp syntax @rifelpet #11384
• Kubetest2 - fix wget flag in kops download @rifelpet #11385
• kubetest2 - remove unnecessary flags from upgrade scripts @rifelpet #11386
• Don’t use PublicJWKS in TestAWSLBController @johngmyers #11391
• Don’t add IRSA env vars if feature flag is not enabled @olemarkus #11392
• Recognize the ServiceAccountIssuerDiscovery featue gate @johngmyers #11395
• Quote grep patterns in docs/rotate-secrets.md @keithlayne #10656
• Documentation and release note for IRSA @johngmyers #11398
• Remove the PublicJWKS feature flag @johngmyers #11396
• Don’t publish OIDC discovery if DiscoveryStore not set @johngmyers #11397
• Add elasticloadbalancing:ModifyTargetGroupAttributes to aws lb controller @olemarkus #11393
• Add another update cluster dryrun to upgrade tests @rifelpet #11401
• Update default volumes types in Cluster Documentation @allir #11405

1.21.0-beta.1 to 1.21.0-beta.2

• Use etcd-manager built from etcdadm repo @justinsb,@hakman #11098
• Release 1.21.0-beta.1 @johngmyers #11408
• [addons/awscsidriver] Bump to GA release @dntosas #11418
• Verify all versions are set correctly @johngmyers #11413
• Update verify-terraform to use 0.15.3 @rifelpet #11433
• Create new clusters without forcing a container runtime @hakman #11428
• Sort —extra-tags of ebs-csi-driver @codablock #11444
• Allow AWS instance types with multiple architectures @hakman #11463
• Add support for CAS 1.21.0 @olemarkus #11462
• 1.21 branch: Announce k8s removals two kOps versions in advance @johngmyers #11490
• Update cert-manager @olemarkus #11493
• Set priorityClassName on critical addons @olemarkus #11495
• fix(coredns/rbac): add permission to list and watch endpointslices @nettoclaudio #11459
• upup: gcetasks: fix diffs in instance template and router @nicktrav #11460
• upup: gcetasks: force send AutoCreateSubnetworks field when set to false @nicktrav #11457
• Spotinst: Update spotinst/ocean-controller to v1.0.75 @liranp #11512
• bump aws lb controller to 2.2.0 @olemarkus #11502
• Set default fstype for ebs volumes to ext4 @olemarkus #11525
• [addons/networking.cilium.io] enable prometheus scraping @ulfox #11514
• Update containerd to v1.4.6 @hakman #11535
• Release images bundle instead of separate images @hakman #11522
• Bump CoreDNS manifests to latest stable version 1.8.3 @dntosas #11500
• Update CAS manifest @olemarkus #11491
• Make events etcd cluster optional @codablock #11330
• Bump default cilium to 1.9.7 @olemarkus #11554
• Add snapshot-controller @olemarkus #10730
• Add snapshot-controller @olemarkus #11561
• Allow using insecure TLS for metrics-server with Kubernetes 1.19+ @hakman #11559
• Cleanup orphaned IAM service account roles in direct render @johngmyers #11497
• Fix deletion of IAM roles and policies @johngmyers #11558

1.21.0-beta.2 to 1.21.0-beta.3

• Release 1.21.0-beta.2 @johngmyers #11567
• Allow Spotinst to use comma separated instance types @hakman #11560
• Only allow deletion of snapshots owned by the cluster @olemarkus #11571
• Only update kubeconfig user when we have user info @justinsb #11584
• Update Calico to v3.19.1 @hakman #11594
• Use the OnDelete updateStrategy for AWS VPC CNI DaemonSet @johngmyers #11590
• Add init image field for Amazon VPC CNI @ryan-dyer #11602
• Fix duplicate CopyFile tasks @johngmyers #11619
• Update Go to v1.16.4 @hakman #11626
• Set lifecycle on WarmPool task @johngmyers #11618
• Consolidate CSI livenessprobe images for multi-arch support @rifelpet #11652
• Fix jwks object path in S3 for IRSA @h3poteto #11649
• Set canonical location for downloads to artifacts.k8s.io @hakman #11486
• Drop trailing slash from oidc issuer @olemarkus #11682
• Update Go to v1.16.5 @hakman #11686
• Add support for Docker v20.10.7 @hakman #11674

1.21.0-beta.3 to 1.21.0

• Release 1.21.0-beta.3 @johngmyers #11690
• Fix set-version leaving backup files with “-e” suffix @johngmyers #11692
• Fix the CSI EBS DS CRB. @olemarkus #11704
• Add proxy envs to calico to make possible usage of AWS source @DOboznyi #11710
• Generate AWSEBSCSIDriver model only when using AWS @hakman #11718
• Use quay images for cilium @olemarkus #11728
• fix enable default SC when EBS driver is not installed @olemarkus #11773
• Compare OpenStack security groups deterministically
• Make forwardToKubeDNS work in the NodeLocal DNSCache template @ederst #11757
• Bump the cas addon version. @olemarkus #11781
• Don’t try to build etcd-manager secrets for cilium twice @olemarkus #11792
• Also set haveUserInfo=true in case —user was provided in @codablock #11812
• Handle containerExec hooks when using containerd @hakman #11855
• bump the version of gophercloud @cardoe #11830
• support large/slow downloads #11886: Set download timeout to 3 minutes @aojea,@hakman #11891
• Include GCP Project in terraform HCL2 output @rifelpet #11902
• Avoid spurious changes for ASG InstanceProtection and LT @hakman #11882

1.21.0 to 1.21.1

• Release 1.21.0 @justinsb #11908
• Add log rotation for etcd-cilium.log @hakman #11943
• check if the instance is under an asg @olivierpilotte #11958
• Cilium etcd fixes @olemarkus #11961
• Use regional STS endpoint @johngmyers #12043
• Update containerd to v1.4.8 @hakman #12059
• Update core-dns to v1.8.4 @hakman #12062
• Update Docker to v20.10.8 @hakman #12096
• Make metrics-server insecure if insecure is true @olemarkus #12114
• Update Calico to v3.19.2 @hakman #12125
• Fix cases when the VPC doesn’t exist yet for vpccidrblocks in 1.21 @mikesplain #12126
• Fix disabling unattended upgrades @olemarkus #12123
• Support Debian 11 Bullseye @ReillyBrogan #12108
• Bump cilium to 1.9.9 @olemarkus #12146
• Reconcile if managedFile is public or not @olemarkus #12148
• leverage proxy env variables @aojea #12150
• Update Go to v1.16.7 @hakman #12153
• Debian 11: Release AMIs use same AWS Owner ID as Buster @ReillyBrogan #12161
• Log s3 acl in additional cases @olemarkus #12167
• Hardcode Flatcar containerd exec command @hakman #12177
• Backport moving updatePolicy to nodeup config @ReillyBrogan #12175
• Add option in Cluster Autoscaler AddOn for AWS EC2 Static instance list @amitpd #12187

1.21.1 to 1.21.2

• Release 1.21.1 @justinsb #12191
• Update Calico to v3.19.3 for kOps 1.21 @hakman #12362
• Truncate cluster name in NTH EventBridgeRules @rifelpet #12439
• Don’t ignore channel value in toolbox template @hakman #12464
• Update containerd and Docker for kOps 1.21 @hakman #12508

1.21.2 to 1.21.4

• Increase upup http response header timeout @AlexLast #12694
• set calico-node readiness/liveness timeout to 10s @estahn #12713
• Fix out of bounds error when instance detach fails @johngmyers #12698
• Fix that states AWS IAM Instance Profile blocks IAM Role @angeloskaltsikis #12677
• Shorten filenames in the asset store @johngmyers #12765
• Add hashes for latest containerd and Docker versions @hakman #12767
• Update containerd to v1.4.12 @hakman #12772
• Fix volume ratio comparisons @olemarkus #12791
• Bump etcd manager to 20211117 @justinsb #12763
• Upgrade Go to 1.16.10 @hakman #12798

1.21.4 to 1.21.5

• Release 1.21.4 @johngmyers #12800
• Add support for etcd v3.5.1 @hakman #12826
• Add support for —dns flag in Docker config @jwolski2 #12789
• Update Go to v1.16.11 @hakman #12897
• Update Go to v1.16.12 @hakman #12956
• Prevent creation of unsupported etcd clusters @olemarkus #13011
• Add action for automatically tagging releases @johngmyers #12805
• Fix CSI migration feature gates @olemarkus #13203
• upgrade cluster: support comma separated list for machineType [1.21] @MeirP-3 #13210
• Simplify Flatcar containerd exec command @pothos #12900
• Update to etcd-manager v3.0.20220203 @justinsb #13196
• Add support for ed25519 keys in AWS @aclevername #13304