Release notes for kops 1.16 series

Significant changes

• To address the issue of IPv4 only clusters being susceptible to MitM attacks via IPv6 rogue router advertisements, the affected components have been upgraded as follows:

  • Docker version 19.03.11 - CVE-2020-13401 (optional)
  • CNI plugins 0.8.6 - CVE-2020-10749
  • Calico 3.9.6 - CVE-2020-13597
  • Weave Net 2.6.5 - CVE-2020-11091

• If upgrading from 1.11 or earlier, please see the notes in previous releases about upgrading through kubernetes 1.12, with the etcd3 upgrade.

• A new component runs on the master nodes now: kops-controller. kops-controller currently labels nodes, but will likely perform additional functionality in future releases.

Breaking changes

• Support for Docker versions 1.11, 1.12 and 1.13 has been removed because of the dockerproject.org shut down. Those affected must upgrade to a newer Docker version.

• Please see the notes in the 1.15 release about the apiGroup changing from kops to kops.k8s.io

• A controller is now used to apply labels to nodes. If you are not using AWS, GCE or OpenStack your (non-master) nodes may not have labels applied correctly.

Required Actions

• If either a kOps 1.16 alpha release or a custom kOps build was used on a cluster, a kops-controller Deployment may have been created that should get deleted. Run kubectl -n kube-system delete deployment kops-controller after upgrading to kOps 1.16.0-beta.1 or later.

• Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of kOps.

To enable the Pod priority feature, follow these steps:

kops edit cluster
# Add the following section
spec:
  kubelet:
    featureGates:
      PodPriority: "true"

Deprecations

• Support for Kubernetes releases prior to 1.9 is deprecated and will be removed in kops 1.18.

• The kops/v1alpha1 API is deprecated and will be removed in kops 1.18. Users of kops replace will need to supply v1alpha2 resources.

Full change list since 1.15.0 release

1.15.0-alpha.1 to 1.16.0-alpha.1

• Update release notes for 1.15.0-alpha.1 @justinsb #7535
• When fast-building, copy a newer version of utils.tar.gz @justinsb #7536
• Bootstrap: protokube labels its own node with node-role label @justinsb #7537
• Update copyright notices @mikesplain #7542
• Add a few docs comments on gomod and bazel @mikesplain #7541
• Calico update and typha @gjtempleton,@mikesplain #7528
• “Force” k8s 1.11.10 @justinsb #7423
• Log more sensibly when we can’t get sha256 @justinsb #7555
• [Feature] CoreDNS: External CoreFile option @gjtempleton,@mikesplain #7376
• Fix gomod errors @mikesplain #7571
• Add horizontalPodAutoscalerDownscaleStabilization @mikesplain #7573
• Associate subnets to port within OpenStack @mitch000001 #7578
• Fix kops for us-gov-east-1 #7564 @ibrf #7565
• Promote 1.13 AMI from alpha to stable @rifelpet #7590
• Add myself @rifelpet as a reviewer @rifelpet #7587
• Fix mkdocs @mikesplain #7591
• Add missing OpenStack reference @marsavela #7567
• Fix Dropped Errors in upup @alrs #7586
• Promote 2019-08-16 AMIs from alpha -> stable @justinsb #7594
• hack/update-expected.sh: mask development env vars @justinsb #7595
• “Force” k8s 1.11.10 in stable channel @justinsb #7596
• add cilium in error message @PascalBourdier #7601
• Clean security groups if api/ssh ips are removed from config @zetaab #7561
• [DO-7442] Digital Ocean add consistent volume and droplet tags for multi master feature @srikiz #7566
• Expose API Server flags needed for AWS pod identities @rifelpet #7610
• Add logrotate for etcd/etcd-events.log @mikesplain #7614
• Updated container-selinux url to point to the right path @igarcia-sugarcrm,@mikesplain #7609
• Check the HTTP response code when downloading URLs @rifelpet #7611
• Update rules_go with some fixes @mikesplain #7625
• Change Cilium templates to standalone version @nebril,@olemarkus #7474
• Skip Docker install @austinmoore- #6957
• Add —wait argument to kops validate @justinsb #7371
• Fixed “NeedsUpdate” status of nodes in mixedinstancegroups after rolling update @hippolin #7445
• fix instance name @zetaab #7641
• Use without external router (OpenStack) @zetaab #7644
• Openstack: value if spec does not associate public ips @mitch000001 #7649
• Updating master IAM policies. @michalschott #7580
• Machine types g4dn @mikesplain #7653
• OpenStack: Additional security groups for instances @mitch000001 #7581
• Add arg min-port=1024 to dnsmasq container in kube-dns @nr17 #7020
• Release notes for 1.13.1 @justinsb #7666
• Pull centos.org packages from the vault @justinsb #7674
• fix-typo @tanjunchen #7669
• Align AWS and kops validation for spot allocation strategy @coufalja #7660
• Add relnotes for 1.13.2 @justinsb #7681
• Fix some bugs reported by staticcheck @rifelpet #7663
• Bump k8s versions in alpha channel @olemarkus #7647
• Misleading description for KubeProxy MetricsBindAddress @RmMsr #7672
• Fix for tarball image names after 1.16 @justinsb #7686
• Cilium standalone continuation @olemarkus #7646
• Limit calico cpu request to 100m @justinsb #7688
• fix-up some spelling mistakes in /pkg @tanjunchen #7684
• kops-controller @justinsb #7496
• OpenStack: use InstanceGroup zones to populate availability zone @mitch000001 #7690
• alpha channel: image for 1.15 and general update @justinsb #7665
• Calico: upgrade pod2daemon (only) @justinsb #7689
• Add verify-staticcheck script @rifelpet #7687
• Create tools/sha1 and sha256 helpers, simply Makefile @justinsb #7702
• kops-controller version should match version of kops @justinsb #7700
• Publish kops-controller container dump to S3/GCS @justinsb #7701
• Change from float -> resource.Quantity @justinsb #7708
• More staticcheck bugfixes and cleanup @rifelpet #7696
• Correct word misspelling @yuxiaobo96 #7705
• fix-up some spelling mistakes @tanjunchen #7704
• Add calico 3.9.1 @mikesplain #7694
• Allow to use custom rootCAs @zetaab #7643
• cleanup code to cancel some staticcheck warnings @beautytiger #7661
• Use helpers to move gzip & sha from makefile to bazel @justinsb #7703
• Update etcd-manager with OpenStack fixes @justinsb #7710
• Update etcd-manager backup image @justinsb #7713
• Update DigitalOcean CCM to v0.1.20 @timoreimann #7714
• ineffectual assignment to @tanjunchen #7560
• remove duplicated entry in notes @beautytiger #7715
• docs: fix spelling mistakes @hwdef #7709
• Docs: Adding a doc on how to propose a cherry-pick @justinsb #7717
• relnotes for 1.14.0 @justinsb #7725
• bazel: fix hashes rule to generate outputs @justinsb #7724
• remove the repeat word in docs/authentication.md b/docs/authentication.md @tanjunchen #7729
• Rollback alpha channels 1.14.7 @mikesplain #7734
• Openstack block device mapping support @Shonei #7652
• Update controller-tools and CRDs @rifelpet #7634
• Upgrade bazel’s rules_go and rules_docker @rifelpet #7727
• simplfy code @tanjunchen #7745
• fix-up some staticcheck error @tanjunchen #7744
• nodeup download: try to use compression @justinsb #7751
• Add optional RBE support for kops @fejta #7756
• Update readme for 1.14 @mikesplain #7757
• Add a BAZEL_CONFIG Makefile arg to bazel commands @fejta #7758
• Memberlist gossip implementation @jacksontj #7521
• bazel: comment out shallow_since as fails to build with bazel 1.0 @justinsb #7771
• kOps controller support for OpenStack @zetaab #7692
• Upgrade Amazon VPC CNI plugin to 1.5.4 @rifelpet #7398
• Add documentation for updating CRDs when making API changes @rifelpet #7728
• Kubelet configuration: Maximum pods flag is miscalculated when using Amazon VPC CNI @liranp #7539
• Add event ttl flag @tioxy #7487
• docs: document state store configuration @mitch000001 #7750
• Add artifacts.k8s.io to mirror list @justinsb #7378
• fix-up gosimple check error @tanjunchen #7754
• fix-up staticcheck error @tanjunchen #7755
• remove the unnecessary newline and unused vars @tanjunchen #7760
• Upload dns-controller archive, use in KOPS_BASE_URL @justinsb #7777
• Move kops-controller to use a yaml configuration file @justinsb #7774
• fix(apiserver): allow multiple service-account-key-file @hatappi #7781
• Move kops-controller to daemonset @justinsb #7783
• Change default port for memberlist from 3997 @justinsb #7778
• bazel: remove deprecated stamp attribute from container building @justinsb #7779
• Promote alpha to stable, bump alpha @mikesplain #7795
• Fix network changed in openstack ports @zetaab #7807
• Upgrade go version to 1.12.11 @rifelpet #7811
• Rename upload command variable in Makefile @bittopaz #7798
• fix-up bug in nodeup/pkg/model @tanjunchen #7793
• fix string trim func in main @beautytiger #7801
• Alicloud: add OSS as upload dest @bittopaz #7802
• Alicloud: fix status discovery @bittopaz #7804
• Alicloud: add hostname override @bittopaz #7803
• Alicloud: fix error msg when check hostname @bittopaz #7814
• replace slice loop with append for simple and clear @beautytiger #7759
• dnsprovider,nodeup: fix static check @hwdef #7818
• pkg: fix static check @hwdef #7819
• Add relnotes for 1.15.0-beta.1 @justinsb #7797
• Docs cleanup / mkdocs migration @mikesplain #7593
• Allow for override of CoreDNS version @gjtempleton #7794
• Add netlify config @mikesplain #7823
• Update etcd-manager to 3.0.20191025 @justinsb #7822
• Document eventTTL @tioxy #7826
• use existing network and subnet in OpenStack @zetaab #7699
• fix static check @hwdef #7831
• fix firewalls for OpenStack @zetaab #7829
• Set default image for OpenStack CCM @zetaab #7773
• Add protocol rules to master as well @zetaab #7834
• Fix permalink @mikesplain #7836
• Remove extraneous document separator causing failures applying addons @ripta #7857
• docs(addons): fix broken links @mitch000001 #7846
• Fix extraneous whitespace in warning message @johngmyers #7869
• Revert “Upgrade Amazon VPC CNI plugin to 1.5.4” @rifelpet #7847
• mark weavenet-pod as system-critical @jochen42 #7874
• increase retry count @zetaab #7881
• awsup: fix shadowed var when looking for etcd cluster name @diversario #7868
• Add back calico metrics options @mikesplain #7885
• Fix kops upgrade cluster link @flackdl #7887
• Fix doc linkages to addons @s3than,@justinsb #7830
• Alicloud: remove unnecessary if when evaluateHostnameOverride @bittopaz #7850
• Alicloud: split ProviderID with “.” @bittopaz #7852
• Fix behavior of mock DescribeAutoScalingGroups when no names supplied @johngmyers #7867
• Update “Guide” links for DigitalOcean & OpenStack @jcodybaker #7884
• Add ci postsubmit script for pushing images to staging @justinsb #7697
• remove the unnecessary break @tanjunchen #7791
• [DO-7442] Add gossip cluster implementation for Digital Ocean cloud provider @srikiz #7838
• fix-up static-check @tanjunchen #7841
• remove myself from OWNERS @andrewsykim #7888
• Cleanup make targets @rifelpet #7863
• fix golint failures @FayerZhang #7854
• Recommend kops 1.11.1 @justinsb #7892
• fix-up staticcheck problems @tanjunchen #7839
• Add hint how to determine mount path of etcd data @FuriKuri #7735
• stable channel: promote default AMIs from alpha -> stable @justinsb #7893
• Release notes for 1.14.1 @justinsb #7895

1.16.0-alpha.1 to 1.16.0-alpha.2

• Add release notes for 1.16.0-alpha.1 @justinsb #7896
• stable channel: promote kubernetes 1.13.12, 1.14.8 etc @justinsb #7891
• Don’t update first node in instancegroup if cluster fails validation @johngmyers,@justinsb #7872
• add missing priorityClassName to flannel DaemonSet @EladDolev #7842
• fix broken links @dj80hd #7901
• Fix rendering of the Node Authorizer template @KashifSaadat #7916
• Fix fork bomb in Makefile @johngmyers #7935
• Unhide docs make logging @mikesplain #7936
• Upgrade AWS VPC CNI to 1.5.5 @rifelpet #7938
• Correct spelling mistakes @yuxiaobo96 #7922
• Fix flannel CNI version to use 0.2.0 @srikiz #7924
• Update vendoring documentation for go modules @rifelpet #7937
• Remove duplication and update release details @mikesplain #7939
• Updated documentation on how to move from single to multi master @mccare #7439
• Create PodDisruptionBudget for kube-dns in kube-system namespace @hakman #7856
• Add support for newer Docker versions @hakman #7860
• Machine types updates @mikesplain #7947
• fix 404 urls in docs @tanjunchen #7943
• Fix generation of documentation /sitemap.xml file @aledbf #7949
• kOps site link @mikesplain #7950
• Fix netlify mixed content @mikesplain #7953
• Fix goimports errors @rifelpet #7955
• Upate Lyft CNI to v0.5.1 @maruina #7402

1.16.0-alpha.2 to 1.16.0-beta.1

• Complete support for Flatcar @mazzy89 #7545
• Fix mounting Calico “flexvol-driver-host” in CoreOS @hakman #8062
• fix(openstack): fix additional security groups on instance groups @mitch000001 #8004
• Cloud controller template function @DavidSie #7992
• Add CapacityOptimized to list of supported spot allocation strategies @gjtempleton #7406
• Add inf1 isntances @mikesplain #8128
• Openstack: Fix cluster floating ips @mitch000001 #8115
• [Issue-7870] kops controller support for digital ocean @srikiz #7961
• Fix Handling of LaunchTemplate Versions for MixedInstancePolicy @granular-ryanbonham #8038
• Bump cilium version to 1.6.4 @olemarkus #8022
• Update copyrights for 2020 @johngmyers #8241
• cilium: don’t try to mount sys/fs/bpf if already mounted @justinsb #7832
• Fix protokube osx build @mikesplain #8263
• Add deprecation warning for older k8s versions @rifelpet #8176
• Remove kops-controller deployment @rifelpet #8273
• Promote peter & ryan & zetaab to approvers @justinsb #7983
• Fix crossbuild-nodeup-in-docker @johngmyers #8343
• Add release notes for deleting the kops-controller deployment @rifelpet #8321
• Configuration to specify no SSH key @austinmoore- #7096
• Set CLUSTER_NAME env var on amazon-vpc-cni pods @rifelpet #8274
• Don’t output empty sections in the manifests @justinsb #8317
• Fix issues with older versions of k8s for basic clusters @hakman,@rifelpet #8248
• Backport the k8s 1.9 required action release note @johngmyers #8378
• Fix scheduler policy configmap args @vvbogdanov87 #8386
• Use IAMPrefix() for hostedzone @lazzarello #8366
• Add Cilium.EnablePolicy back into templates @olemarkus #8379
• CoreDNS default image bump to 1.6.6 to resolve CVE @gjtempleton #8333
• Don’t load nonexistent calico-client cert when CNI is Cilium @johngmyers #8338
• kOps releases - prefix git tags with v @rifelpet #8373
• EBS Root Volume Termination @tioxy #7865
• Announce impending removal of v1alpha1 API @johngmyers #8064
• Add missing priorityClassName for critical pods @johngmyers #8200

1.16.0-beta.1 to 1.16.0-beta.2

• Fix Github download url for nodeup @adri,@justinsb #8468
• GCS: Don’t try to set ACLs if bucket-policy only is set @justinsb #8493
• Alicloud: allow use RAM role for OSS client @bittopaz #8025
• Cilium - Add missing Identity Allocation Mode to Operator Template @daviddyball #8445
• Make it possible to enable Prometheus metrics for Cilium @olemarkus #8433
• Update cilium to 1.6.6 @olemarkus #8484

1.16.0-beta.2 to 1.16.0

• Stabilize sequence of “export xx=xxx” statements @bittopaz #8247
• Add events RBAC permissions to kops-controller @rifelpet #8535
• Update AWS IAM Authenticator to 0.5.0 @rifelpet #8423
• Update IAM permissions for amazon-vpc-cni-k8s 1.6.0 @rifelpet #8548
• Update amazon-vpc-cni-k8s to v1.6.0 @hakman #8538
• Switch AWS IAM Authenticator to use non-scratch image @rifelpet #8555
• Fix DNS loop on Ubuntu 18.04 (Bionic) @hakman #8353
• Revert update of AWS IAM Authenticator to 0.5.0 for 1.16 @rifelpet #8583
• add s3 region @zetaab #8592
• Update coredns to 1.6.7 @maruina #8602
• Cilium fix bpffs check @olemarkus #8599
• Fix periodic e2e test for Ubuntu 16.04 @hakman #8160

1.16.0 to 1.16.1

• Add indent template function and use it to fix KubeDNS.ExternalCoreFile rendering @rochacon #7979
• Bump Cilium to 1.7 for k8s 1.12+ @olemarkus #8589
• Implementing audit dynamic configuration (#7392) @mmerrill3 #7424
• Revert “Automated cherry pick of #8589: Bump Cilium to 1.7 for k8s 1.12+ #8591: Fix typo in the cilium default version” @olemarkus #8677
• Use latest patch release for Calico, Canal and Cilium @hakman #8698
• Fix uploading of file assets @johngmyers #8694
• Tag EBS volumes when using launch templates with AWS API target @johngmyers,@hakman #8462
• Fix RollingUpdate behaviour when using LaunchTemplates for both kops & terraform spec updates @KashifSaadat,@qqshfox #8261
• Enable stamping on bazel image builds @rifelpet #8835
• Update lyft CNI to 0.6.0 @maruina #8757
• Remove support for Docker 1.11, 1.12 and 1.13 @hakman #8855
• Fix kuberouter for k8s 1.16+ @UnderMyBed,@hakman #8697
• Fix tests for obsolete Docker versions in 1.16 @hakman #8890
• Load the correct certificate before deleting @olemarkus #8945
• Use non-experimental version of encryption provider config flag in 1.13+ @zacblazic #7900

1.16.1 to 1.16.2

• Add support for Ubuntu 20.04 (Focal) @hakman #8925
• feat(openstack): propagate cloud labels to machines @mitch000001 #9013
• Back-port well known owner aliases and SSH users to 1.16 @hakman #9036
• Use Ubuntu 18.04 Docker packages for Ubuntu 20.04 setups @hakman #9046
• Make cilium operator health check go against localhost IP @olemarkus #9045
• Update to etcd-manager 3.0.20200428 @justinsb #9042

1.16.2 to 1.16.3

• Revert “Automated cherry pick of #8999: feat(openstack): propagate cloud labels to machines” @zetaab #9089
• Reduce the number of TravisCI jobs for release branch @hakman #9081
• Fix zsh completion @olemarkus #9108
• Allow cluster maintenance when channel is unavailable @johngmyers #9053
• Upgrade amazon vpc cni to 1.6.1 @rifelpet #9020
• Use systemd-timesyncd for Ubuntu 20.04 @hakman #9182
• Remove all versions of a file from the S3 bucket @hakman #9171
• Allow listing versions for objects in the S3 bucket @hakman #9205

1.16.3 to 1.16.4

• Update etcd-manager to 3.0.20200531 @hakman #9237
• Use CNI 0.8.6 for Kubernetes 1.15+ @hakman #9256
• Use Docker 19.03.11 for Kubernetes 1.17+ @hakman #9314
• Fix missing changes in Weave manifest @hakman #8965
• Update Weave Net to 2.6.5 @hakman #9330
• Update Calico for CVE-2020-13597 @hakman #9331
• Add support for c5a aws ec2 instance types @coolstang #9386