Release notes for kops 1.15 series

Breaking changes

The kops apiGroup is changing from kops to kops.k8s.io, which means that downgrading to kops 1.14 after upgrading to kops 1.15 will not recognize the newer objects. (In general it’s better not to mix kops versions, but it is more visible here.) Please back up your manifest files using kops get <clustername> -oyaml before upgrading, if the need arises these can later be restored with kops 1.14 with kops replace -f.

It should also be possibe to rewrite the apiGroup fom kops to kops.k8s.io on a yaml backup using sed or a similar tool, but taking a precautionary backup is safer.

Significant changes

• kops now supports running with objects as CRDs, stored in a kubernetes apiserver.
• The apiGroup for kops objects has changed from kops to kops.k8s.io, to support CRDs. You can continue to provide either apiGroup as input (but you should ideally move to kops.k8s.io), but the output will always be of the kops.k8s.io form.
• Rolling updates are much faster by default. A lot of the time-padding that was in previous versions has been replaced with reliance on validation. The --cloudonly case is much faster than previously, which we believe to be correct because we expect this is normally for disaster-recovery scenarios, but you may want to specify longer timings via flags if you are relying on time-based delays.

Required Actions

• If checking the output as a string (yaml or json), please note that the apiGroup will now be kops.k8s.io, not kops. If performing strict string comparison you will need to update your expected values.

• Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of kOps.

To enable the Pod priority feature, follow these steps:

kops edit cluster
# Add the following section
spec:
  kubelet:
    featureGates:
      PodPriority: "true"

Full change list since 1.14.0 release

kops 1.14.0-beta.2 to 1.15.0-alpha.1

• Release 1.14.0-alpha.1 @justinsb #6772
• Put 1.13 and 1.14 into channels @justinsb #6781
• 1.12 release notes: populate list of PRs @justinsb #6780
• Carry Provisioned IOPS to Terraform and CloudFormation templates @mmailhos #6776
• pin nvidia-docker2 version to avoid installation failure @adrianlyjak #6768
• pkg/model: Fix dropped error @alrs #6769
• Using const() defines constants together (part:1) @alrs,@xichengliudui #6789
• Using const() defines constants together (part:3) @xichengliudui #6791
• Update rules go @mikesplain #6766
• Update etcd3-migration.md doc @bksteiny,@mikesplain #6774
• KubeAPIServer HTTP2 Stream Parameter @gambol99 #6787
• Refactor names of URLs in assets to clarify their purpose @justinsb #6420
• Update docker README.md file, delete $ @xichengliudui #6802
• Switch to golang 1.11.5 @justinsb #6798
• Switch to golang 1.12.1 @justinsb #6799
• Using const() defines constants together (part:3) @xichengliudui #6809
• Using const() defines constants together (part:4) @xichengliudui #6810
• [docs] Use env var for state store examples @elithrar #6278
• Update README.md @wangxy518 #6820
• set kubernetes version to 1.12.8 to match current release @chrisz100 #6833
• Canal manifest updates for k8s v1.12+ @KashifSaadat #6823
• Update readme chart given alphas and betas @mikesplain #6836
• Fix typo in aws-iam-authenticator image field name @rifelpet #6840
• Remove verify bazel and expose error @mikesplain #6841
• Add t3a family @mikesplain #6837
• Add support for AWS ap-east-1 region @wxdao #6835
• update tolerations to openstack external cloud provider @zetaab #6821
• [Unit Tests] Add unit tests for create_kubecfg file @srikiz #6826
• Protect against nil derefence @justinsb #6859
• Support Scale from 0 with Lauch Templates @granular-ryanbonham #6861
• Remove spurious cadvisor dependency @justinsb #6860
• makefile: add gazelle alias for bazel-gazelle @justinsb #6876
• bazel: fix distroless imports for latest bazel @justinsb #6877
• Update kubernetes dependencies to k8s 1.13.5 @justinsb #6857
• Switch from glog to klog @justinsb #6878
• travis: Remove go-vet and boilerplate checking @justinsb #6882
• Use existing SSHKeyName if no public key is created. @rralcala #6886
• Start CRDification: Change apigroup to kops.k8s.io @justinsb #6887
• Simply bazel test using exclude pattern @justinsb #6896
• Include aws-cloud-provider roles in 1.15 @justinsb #6899
• Fix machine types with klog @mikesplain #6890
• Avoid concurrent write corruption to /etc/hosts @justinsb #6893
• Add i3en instance types @mikesplain #6898
• Fix typo in docker healthcheck @TristanPeers #6901
• Update to etcd-manager 1.0.20190509 @justinsb #6917
• Call klog.InitFlags in dns-controller @justinsb #6925
• Use klog logging from 1.15 @justinsb #6924
• S3 VFS: Default to current region from metadata service @justinsb #6943
• Canal v3.7.2 for k8s v1.12+ @KashifSaadat #6950
• Subnet Update Consistency @drekle #6941
• Configure AMIs for 1.12 @justinsb #6963
• Fix Docker not being installed on Ubuntu 16.04 @mfrister #6965
• bumped k8s 1.11 versions to 1.11.10 in alpha channel @idealhack #6969
• Issue #6945 @pkutishch #6951
• Generate CRDs for kops API types @justinsb #6891
• etcd-manager: Update to 3.0.20190513 @justinsb #6959
• add node-exporter to allowed ports @zetaab #6944
• Make gofmt fails find usage @drekle #6954
• Update commitlog relnotes for 1.12.0 @justinsb #6981
• 1.12 highlight changelog @granular-ryanbonham #6982
• Mention version of kOps that introduced new features @rifelpet #6983
• Terraform: fix options field, should be spot_options @kimxogus #6988
• Add shortNames and columns to InstanceGroup CRD @justinsb #6995
• Add script to verify CRD generation @justinsb #6996
• Update README.md to reflect 1.12 release @natebwangsut #7002
• add kops instancegroup tag to metadata @zetaab #6991
• Don’t panic when deleting instancegroups @justinsb #7000
• Support using kops CLI with CRDs @justinsb #7006
• etcd-manager: update to 3.0.20190516 @justinsb #7007
• VPC cleanup: recognize the error code for concurrent VPC deletion @justinsb #7008
• Recommend kops 1.12.1 @justinsb #7023
• Add relnotes list for 1.12.1 @justinsb #7022
• Makefile: keep go vet simple @justinsb #7030
• Update go_version to 1.12.5 @justinsb #7035
• Start relnotes for 1.13 and 1.14 @justinsb #7037
• Speed up rolling-update - longer timeout on validation, less scheduled holds @justinsb #6747
• Update stretch dependencies and kubeup @mikesplain #7041
• Bump alpha-channel of k8s @justinsb #7038
• Fix machine empheral disks @mikesplain #7062
• Add docs for cpuCFSQuota / cpuCFSQuotaPeriod @thomaspeitz #7074
• implement append admission controllers @zetaab #7070
• Add documentation for etcd-manager backup/restore procedures @dzoeteman #7072
• Fix typo on node-authorizer prometheus metric @KashifSaadat #7080
• Openstack delete dynamic floating ip in delete cluster @drekle #7045
• Updated docs for openstack cloud provider. @prankul88 #7092
• Change versions to fix memory.limit_in_bytes: device or resource busy @flouthoc #7067
• K8s 1.12.8 to stable 1.12.9 to alpha @granular-ryanbonham #7090
• Fix link to Calico route reflectors documentation + typo @Misdre #7088
• Mark ENI 0 as delete_on_termination for LaunchTemplates @granular-ryanbonham #7094
• Add Debian 10 (buster) support @zetaab #7071
• Openstack support for rolling-update status @drekle #7050
• Upgrade AWS VPC CNI provider to 1.5.0 @rifelpet #7122
• Documentation Cleanup @rifelpet #7123
• Adding affinity and PDB to dns. @michalschott #7077
• bumped k8s 1.11 versions to 1.11.10 in stable channel @idealhack #6984
• Add support for SpotPrice and Mixed Instance ASGs @rifelpet #7066
• support apiserver admission-control-config-file flag @r0fls #7109
• updated link to kubernetes bot commands @opowbow #7144
• typo fix: fix kops-server-push -> make kops-server-push @Sn0rt #7150
• Add rdrgmnzs to the approvers list in OWNERS @rdrgmnzs #7154
• Flatcar support @mazzy89 #7084
• Don’t precreate etcd DNS records if we’re using etcd-manager @rifelpet #7141
• Update Docs for Calico Backend for kops 1.12 @gjtempleton #7164
• Update Canal to v3.7.3 @KashifSaadat #7169
• Improve docs on labels @granular-ryanbonham #7139
• Allow user to set the —kube-api-qps and —kube-api-burst flags on KubeControllerManager @rdrgmnzs #7153
• Egress proxy for etcd manager @austinmoore- #7103
• [Unit Tests] Added unit test for kube proxy builder @srikiz #7124
• add c5.12xlarge, c5.24xlarge, c5.metal, i3en.metal @rekcah78 #7166
• Simplify go test command @justinsb #7003
• Spotinst: New instance group type: Ocean @liranp #7040
• Fix the link to the Prow commands. @cjwagner #7162
• add masterPublicName support in kops set cluster @camilosantana #7160
• Update aws-iam-authenticator image to 0.4.0 @rifelpet #6803
• Added some changes to openstack.md file @prankul88 #6985
• Relnotes for 1.12.2 @justinsb #7174
• Add updated 1.12 image to the alpha channel @justinsb #7176
• Instance protection @mikesplain #7177
• Remove kube-proxy resource-container flag @justinsb #7224
• Drop missing sources when building utils image @KashifSaadat #7217
• goimports update @KashifSaadat #7218
• Add more debug info for when cluster path doesnt match @jayunit100 #7202
• Canal v3.7.4 @KashifSaadat #7206
• Upgrade Calico to 3.7.2 @asincu #7051
• Spotinst: Ocean’s Strategy object is optional @liranp #7183
• update instances list with make update-machine-types @rekcah78 #7195
• Possibility to use OpenStack without lbaas (loadbalancer) @zetaab #7178
• Clear append admission plugins before inserting flags to kube-apiserver @zetaab #7182
• Use NodeAuthorizer config options instead of soely hard-codes @jacksontj #7211
• doc: support to debug kops-apiserver @Sn0rt #7151
• GCE tutorial markdown formatting @flaviamissi #7188
• Make an actual deep-copy of the state @jacksontj #7219
• Set priority for static pods @vainu-arto #6897
• Allow setting Limit & Request for aws-iam-authenticator @rdrgmnzs #7260
• Delete the function keyword to prevent shellcheck from failing @xichengliudui #6811
• Bumping calico to 3.7.4. @michalschott #7249
• Update metrics server image @elisiano #6871
• Use readinessProbe for weave-net instead of livenessProbe @ReillyProcentive #7102
• Add some permissions to cluster-autoscaler clusterrole @Coolknight #7248
• Spotinst: Rolling update always reports NeedsUpdate @liranp #7251
• Add documentation example for running kOps in a CI environment @rifelpet #7256
• Calico -> 3.7.4 for older versions @justinsb #7282
• [Issue-7148] Legacyetcd support for Digital Ocean @srikiz #7221
• Stop .gitignoring all files named go-bindata @justinsb #7288
• Create hack/update-expected.sh to update test output @justinsb #7291
• replace behavior for @aws hostnameOverride @jacksontj #7185
• Rhel8 support @cassandracomar #7287
• Update DigitalOcean CCM to v0.1.16 @timoreimann #7293
• Replace use of cmdutil IsFilenameSliceEmpty @justinsb #7289
• GCE: support ipalias networking mode, named “gce” @justinsb #6229
• Move NTP and misc packages initialization to code @justinsb #6236
• Machine types fix @mikesplain #7300
• Improve channel updates @granular-ryanbonham,@justinsb #7133
• Rationalize golden-output comparison @justinsb #7290
• hack/update-expected: regenerate gobindata @justinsb #7311
• Add me as reviewer @granular-ryanbonham #7313
• Update Calico to v3.8.0 @tmjd #7257
• Manifest hashing: move trimming out of hash function @justinsb #7312
• Adding documentation to mitigate workload outages on AWS @endzyme #7292
• Mount FlexVolume directory in kube-controller-manager pod @kellanburket #6874
• remove code: remove kops-server chart @Sn0rt #7324
• Bump alpha channel with latest kubernetes versions @justinsb #7338
• Default etcd-version to 3.3.10 for >= 1.14 @justinsb #7341
• Warn/prevent if the version of etcd is unsupported with etcd-manager @justinsb #7340
• Update Image version and RBAC for Citrix Ingress Controller @christus02 #7335
• Promote k8s 1.12.9 from alpha -> stable @justinsb #7337
• Update repo-infra and distroless for bazel fixes @rifelpet #7348
• Cross-Zone Load Balancing for API ELB @austinmoore- #6958
• stop kubelet to prevent orphan containers @qqshfox #7345
• Update default flexvolumepath for COS @justinsb #7339
• Promote 1.12 image from alpha to stable @idealhack #7343
• Update kube-router to 0.3.1 @combor #7317
• Enable scraping of weave metrics @zacblazic #7326
• Bump etcd-manager to 3.0.20190801 @justinsb #7349
• Add mappings for Webhook authorization mode. @anderseknert #7344
• Set and mount the correct volume plugin dir based on OS @KashifSaadat #7355
• Don’t default adding MIMEBOUNDARY headers when a mixed instances policy is set @KashifSaadat #7370
• Add release notes for 1.13.0 beta.2 -> 1.13.0 @justinsb #7372
• Release notes for 1.12.2 -> 1.12.3 @justinsb #7373
• correct typo in output message @beautytiger #7380
• Remove extraneous note in 1.13 release notes @rifelpet #7374
• protokube/gce_volume.go: error info correction @beautytiger #7382
• cleanup: client.go error message words correction @beautytiger #7394
• awstasks: fix misspelled words in logging. @beautytiger #7412
• AWS SDK v1.23.0 @gjtempleton #7404
• Update Compatibility Table in Readme @austinorth #7408
• Add maxPersistentVolumes to support the KUBE_MAX_PD_VOLS scheduler setting @ripta #7346
• add zetaab as reviewer @zetaab #7403
• Support mirroring for nodeup also @justinsb #7377
• Use Cluster Proportional Autoscaler for CoreDNS 1.12+ @gjtempleton #7400
• Upgrading k8s-srcdst to v0.2.2 @michalschott #7388
• add OpenStack etcd-manager support @zetaab #7395
• Bump k8s versions in alpha channel @justinsb #7422
• Update AMIs in alpha channels @justinsb #7420
• skip verification when the file already installed @qqshfox #7387
• cleanup: fix error message typos @beautytiger #7425
• fix Typo ‘the the’ -> ‘the’ @xichengliudui #7441
• ali.go: cleanup error words in messages @beautytiger #7433
• promote k8s versions @zetaab #7437
• Dont set ExperimentalCriticalPodAnnotation feature gate in k8s 1.16 @rifelpet #7430
• fix typo “specifiction” -> “specification” @928234269 #7440
• Add relnotes for 1.14.0-beta.1 @justinsb #7443
• bash script: don’t assume nodeup filename @justinsb #7448
• Add doc for using custom CA @joshbranham #7434
• nodeup download should try all mirrors @justinsb #7447
• Add nodeup to shipbot targets for release upload @justinsb #7449
• Update channel recommended versions for kops versions @justinsb #7446
• fix typo “in ingards to” -> “in regard to” @928234269 #7451
• Update weave to 2.5.2 @while1eq1 #7444
• cloudformation tests: use standard file comparison @justinsb #7450
• Relnotes for 1.14.0-beta.2 @justinsb #7461
• Look for sha256 and sha1 files for artifacts @justinsb #7468
• [Digital Ocean] DO-7442 upgrade godo client to latest version @srikiz #7467
• Fix Flatcar distro @mazzy89 #7464
• fix typo “new” -> “newer” @928234269 #7462
• print all failure messages @zetaab #7465
• [DO-7148] Digital Ocean support for etcd-manager @srikiz #7435
• Publish sha256 artifacts for kops itself @justinsb #7471
• Ignore empty hashfiles @justinsb #7472
• Update to kubernetes 1.15 @justinsb #7470
• util/pkg/vfs: Fix swallowed errors @alrs #7483
• Set GOPROXY in travis builds @justinsb #7485
• Allow configure ip to ip mode in calico @zetaab #7481
• Add exec-opts options to dockerconfig @tvi #7460
• move OpenStack from alpha to beta @zetaab #7488
• Add support for netExtraArgs @ReillyProcentive #7429
• Cleanup versions, deprecate kops 1.10, remove k8s 1.8 @mikesplain #7491
• Create verify-gomod script @justinsb #7498
• Update to golang 1.12.9 @justinsb #7499
• Corrected spelling of ‘we’ in the documentation @ashishbharthi #7495
• fix-up docs/releases/1.11-NOTE.md spelling mistake @tanjunchen #7501
• Copy well-known users from apiserver @justinsb #7497
• Replace resource.FilenameOptions with []string @justinsb #7500
• Configure calico MTU @zetaab #7480
• Create env-var helper function @justinsb #7505
• Label AWS ASGs with kops.k8s.io/instancegroup @justinsb #7504
• Support for using hostPort when using flannel @shamil #7295
• Remove unused ClientGetter from Drain code @justinsb #7509
• DeleteLocalData on drain @justinsb #7510
• Updating the vendored gazelle to match workspace. @mikesplain #7511
• Add verbosity @mikesplain #7514
• fix(addons/coredns.addons.k8s.io) Workaound to stop coredns crashing on 1.3.1 version @phspagiari #7492
• Update rules_docker with python2 workaround @rifelpet #7508
• Update and add back some sizes @mikesplain #7515
• Don’t try to delete ElasticIPs of NatGateway is shared @dzoeteman #7525
• fix(addons/networking.projectcalico.org) calico kube-controllers is needed in CRD mode @phspagiari #7517
• remove default insecure from openstack @zetaab #7524
• docs: fix link to Metrics Server user guide @ruxandrafed #7479
• fix static check error in vfssync.go @beautytiger #7482
• fix(upup/models/cloudup/resources/addons/coredns.addons.k8s.io) missing resourceVersion @phspagiari #7477
• modify-doc-small-mistake @tanjunchen #7519

kops 1.15.0-alpha.1 to 1.15.0-beta.1

• Calico update and typha @gjtempleton,@mikesplain #7528
• [Feature] CoreDNS: External CoreFile option @gjtempleton #7376
• Fix gomod errors @mikesplain #7571
• Log more sensibly when we can’t get sha256 @justinsb #7555
• Add horizontalPodAutoscalerDownscaleStabilization @mikesplain #7573
• Fix kops for us-gov-east-1 #7564 @ibrf #7565
• Fix Dropped Errors in upup @alrs #7586
• add cilium in error message @PascalBourdier #7601
• [DO-7442] Digital Ocean add consistent volume and droplet tags for multi master feature @srikiz #7566
• Expose API Server flags needed for AWS pod identities @rifelpet #7610
• Add logrotate for etcd/etcd-events.log @mikesplain #7614
• Updated container-selinux url to point to the right path @igarcia-sugarcrm,@mikesplain #7609
• Check the HTTP response code when downloading URLs @rifelpet #7611
• Clean security groups if api/ssh ips are removed from config @zetaab #7561
• Skip Docker install @austinmoore- #6957
• Add —wait argument to kops validate @justinsb #7371
• Fixed “NeedsUpdate” status of nodes in mixedinstancegroups after rolling update @hippolin #7445
• Associate subnets to port within OpenStack @mitch000001 #7578
• fix instance name @zetaab #7641
• Use without external router (OpenStack) @zetaab #7644
• Updating master IAM policies. @michalschott #7580
• Cherrypick #7581 into release 1.15 @mitch000001 #7671
• Pull centos.org packages from the vault @justinsb #7674
• Align AWS and kops validation for spot allocation strategy @coufalja #7660
• Limit calico cpu request to 100m @justinsb #7688
• Cherrypick #7690 onto release 1.15 @mitch000001 #7693
• Update etcd-manager with OpenStack fixes @justinsb #7710
• Change Cilium templates to standalone version @nebril,@olemarkus #7474
• Update DigitalOcean CCM to v0.1.20 @timoreimann #7714
• Cilium standalone continuation @olemarkus #7646
• Add calico 3.9.1 @mikesplain #7694
• Fix some bugs reported by staticcheck @rifelpet #7663
• Add arg min-port=1024 to dnsmasq container in kube-dns @nr17 #7020
• Add artifacts.k8s.io to mirror list @justinsb #7378
• Upgrade Amazon VPC CNI plugin to 1.5.4 @rifelpet #7398
• Add event ttl flag @tioxy #7487
• Kubelet configuration: Maximum pods flag is miscalculated when using Amazon VPC CNI @liranp #7539
• fix(apiserver): allow multiple service-account-key-file @hatappi #7781
• Openstack: value if spec does not associate public ips @mitch000001 #7649

1.15.0-beta.1 to 1.15.0

• Cherry-pick #7807 to release-1.15 @zetaab #7809
• allow protocol rules in master @zetaab #7835
• Revert “Upgrade Amazon VPC CNI plugin to 1.5.4” @rifelpet #7847
• Add back calico metrics options @mikesplain #7885
• Remove extraneous document separator causing failures applying addons @ripta #7857
• add missing priorityClassName to flannel DaemonSet @EladDolev #7842
• Create PodDisruptionBudget for kube-dns in kube-system namespace @hakman,@justinsb #7856
• Machine types updates @mikesplain #7947
• Add support for newer Docker versions @hakman #7860

1.15.0 to 1.15.1

• Add indent template function and use it to fix KubeDNS.ExternalCoreFile rendering @rochacon #7979
• fix(openstack): fix additional security groups on instance groups @mitch000001 #8004
• Fix Handling of LaunchTemplate Versions for MixedInstancePolicy @granular-ryanbonham #8038
• Fix mounting Calico “flexvol-driver-host” in CoreOS @hakman #8062
• Complete support for Flatcar @mazzy89 #7545
• Openstack: Fix cluster floating ips @mitch000001 #8115
• Bump cilium version to 1.6.4 @olemarkus #8022
• mark weavenet-pod as system-critical @jochen42 #7874
• cilium: don’t try to mount sys/fs/bpf if already mounted @justinsb #7832
• Update copyrights for 2020 @hakman #8241
• Fix rendering of the Node Authorizer template @KashifSaadat #7916
• Cherry pick #7874 onto 1.15 @k8s-ci-robot #8090
• Backport the k8s 1.9 required action release note @johngmyers #8378
• Don’t output empty sections in the manifests @justinsb,@rifelpet #8317
• Fix issues with older versions of k8s for basic clusters @hakman,@rifelpet #8248
• CoreDNS default image bump to 1.6.6 to resolve CVE @gjtempleton #8333
• Don’t load nonexistent calico-client cert when CNI is Cilium @johngmyers #8338
• kOps releases - prefix git tags with v @rifelpet #8373

1.15.1 to 1.15.2

• Fix Github download url for nodeup @adri,@justinsb #8468
• GCS: Don’t try to set ACLs if bucket-policy only is set @justinsb #8493
• Cilium - Add missing Identity Allocation Mode to Operator Template @daviddyball #8445
• Make it possible to enable Prometheus metrics for Cilium @olemarkus #8433

1.15.2 to 1.15.3

• Stabilize sequence of “export xx=xxx” statements @mitch000001 #8530
• Properly detect that bpffs has been mounted @olemarkus #8612
• Fix uploading of file assets @johngmyers #8720
• Update to etcd-manager 3.0.20200428 @justinsb #9043