Release notes for kOps 1.18 series

Significant changes

  • The default image has been updated to Ubuntu 20.04 (Focal). Consequently, the SSH user changed to ubuntu and the Linux kernel changed to version 5.4.

  • To address the issue of IPv4 only clusters being susceptible to MitM attacks via IPv6 rogue router advertisements, the affected components have been upgraded as follows:

  • Support for RHEL 8 and CentOS 8 has been added.

  • Support for Amazon Linux 2 has been improved and will work with the default Docker version.

  • containerd has been added and can be selected as an alternate container runtime for Kubernetes. Enable by using the --container-runtime containerd flag when creating a cluster or by setting spec.containerRuntime: containerd.

  • Rolling updates now support surging and parallelism within an instance group. For details see the documentation.

  • Cilium CNI can now use AWS networking natively through the AWS ENI IPAM mode. kOps can also run a Kubernetes cluster entirely without kube-proxy using Cilium’s BPF NodePort implementation.

  • Cilium CNI can now use a dedicated etcd cluster managed by etcd-manager for synchronizing agent state instead of CRDs.

  • The Terraform target now supports Terraform 0.12 syntax (HCL2) by default. See the Required Actions item below.

  • New clusters in GCE are configured to run the metadata-proxy by default. The proxy runs as a DaemonSet and lands on nodes with the nodeLabel cloud.google.com/metadata-proxy-ready: "true". If you want to enable metadata-proxy on an existing cluster/instance group, add that nodeLabel to your instancegroup specs (kops edit ig ...) and run kops update cluster. When the changes are applied, the proxy will roll out to those targeted nodes.

  • GCE has a new flag: --gce-service-account. This takes the email of an existing GCP service account and launches the instances with it. This setting applies to the whole cluster (ie: it is not currently designed to support Instance Groups with different service accounts). If you do not specify a service account during cluster creation, the default compute service account will be used which matches the prior behavior.

  • Google API client libraries updated from v0.beta to v1.

  • Support for NodeLocalDNS cache.

Breaking changes

  • Support for Docker versions 1.11, 1.12 and 1.13 has been removed because of the dockerproject.org shut down. Those affected must upgrade to a newer Docker version.

  • Terraform users on AWS may need to rename some resources in their state file in order to prepare for Terraform 0.12 support. See Required Actions below.

  • Support for the CoreOS OS distribution has been removed. Users should consider Flatcar as a replacement.

  • Support for the Debian 8 (Jessie) OS distribution has been removed.

  • The Docker health-check service has been disabled by default. It shouldn’t be needed anymore, but it can still be enabled by setting spec.docker.healthCheck: true. It is recommended to also check node-problem-detector and draino as replacements. See Required Actions below.

  • Network and internet access for docker run containers has been disabled by default, to avoid any unwanted interaction between the Docker firewall rules and the firewall rules of netwok plugins. This was the default since the early days of kOps, but a race condition in the Docker startup sequence changed this behaviour in more recent years. To re-enable, set spec.docker.ipTables: true and spec.docker.ipMasq: true.

  • Lyft CNI plugin default subnet tags changed from from Type: pod to KubernetesCluster: myclustername.mydns.io. Subnets intended for use by the plugin will need to be tagged with this new tag and additional tag filters may need to be added to the cluster spec in order to achieve the desired set of subnets.

  • Support for basic authentication has been disabled by default for Kubernetes 1.18 and will be removed in Kubernetes 1.19.

  • Support for static tokens has been disabled by default for Kubernetes 1.18 and later. To re-enable, see the Security Notes for Kubernetes. We intend to remove support entirely in a future kops version, so file an issue with your use case if you need this feature.

  • Support for Kubernetes versions prior to 1.9 has been removed.

  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. See Required Actions below.

  • Support for the “Legacy” etcd provider has been removed for Kubernetes versions 1.18 and higher. Such clusters will need to migrate to the default “Manager” etcd provider. To migrate, see the etcd migration documentation.

  • A controller is now used to apply labels to nodes. If you are not using AWS, GCE or OpenStack your (non-master) nodes may not have labels applied correctly.

  • The kops.k8s.io/v1alpha1 API has been removed. Users of kops replace will need to supply v1alpha2 resources.

  • Please see the notes in the 1.15 release about the apiGroup changing from kOps to kops.k8s.io

Required Actions

  • Terraform users on AWS may need to rename resources in their terraform state file in order to support Terraform 0.12. Terraform 0.12 no longer supports resource names starting with digits. In kOps, both the default route and additional VPC CIDR associations are affected. See #7957 for more information.
  • The default route was named aws_route.0-0-0-0--0 and will now be named aws_route.route-0-0-0-0--0.
  • Additional CIDR blocks associated with a VPC were similarly named the hyphenated CIDR block with two hyphens for the /, for example aws_vpc_ipv4_cidr_block_association.10-1-0-0--16. These will now be prefixed with cidr-, for example aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16.

To prevent downtime, follow these steps with the new version of kOps:

  1. KOPS_FEATURE_FLAGS=-Terraform-0.12 kops update cluster --target terraform ...
  2. # Use Terraform <0.12
  3. terraform plan
  4. # Observe any aws_route or aws_vpc_ipv4_cidr_block_association resources being destroyed and recreated
  5. # Run these commands as necessary. The exact names may differ; use what is outputted by terraform plan
  6. terraform state mv aws_route.0-0-0-0--0 aws_route.route-0-0-0-0--0
  7. terraform state mv aws_vpc_ipv4_cidr_block_association.10-1-0-0--16 aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16
  8. terraform plan
  9. # Ensure these resources are no longer being destroyed and recreated
  10. terraform apply

kOps will now output Terraform 0.12 syntax with the normal workflow:

  1. kops update cluster --target terraform ...
  2. # Use Terraform 0.12. This plan should be a no-op
  3. terraform plan

  • Users that need the Docker health-check service will need to explicitly enable it: kops edit cluster # Add the following section spec: docker: healthCheck: true

  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of kOps.

To enable the Pod priority feature, follow these steps: kops edit cluster # Add the following section spec: kubelet: featureGates: PodPriority: "true"

  • If a custom kOps build was used on a cluster, a kops-controller Deployment may have been created that should get deleted. Run kubectl -n kube-system delete deployment kops-controller after upgrading to kOps 1.16.0-beta.1 or later.

Known Issues

  • AWS clusters with an ACM certificate attached to the API ELB (the cluster’s spec.api.loadBalancer.sslCertificate is set) will need to reenable basic auth to use the kubeconfig context created by kOps export kubecfg. Set spec.kubeAPIServer.disableBasicAuth: false before running kOps export kubecfg. See #9756 for more information.

Deprecations

  • Support for Kubernetes versions 1.9 and 1.10 are deprecated and will be removed in kOps 1.19.

  • Support for Ubuntu 16.04 (Xenial) has been deprecated and will be removed in future versions of kOps.

  • Support for the Romana networking provider is deprecated and will be removed in kOps 1.19.

  • Support for legacy IAM permissions is deprecated and will be removed in kOps 1.19.

Full change list since 1.17.0 release

1.17.0-alpha.1 to 1.18.0-alpha.1

1.18.0-alpha.1 to 1.18.0-alpha.2

v1.18.0-alpha.2 to v1.18.0-alpha.3

v1.18.0-alpha.3 to v1.18.0-beta.1

1.18.0-beta.1 to 1.18.0-beta.2

1.18.0-beta.2 to 1.18.0

1.18.0 to 1.18.1

1.18.1 to 1.18.2

1.18.2 to 1.18.3