Upgrading Kubernetes in Kubespray

Kubespray handles upgrades the same way it handles initial deployment. That is to say that each component is laid down in a fixed order.

You can also individually control versions of components by explicitly defining their versions. Here are all version vars for each component:

  • docker_version
  • containerd_version
  • kube_version
  • etcd_version
  • calico_version
  • calico_cni_version
  • weave_version
  • flannel_version
  • kubedns_version

warning Attempting to upgrade from an older release straight to the latest release is unsupported and likely to break something warning

See Multiple Upgrades for how to upgrade from older Kubespray release to the latest release

Unsafe upgrade example

If you wanted to upgrade just kube_version from v1.18.10 to v1.19.7, you could deploy the following way:

  1. ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.18.10 -e upgrade_cluster_setup=true

And then repeat with v1.19.7 as kube_version:

  1. ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.19.7 -e upgrade_cluster_setup=true

The var -e upgrade_cluster_setup=true is needed to be set in order to migrate the deploys of e.g kube-apiserver inside the cluster immediately which is usually only done in the graceful upgrade. (Refer to #4139 and #4736)

Graceful upgrade

Kubespray also supports cordon, drain and uncordoning of nodes when performing a cluster upgrade. There is a separate playbook used for this purpose. It is important to note that upgrade-cluster.yml can only be used for upgrading an existing cluster. That means there must be at least 1 kube_control_plane already deployed.

  1. ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.19.7

After a successful upgrade, the Server Version should be updated:

  1. $ kubectl version
  2. Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
  3. Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

If you want to manually control the upgrade procedure, you can use the variables upgrade_node_confirm or upgrade_node_pause_seconds:

upgrade_node_confirm: true - waiting to confirmation to upgrade next node upgrade_node_pause_seconds: 60 - pause 60 seconds before upgrade next node

Multiple upgrades

warning Do not skip releases when upgrading—upgrade by one tag at a time. warning

For instance, if you’re on v2.6.0, then check out v2.7.0, run the upgrade, check out the next tag, and run the next upgrade, etc.

Assuming you don’t explicitly define a kubernetes version in your k8s-cluster.yml, you simply check out the next tag and run the upgrade-cluster.yml playbook

  • If you do define kubernetes version in your inventory (e.g. group_vars/k8s-cluster.yml) then either make sure to update it before running upgrade-cluster, or specify the new version you’re upgrading to: ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml -e kube_version=v1.11.3

    Otherwise, the upgrade will leave your cluster at the same k8s version defined in your inventory vars.

The below example shows taking a cluster that was set up for v2.6.0 up to v2.10.0

  1. $ kubectl get node
  2. NAME      STATUS    ROLES         AGE       VERSION
  3. apollo    Ready     master,node   1h        v1.10.4
  4. boomer    Ready     master,node   42m       v1.10.4
  5. caprica   Ready     master,node   42m       v1.10.4
  7. $ git describe --tags
  8. v2.6.0
  10. $ git tag
  11. ...
  12. v2.6.0
  13. v2.7.0
  14. v2.8.0
  15. v2.8.1
  16. v2.8.2
  17. ...
  19. $ git checkout v2.7.0
  20. Previous HEAD position was 8b3ce6e4 bump upgrade tests to v2.5.0 commit (#3087)
  21. HEAD is now at 05dabb7e Fix Bionic networking restart error #3430 (#3431)
  23. # NOTE: May need to sudo pip3 install -r requirements.txt when upgrading.
  25. ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  27. ...
  29. $ kubectl get node
  30. NAME      STATUS    ROLES         AGE       VERSION
  31. apollo    Ready     master,node   1h        v1.11.3
  32. boomer    Ready     master,node   1h        v1.11.3
  33. caprica   Ready     master,node   1h        v1.11.3
  35. $ git checkout v2.8.0
  36. Previous HEAD position was 05dabb7e Fix Bionic networking restart error #3430 (#3431)
  37. HEAD is now at 9051aa52 Fix ubuntu-contiv test failed (#3808)

info NOTE: Review changes between the sample inventory and your inventory when upgrading versions. info

Some deprecations between versions that mean you can’t just upgrade straight from 2.7.0 to 2.8.0 if you started with the sample inventory.

In this case, I set “kubeadm_enabled” to false, knowing that it is deprecated and removed by 2.9.0, to delay converting the cluster to kubeadm as long as I could.

  1. $ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  2. ...
  3.     "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
  4. ...
  5. Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
  6. yes
  7. ...
  9. $ kubectl get node
  10. NAME      STATUS   ROLES         AGE    VERSION
  11. apollo    Ready    master,node   114m   v1.12.3
  12. boomer    Ready    master,node   114m   v1.12.3
  13. caprica   Ready    master,node   114m   v1.12.3
  15. $ git checkout v2.8.1
  16. Previous HEAD position was 9051aa52 Fix ubuntu-contiv test failed (#3808)
  17. HEAD is now at 2ac1c756 More Feature/2.8 backports for 2.8.1 (#3911)
  19. $ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  20. ...
  21.     "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
  22. ...
  23. Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
  24. yes
  25. ...
  27. $ kubectl get node
  28. NAME      STATUS   ROLES         AGE     VERSION
  29. apollo    Ready    master,node   2h36m   v1.12.4
  30. boomer    Ready    master,node   2h36m   v1.12.4
  31. caprica   Ready    master,node   2h36m   v1.12.4
  33. $ git checkout v2.8.2
  34. Previous HEAD position was 2ac1c756 More Feature/2.8 backports for 2.8.1 (#3911)
  35. HEAD is now at 4167807f Upgrade to 1.12.5 (#4066)
  37. $ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  38. ...
  39.     "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
  40. ...
  41. Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
  42. yes
  43. ...
  45. $ kubectl get node
  46. NAME      STATUS   ROLES         AGE    VERSION
  47. apollo    Ready    master,node   3h3m   v1.12.5
  48. boomer    Ready    master,node   3h3m   v1.12.5
  49. caprica   Ready    master,node   3h3m   v1.12.5
  51. $ git checkout v2.8.3
  52. Previous HEAD position was 4167807f Upgrade to 1.12.5 (#4066)
  53. HEAD is now at ea41fc5e backport cve-2019-5736 to release-2.8 (#4234)
  55. $ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  56. ...
  57.     "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
  58. ...
  59. Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
  60. yes
  61. ...
  63. $ kubectl get node
  64. NAME      STATUS   ROLES         AGE     VERSION
  65. apollo    Ready    master,node   5h18m   v1.12.5
  66. boomer    Ready    master,node   5h18m   v1.12.5
  67. caprica   Ready    master,node   5h18m   v1.12.5
  69. $ git checkout v2.8.4
  70. Previous HEAD position was ea41fc5e backport cve-2019-5736 to release-2.8 (#4234)
  71. HEAD is now at 3901480b go to k8s 1.12.7 (#4400)
  73. $ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  74. ...
  75.     "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
  76. ...
  77. Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
  78. yes
  79. ...
  81. $ kubectl get node
  82. NAME      STATUS   ROLES         AGE     VERSION
  83. apollo    Ready    master,node   5h37m   v1.12.7
  84. boomer    Ready    master,node   5h37m   v1.12.7
  85. caprica   Ready    master,node   5h37m   v1.12.7
  87. $ git checkout v2.8.5
  88. Previous HEAD position was 3901480b go to k8s 1.12.7 (#4400)
  89. HEAD is now at 6f97687d Release 2.8 robust san handling (#4478)
  91. $ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  92. ...
  93.     "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
  94. ...
  95. Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
  96. yes
  97. ...
  99. $ kubectl get node
  100. NAME      STATUS   ROLES         AGE     VERSION
  101. apollo    Ready    master,node   5h45m   v1.12.7
  102. boomer    Ready    master,node   5h45m   v1.12.7
  103. caprica   Ready    master,node   5h45m   v1.12.7
  105. $ git checkout v2.9.0
  106. Previous HEAD position was 6f97687d Release 2.8 robust san handling (#4478)
  107. HEAD is now at a4e65c7c Upgrade to Ansible >2.7.0 (#4471)

warning IMPORTANT: Some of the variable formats changed in the k8s-cluster.yml between 2.8.5 and 2.9.0 warning

If you do not keep your inventory copy up to date, your upgrade will fail and your first master will be left non-functional until fixed and re-run.

It is at this point the cluster was upgraded from non-kubeadm to kubeadm as per the deprecation warning.

  1. ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  3. ...
  5. $ kubectl get node
  6. NAME      STATUS   ROLES         AGE     VERSION
  7. apollo    Ready    master,node   6h54m   v1.13.5
  8. boomer    Ready    master,node   6h55m   v1.13.5
  9. caprica   Ready    master,node   6h54m   v1.13.5
  11. # Watch out: 2.10.0 is hiding between 2.1.2 and 2.2.0
  13. $ git tag
  14. ...
  15. v2.1.0
  16. v2.1.1
  17. v2.1.2
  18. v2.10.0
  19. v2.2.0
  20. ...
  22. $ git checkout v2.10.0
  23. Previous HEAD position was a4e65c7c Upgrade to Ansible >2.7.0 (#4471)
  24. HEAD is now at dcd9c950 Add etcd role dependency on kube user to avoid etcd role failure when running scale.yml with a fresh node. (#3240) (#4479)
  26. ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
  28. ...
  30. $ kubectl get node
  31. NAME      STATUS   ROLES         AGE     VERSION
  32. apollo    Ready    master,node   7h40m   v1.14.1
  33. boomer    Ready    master,node   7h40m   v1.14.1
  34. caprica   Ready    master,node   7h40m   v1.14.1

Upgrade order

As mentioned above, components are upgraded in the order in which they were installed in the Ansible playbook. The order of component installation is as follows:

  • Docker
  • Containerd
  • etcd
  • kubelet and kube-proxy
  • network_plugin (such as Calico or Weave)
  • kube-apiserver, kube-scheduler, and kube-controller-manager
  • Add-ons (such as KubeDNS)

Component-based upgrades

A deployer may want to upgrade specific components in order to minimize risk or save time. This strategy is not covered by CI as of this writing, so it is not guaranteed to work.

These commands are useful only for upgrading fully-deployed, healthy, existing hosts. This will definitely not work for undeployed or partially deployed hosts.

Upgrade docker:

  1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=docker

Upgrade etcd:

  1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=etcd

Upgrade kubelet:

  1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=node --skip-tags=k8s-gen-certs,k8s-gen-tokens

Upgrade Kubernetes master components:

  1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=master

Upgrade network plugins:

  1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=network

Upgrade all add-ons:

  1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=apps

Upgrade just helm (assuming helm_enabled is true):

  1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=helm