Performing automated in-place cluster upgrades

If you installed using the standard cluster installation process, and the inventory file that was used is available, you can use upgrade playbooks to automate the cluster upgrade process.

To upgrade OKD, you run Ansible playbooks with the same inventory file that you used during installation. You run the same v3_11 upgrade playbooks to:

  • Upgrade existing OKD version 3.10 clusters to version 3.11.

  • Upgrade OKD version 3.11 clusters to the latest asynchronous errata update.

Running Ansible playbooks with the —tags or —check options is not supported by Red Hat.

Upgrade workflow

The 3.10 to 3.11 control plane upgrade performs the following steps for you:

  • Back up all etcd data for recovery purposes.

  • Update the API and controllers from 3.10 to 3.11.

  • Update internal data structures to 3.11.

  • Update the default router, if one exists, from 3.10 to 3.11.

  • Update the default registry, if one exists, from 3.10 to 3.11.

  • Update the default image streams and InstantApp templates.

The 3.10 to 3.11 node upgrade performs a rolling update of nodes, which:

  • Marks a subset of nodes unschedulable and drains them of pods.

  • Updates node components from 3.10 to 3.11.

  • Returns those nodes to service.

Prerequisites

Before you upgrade your cluster:

  • Review the OKD 3.11 Release Notes. The release notes contain important notices about changes to OKD and its function.

  • If you are completing a large-scale upgrade, which involves at least 10 worker nodes and thousands of projects and pods, review Special considerations for large-scale upgrades to prevent upgrade failures.

  • If you are completing a disconnected cluster update, you must update your image registry with new image versions or the cluster update will fail. For example, when updating from 3.11.153 to 3.11.157, make sure the v3.11.157 image tags are present.

  • Upgrade the cluster to the latest asynchronous release of version 3.10. If your cluster is at a version earlier than 3.10, you must first upgrade incrementally. For example, upgrade from 3.7 to 3.9 (the 3.8 version was skipped), and then from 3.9 to 3.10.

  • Run the Environment health checks to verify the cluster’s health. In this process, you confirm that the nodes are in the Ready state and running the expected starting version and that there are no diagnostic errors or warnings.

  • Ensure that your cluster meets the current prerequisites. If it does not, your upgrade might fail.

  • The day before the upgrade, validate OKD storage migration to ensure potential issues are resolved before the outage window:

    1. $ oc adm migrate storage --include=* --loglevel=2 --confirm --config /etc/origin/master/admin.kubeconfig

Upgrading OKD

To upgrade OKD: . Check out the latest openshift-ansible code:

+

  1. # cd ~/openshift-ansible
  2. # git pull https://github.com/openshift/openshift-ansible master

  1. Run the correct upgrade playbook. Use the same inventory file that you used to install OKD. If your inventory file is not in the /etc/ansible/hosts directory, add the -i flag and specify the location.

    1. To upgrade to a new OKD minor version, run:

      1. # ansible-playbook \
      2.     -i </path/to/inventory/file> \
      3.     playbooks/byo/openshift-cluster/upgrades/<version>/upgrade.yml

    2. To upgrade to the latest z-stream version of a minor release, such as 3.11.z, run:

      1. # ansible-playbook \
      2.     -i </path/to/inventory/file> \
      3.     playbooks/byo/openshift-cluster/upgrades/<version>/upgrade.yml

  1. After all master and node upgrades have completed, reboot all hosts.

  2. Verify the upgrade.