Performing Automated In-place Cluster Upgrades

You are viewing documentation for a release that is no longer supported. The latest supported version of version 3 is [3.11]. For the most recent version 4, see [4]

You are viewing documentation for a release that is no longer supported. The latest supported version of version 3 is [3.11]. For the most recent version 4, see [4]

About This Upgrade

If you installed using the standard cluster installation process, and the inventory file that was used is available, you can use upgrade playbooks to automate the OpenShift cluster upgrade process.

OKD 3.10 involves significant changes to the architecture of the cluster, which are discussed further in the OKD 3.10 Release Notes. The following sections detail how these notable technical changes affect the upgrade process from OKD 3.9.

Running Ansible playbooks with the —tags or —check options is not supported by Red Hat.

Defining Node Groups and Host Mappings

Starting in OKD 3.10, node configurations are now bootstrapped from the master. When node services are started, the node checks if a kubeconfig and other node configuration files exist before joining the cluster. If they do not, the node pulls the configuration from the master, then joins the cluster.

This process replaces administrators having to manually maintain the node configuration uniquely on each node host. Instead, the contents of a node host’s /etc/origin/node/node-config.yaml file are now provided by ConfigMaps sourced from the master.

To map which ConfigMap to use for which node host, node groups must be defined and then set for each host in the inventory file. These procedures are handled during the Preparing for an Automated Upgrade section.

Updating Host Name Parameters

In OKD version 3.10, the openshift_hostname parameter was removed. If you used the openshift_hostname parameter in your inventory file in OKD version 3.9, you must remove that parameter from your inventory file. Before you upgrade, you must set the openshift_kubelet_name_override parameter to the openshift_hostname that you used in previous versions.

You must not remove the openshift_kubelet_name_override parameter from your inventory file after you upgrade.

Node ConfigMaps

The Configmaps for defining the node configurations must be available in the openshift-node project. ConfigMaps are also now the authoritative definition for node labels; the old openshift_node_labels value is effectively ignored.

The installer provides a playbook for creating the following default ConfigMaps:

  • node-config-master

  • node-config-infra

  • node-config-compute

The following ConfigMaps are also created, which label nodes into multiple roles:

  • node-config-all-in-one

  • node-config-master-infra

Changes should not be made to a node host’s /etc/origin/node/node-config.yaml file. They will be overwritten by the configuration defined in the ConfigMap used by the node.

Node Group Definitions

After updating the openshift-ansible package, you can view what the default set of node group definitions looks like in YAML format in the ~/openshift-ansible/roles/openshift_facts/defaults/main.yml file:

  1. openshift_node_groups:
  2.   - name: node-config-master (1)
  3.     labels:
  4.       - 'node-role.kubernetes.io/master=true' (2)
  5.     edits: [] (3)
  6.   - name: node-config-infra
  7.     labels:
  8.       - 'node-role.kubernetes.io/infra=true'
  9.     edits: []
  10.   - name: node-config-compute
  11.     labels:
  12.       - 'node-role.kubernetes.io/compute=true'
  13.     edits: []
  14.   - name: node-config-master-infra
  15.     labels:
  16.       - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true'
  17.     edits: []
  18.   - name: node-config-all-in-one
  19.     labels:
  20.       - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true,node-role.kubernetes.io/compute=true'
  21.     edits: []
1Node group name.
2List of node labels associated with the node group.
3Any edits to the node group’s configuration.

If you do not set the openshift_node_groups variable in your inventory file’s [OSEv3:vars] group, the defaults defined above will be used. However, if you want to deviate from these defaults, you must define the entire openshift_node_groups structure (including all desired node groups) in your inventory file.

The openshift_node_groups value is not merged with the defaults, and the YAML definition must first be translated into a Python dictionary. You can then use the edits field to modify any node configuration variables as desired by specifying the key-value pairs.

See Master and Node Configuration Files for reference on configurable node variables.

For example, the following entry in an inventory file defines groups named node-config-master, node-config-infra, and node-config-compute, and edits the ConfigMap for node-config-compute to set kubeletArguments.pods-per-core to 20:

  1. openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true',]}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]

Whenever the openshift_node_group.yml playbook is run, the changes defined in the edits field will update the related ConfigMap (node-config-compute in this example), which will ultimately affect the node’s configuration file on the host.

Upgrade Workflow

The 3.9 to 3.10 control plane upgrade performs the following steps for you:

  • A backup of all etcd data is taken for recovery purposes.

  • The API and controllers are updated from 3.9 to 3.10.

  • Internal data structures are updated to 3.10.

  • The default router, if one exists, is updated from 3.9 to 3.10.

  • The default registry, if one exists, is updated from 3.9 to 3.10.

  • The default image streams and InstantApp templates are updated.

The 3.9 to 3.10 node upgrade performs a rolling update of nodes, which:

  • Marks a subset of nodes unschedulable and drains them of pods.

  • Updates node components from 3.9 to 3.10.

  • Converts from local configuration to bootstrapped TLS and Config.

  • Converts SDN components from systemd services to DaemonSets.

  • Returns those nodes to service.

  • Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

  • If you are using GlusterFS, see Special Considerations When Using Containerized GlusterFS before proceeding.

  • If you are using GCE Persistent Disk (gcePD), see Special Considerations When Using gcePD before proceeding.

  • The day before the upgrade, validate OKD storage migration to ensure potential issues are resolved prior to the outage window:

    1. $ oc adm migrate storage —include=* —loglevel=2 —confirm —config /etc/origin/master/admin.kubeconfig

Automated upgrade playbooks are run via Ansible directly using the ansible-playbook command with an inventory file, similar to the advanced installation method. The same v3_10 upgrade playbooks can be used for either of the following scenarios:

Running Upgrade Playbooks

Ensure that you have the latest openshift-ansible code checked out:

  1. # cd ~/openshift-ansible
  2. # git pull https://github.com/openshift/openshift-ansible master

Then run one of the following upgrade playbooks utilizing the inventory file you used during cluster installation. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location.

Upgrading OpenShift Origin Minor Versions

To upgrade to a new OpenShift Origin minor version run the following playbook:

  1. # ansible-playbook \
  2.     -i </path/to/inventory/file> \
  3.     playbooks/byo/openshift-cluster/upgrades/<version>/upgrade.yml

Upgrading to OpenShift Origin z-Stream Releases

To upgrade an existing OpenShift Origin latest z-stream of a minor release (e.g., 3.10.z), run the following playbook:

  1. # ansible-playbook \
  2.     -i </path/to/inventory/file> \
  3.     playbooks/byo/openshift-cluster/upgrades/<version>/upgrade.yml

After rebooting, continue to Verifying the Upgrade.

  1. If you do not set the openshift_node_groups variable in the inventory file’s [OSEv3:vars] group, the default set of node groups and ConfigMaps will be created when the openshift_node_group.yml playbook is run. If you want to deviate from the defaults, define the entire node group set using a Python dictionary format as outlined in Defining Node Groups and Host Mappings, specifying their name, labels, and any edits to modify the ConfigMaps. For example:

    1. [OSEv3:vars]
    3. openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true',]}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]

  2. In order to convert a OKD 3.9 cluster to using the new node group definitions and mappings, all hosts previously defined in the [nodes] inventory group must be assigned an openshift_node_group_name. This value is used to select the ConfigMap that configures each node.

    For example:

    1. [nodes]
    2. master[1:3].example.com openshift_node_group_name='node-config-master'
    3. infra-node1.example.com openshift_node_group_name='node-config-infra'
    4. infra-node2.example.com openshift_node_group_name='node-config-infra'
    5. node1.example.com openshift_node_group_name='node-config-compute'
    6. node2.example.com openshift_node_group_name='node-config-compute'

    In addition, remove the openshift_node_labels setting from any existing host entries [nodes] group if they are set. Node labels should now be defined in the ConfigMap associated with the host’s openshift_node_group_name instead.

  3. The upgrade process will be blocked until you have the newly required ConfigMaps for bootstrapping nodes in the openshift-node project. Run the following playbook to have the defaults created (or if you defined the openshift_node_groups structure in the previous step, your custom sets will be created):

    1. # ansible-playbook -i </path/to/inventory/file> \
    2.     ~/openshift-ansible/playbooks/openshift-master/openshift_node_group.yml

  4. Carefully verify that each ConfigMap has created and defined as expected. Get your list of all ConfigMaps in the openshift-node project:

    1. $ oc get configmaps -n openshift-node

    Then use oc describe to inspect them individually:

    1. $ oc describe configmaps -n openshift-node <configmap_name>

    1. If you have applied manual configuration changes to your master or node configuration files since your last Ansible playbook run (whether that was initial installation or your most recent cluster upgrade), and you have not yet made the equivalent changes to your inventory file, review Configuring Your Inventory File. For any variables that are relevant to the manual changes you made, apply the equivalent appropriate changes to your inventory files before running the upgrade. Otherwise, your manual changes may be overwritten by default values during the upgrade, which could cause pods to not run properly or other cluster stability issues.

      Additionally, verify the following settings:

  5. If you made any changes to admissionConfig settings in your master configuration files, review the openshift_master_admission_plugin_config variable in Configuring Your Inventory File. Failure to do so could cause pods to get stuck in Pending state if you had ClusterResourceOverride settings manually configured previously (as described in Configuring Masters for Overcommitment).

  6. If you manually manage the cluster’s /etc/origin/master/htpasswd file, add openshift_master_manage_htpasswd=false to your inventory file to prevent the upgrade process from overwriting the htpasswd file.

After satisfying these steps, you can review the following sections for more information on how the upgrade process works and make decisions on additional upgrade customization options if you so choose. When you are prepared to run the upgrade, you can continue to Upgrading to the Latest OKD 3.10 Release.

Updating Policy Definitions

During a cluster upgrade, and on every restart of any master, the default cluster roles are automatically reconciled to restore any missing permissions.

  1. If you customized default cluster roles and want to ensure a role reconciliation does not modify them, protect each role from reconciliation:

    1. $ oc annotate clusterrole.rbac <role_name> --overwrite rbac.authorization.kubernetes.io/autoupdate=false

    You must manually update the roles that contain this setting to include any new or required permissions after upgrading.

  2. Generate a default bootstrap policy template file:

    1. $ oc adm create-bootstrap-policy-file --filename=policy.json

    The contents of the file vary based on the OKD version, but the file contains only the default policies.

  3. Update the policy.json file to include any cluster role customizations.

  4. Use the policy file to automatically reconcile roles and role bindings that are not reconcile protected:

    1. $ oc auth reconcile -f policy.json

  5. Reconcile security context constraints:

    1. # oc adm policy reconcile-sccs \
    2.     --additive-only=true \
    3.     --confirm

Upgrading the Control Plane and Nodes in Separate Phases

An OKD cluster can be upgraded in one or more phases. You can choose whether to upgrade all hosts in one phase by running a single Ansible playbook, or upgrade the control plane (master components) and nodes in multiple phases using separate playbooks.

Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OKD 3.10 Release.

If your OKD cluster uses GlusterFS pods, you must perform the upgrade in multiple phases. See Special Considerations When Using Containerized GlusterFS for details on how to upgrade with GlusterFS.

When upgrading in separate phases, the control plane phase includes upgrading:

  • master components

  • node services running on masters

  • Docker or CRI-O running on masters

  • Docker or CRI-O running on any stand-alone etcd hosts

When upgrading only the nodes, the control plane must already be upgraded. The node phase includes upgrading:

  • node services running on stand-alone nodes

  • Docker or CRI-O running on stand-alone nodes

Nodes running master components are not included during the node upgrade phase, even though they have node services and Docker running on them. Instead, they are upgraded as part of the control plane upgrade phase. This ensures node services and Docker on masters are not upgraded twice (once during the control plane phase and again during the node phase).

Customizing Node Upgrades

Whether upgrading in a single or multiple phases, you can customize how the node portion of the upgrade progresses by passing certain Ansible variables to an upgrade playbook using the -e option.

Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OKD 3.10 Release.

The openshift_upgrade_nodes_serial variable can be set to an integer or percentage to control how many node hosts are upgraded at the same time. The default is 1, upgrading nodes one at a time.

For example, to upgrade 20 percent of the total number of detected nodes at a time:

  1. $ ansible-playbook -i <path/to/inventory/file> \
  2.     </path/to/upgrade/playbook> \
  3.     -e openshift_upgrade_nodes_serial="20%"

The openshift_upgrade_nodes_label variable allows you to specify that only nodes with a certain label are upgraded. This can also be combined with the openshift_upgrade_nodes_serial variable.

For example, to only upgrade nodes in the group1 region, two at a time:

  1. $ ansible-playbook -i <path/to/inventory/file> \
  2.     </path/to/upgrade/playbook> \
  3.     -e openshift_upgrade_nodes_serial="2" \
  4.     -e openshift_upgrade_nodes_label="region=group1"

See Managing Nodes for more on node labels.

The openshift_upgrade_nodes_max_fail_percentage variable allows you to specify how many nodes may fail in each batch. The percentage of failure must exceed your value before the playbook aborts the upgrade.

The openshift_upgrade_nodes_drain_timeout variable allows you to specify the length of time to wait before giving up.

In this example, 10 nodes are upgraded at a time, the upgrade will abort if more than 20 percent of the nodes fail, and there is a 600-second wait to drain the node:

  1. $ ansible-playbook -i <path/to/inventory/file> \
  2.     </path/to/upgrade/playbook> \
  3.     -e openshift_upgrade_nodes_serial=10 \
  4.     -e openshift_upgrade_nodes_max_fail_percentage=20 \
  5.     -e openshift_upgrade_nodes_drain_timeout=600

Customizing Upgrades With Ansible Hooks

When upgrading OKD, you can execute custom tasks during specific operations through a system called hooks. Hooks allow cluster administrators to provide files defining tasks to execute before and/or after specific areas during upgrades. This can be very helpful to validate or modify custom infrastructure when upgrading OKD.

It is important to remember that when a hook fails, the operation fails. This means a good hook can run multiple times and provide the same results. A great hook is idempotent.

Limitations

  • Hooks have no defined or versioned interface. They can use internal openshift-ansible variables, but there is no guarantee these will remain in future releases. In the future, hooks may be versioned, giving you advance warning that your hook needs to be updated to work with the latest openshift-ansible.

  • Hooks have no error handling, so an error in a hook will halt the upgrade process. The problem will need to be addressed and the upgrade re-run.

Using Hooks

Hooks are defined in the hosts inventory file under the OSEv3:vars section.

Each hook must point to a YAML file which defines Ansible tasks. This file will be used as an include, meaning that the file cannot be a playbook, but a set of tasks. Best practice suggests using absolute paths to the hook file to avoid any ambiguity.

Example Hook Definitions in an Inventory File

  1. [OSEv3:vars]
  2. openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
  3. openshift_master_upgrade_hook=/usr/share/custom/master.yml
  4. openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml
  6. openshift_node_upgrade_pre_hook=/usr/share/custom/pre_node.yml
  7. openshift_node_upgrade_hook=/usr/share/custom/node.yml
  8. openshift_node_upgrade_post_hook=/usr/share/custom/post_node.yml

Example pre_master.yml Task

  1. ---
  2. # Trivial example forcing an operator to ack the start of an upgrade
  3. # file=/usr/share/custom/pre_master.yml
  5. - name: note the start of a master upgrade
  6.   debug:
  7.       msg: "Master upgrade of {{ inventory_hostname }} is about to start"
  9. - name: require an operator agree to start an upgrade
  10.   pause:
  11.       prompt: "Hit enter to start the master upgrade"

Available Upgrade Hooks

Table 1. Master Upgrade Hooks
Hook NameDescription

openshift_master_upgrade_pre_hook

  • Runs before each master is upgraded.

  • This hook runs against each master in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

openshift_master_upgrade_hook

  • Runs after each master is upgraded, but before its service or system restart.

  • This hook runs against each master in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

openshift_master_upgrade_post_hook

  • Runs after each master is upgraded and has had its service or system restart.

  • This hook runs against each master in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

Table 2. Node Upgrade Hooks
Hook NameDescription

openshift_node_upgrade_pre_hook

  • Runs before each node is upgraded.

  • This hook runs against each node in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

openshift_node_upgrade_hook

  • Runs after each node is upgraded, but before it’s marked schedulable again.

  • This hook runs against each node in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

openshift_node_upgrade_post_hook

  • Runs after each node is upgraded; it’s the last node upgrade action.

  • This hook runs against each node in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

Upgrading to the Latest OKD 3.10 Release

To upgrade an existing OKD 3.9 or 3.10 cluster to the latest 3.10 release:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Ensure the openshift_deployment_type parameter in your inventory file is set to openshift-enterprise.

  3. If you want to enable rolling, full system restarts of the hosts, you can set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the default value services performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details.

  4. At this point, you can choose to run the upgrade in a single or multiple phases. See Upgrading the Control Plane and Nodes in Separate Phases for more details which components are upgraded in each phase.

    If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify its location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts for the last inventory file that was used, if needed.

    • Option A) Upgrade control plane and nodes in a single phase.

      Run the upgrade.yml playbook to upgrade the cluster in a single phase using one playbook; the control plane is still upgraded first, then nodes in-place:

      1. # ansible-playbook -i </path/to/inventory/file> \
      2.     /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.yml

    • Option B) Upgrade the control plane and nodes in separate phases.

      1. To upgrade only the control plane, run the upgrade_control_plane.yaml playbook:

        1. # ansible-playbook -i </path/to/inventory/file> \
        2.     /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade_control_plane.yml

      2. To upgrade only the nodes, run the upgrade_nodes.yaml playbook:

        1. # ansible-playbook -i </path/to/inventory/file> \
        2.     /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade_nodes.yml \
        3.     [-e <customized_node_upgrade_variables>] (1)
        1See Customizing Node Upgrades for any desired <customized_node_upgrade_variables>.

        If you are upgrading the nodes in groups as described in Customizing Node Upgrades, continue invoking the upgrade_nodes.yml playbook until all nodes have been successfully upgraded.

  1. After all master and node upgrades have completed, reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features you have previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

Upgrading the EFK Logging Stack

To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.

  1. If you have not already done so, see Specifying Logging Ansible Variables in the Aggregating Container Logs topic and update your Ansible inventory file to at least set the following required variable within the [OSEv3:vars] section:

    1. [OSEv3:vars]
    3. openshift_logging_install_logging=true (1)
    1Enables the ability to upgrade the logging stack.

  2. Add any other openshift_logging_* variables that you want to specify to override the defaults, as described in Specifying Logging Ansible Variables.

  3. When you have finished updating your inventory file, follow the instructions in Deploying the EFK Stack to run the openshift-logging/config.yml playbook and complete the logging deployment upgrade.

If your Fluentd DeploymentConfig and DaemonSet for the EFK components are already set with:

  1.         image: <image_name>:<vX.Y>
  2.         imagePullPolicy: IfNotPresent

The latest version <image_name> might not be pulled if there is already one with the same <image_name:vX.Y> stored locally on the node where the pod is being re-deployed. If so, manually change the DeploymentConfig and DaemonSet to imagePullPolicy: Always to make sure it is re-pulled.

Upgrading Cluster Metrics

To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-metrics/config.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.

  1. If you have not already done so, see Specifying Metrics Ansible Variables in the Enabling Cluster Metrics topic and update your Ansible inventory file to at least set the following required variables within the [OSEv3:vars] section:

    1. [OSEv3:vars]
    3. openshift_metrics_install_metrics=true (1)
    4. openshift_metrics_hawkular_hostname=<fqdn> (2)
    5. openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) (3)
    1Enables the ability to upgrade the metrics deployment.
    2Used for the Hawkular Metrics route. Specify a fully qualified domain name.
    3Choose the same type as the previous deployment.

  2. Add any other openshift_metrics_* variables that you want to specify to override the defaults, as described in Specifying Metrics Ansible Variables.

  3. When you have finished updating your inventory file, follow the instructions in Deploying the Metrics Deployment to run the openshift-metrics/config.yml playbook and complete the metrics deployment upgrade.

Special Considerations for Large-scale Upgrades

For large-scale cluster upgrades, which involve at least 10 worker nodes and thousands of projects and pods, the API object storage migration should be performed prior to running the upgrade playbooks, and then again after the upgrade has successfully completed. Otherwise, the upgrade process will fail.

Refer to the Running the pre- and post- API server model object migration outside of the upgrade window section of the Recommendations for large-scale OpenShift upgrades for further guidance.

Special Considerations for Mixed Environments

If your OKD cluster uses a mixed environment or gcePD storage, you need to take more steps before you upgrade it.

Before you upgrade a mixed environment, such as one with Red Hat Enterprise Linux (RHEL) and RHEL Atomic Host, set values in the inventory file for both the openshift_pkg_version and openshift_image_tag parameters. Setting these values ensures that all nodes in your cluster run the same version of OKD. While this is a best practice for major updates, such as from OKD 2 to OKD 3, setting these values are mandatory for minor version upgrades.

For example, to upgrade from OKD 3.9 to OKD 3.10, set the following parameters and values:

  1. openshift_pkg_version=-3.10.16
  2. openshift_image_tag=v3.10.16

These parameters can also be present in other, non-mixed, environments.

Special Considerations When Using Containerized GlusterFS

When upgrading OKD, you must upgrade the set of nodes where GlusterFS pods are running.

Special consideration must be taken when upgrading these nodes, as drain and unschedule will not terminate and evacuate the GlusterFS pods because they are running as part of a daemonset.

There is also the potential for someone to run an upgrade on multiple nodes at the same time, which would lead to data availability issues if more than one was hosting GlusterFS pods.

Even if a serial upgrade is running, there is no guarantee sufficient time will be given for GlusterFS to complete all of its healing operations before GlusterFS on the next node is terminated. This could leave the cluster in a bad or unknown state. Therefore, the following procedure is recommended.

  1. Upgrade the control plane (the master nodes and etcd nodes).

  2. Upgrade standard infra nodes (router, registry, logging, and metrics).

    If any of the nodes in those groups are running GlusterFS, perform step 4 of this procedure at the same time. GlusterFS nodes must be upgraded along with other nodes in their class (app versus infra), one at a time.

  3. Upgrade standard nodes running application containers.

    If any of the nodes in those groups are running GlusterFS, perform step 4 of this procedure at the same time. GlusterFS nodes must be upgraded along with other nodes in their class (app versus infra), one at a time.

  4. Upgrade the OKD nodes running GlusterFS one at a time.

    1. Add a label to the node you want to upgrade:

      1. $ oc label node <node_name> type=upgrade

    2. To run the upgrade playbook on a single node, use -e openshift_upgrade_nodes_label="type=upgrade".

    3. Wait for the GlusterFS pod to respawn and appear.

    4. oc rsh into the pod and verify all volumes are healed:

      1. $ oc rsh <GlusterFS_pod_name>
      2. $ for vol in `gluster volume list`; do gluster volume heal $vol info; done

      Ensure all of the volumes are healed and there are no outstanding tasks. The heal info command lists all pending entries for a given volume’s heal process. A volume is considered healed when Number of entries for that volume is 0.

    5. Remove the upgrade label and go to the next GlusterFS node.

      1. $ oc label node <node_name> type-

Special Considerations When Using gcePD

Because the default gcePD storage provider uses an RWO (Read-Write Only) access mode, you cannot perform a rolling upgrade on the registry or scale the registry to multiple pods. Therefore, when upgrading OKD, you must specify the following environment variables in your Ansible inventory file:

  1. [OSEv3:vars]
  3. openshift_hosted_registry_storage_provider=gcs
  4. openshift_hosted_registry_storage_gcs_bucket=bucket01
  5. openshift_hosted_registry_storage_gcs_keyfile=test.key
  6. openshift_hosted_registry_storage_gcs_rootdirectory=/registry

Verifying the Upgrade

Ensure that the:

  • cluster is healthy,

  • master, node, and etcd services or static pods are running well,

  • the OKD, docker-registry, and router versions are correct,

  • the original applications are still available and the new application can be created, and

  • running oc adm diagnostics produces no errors.

To verify the upgrade:

  1. Check that all nodes are marked as Ready:

    1. # oc get nodes
    2. NAME                     STATUS    ROLES        AGE       VERSION
    3. master1.example.com      Ready     master       47d       v1.10.0+b81c8f8
    4. master2.example.com      Ready     master       47d       v1.10.0+b81c8f8
    5. master3.example.com      Ready     master       47d       v1.10.0+b81c8f8
    6. infra-node1.example.com  Ready     infra        47d       v1.10.0+b81c8f8
    7. infra-node2.example.com  Ready     infra        47d       v1.10.0+b81c8f8
    8. node1.example.com        Ready     compute      47d       v1.10.0+b81c8f8
    9. node2.example.com        Ready     compute      47d       v1.10.0+b81c8f8

  2. Verify the static pods for the control plane are running:

    1. # oc get pods -n kube-system
    2. NAME                                 READY     STATUS    RESTARTS   AGE
    3. master-api-master1.example.com           1/1       Running   4          1h
    4. master-controllers-master1.example.com   1/1       Running   3          1h
    5. master-etcd-master1.example.com          1/1       Running   6          5d
    6. [...]

  3. Verify that you are running the expected versions of the docker-registry and router images, if deployed.

    1. # oc get -n default dc/docker-registry -o json | grep \"image\"
    2.     "image": "openshift/origin-docker-registry:v3.10",
    3. # oc get -n default dc/router -o json | grep \"image\"
    4.     "image": "openshift/origin-haproxy-router:v3.10",

  4. If you upgraded from Origin 1.0 to Origin 1.1, verify in your old /etc/sysconfig/openshift-master and /etc/sysconfig/openshift-node files that any custom configuration is added to your new /etc/sysconfig/origin-master and /etc/sysconfig/origin-node files.

  5. Use the diagnostics tool on the master to look for common issues:

    1. # oc adm diagnostics
    2. ...
    3. [Note] Summary of diagnostics execution:
    4. [Note] Completed with no errors or warnings seen.