Upgrade Lifecycle

Upgrade Flow

Components

The Elemental Operator supports several ways to configure the OS version on a Cluster level.
Whenever a new ManagedOSIMage is defined, a related Fleet Bundle is also generated by the Elemental Operator.

In this context, Fleet is used to apply an upgrade.cattle.io/v1 Plan on each Cluster targeted by the ManagedOSImage.
The Plan is then executed by the System Upgrade Controller running on the downstream Elemental Cluster.

The System Upgrade Controller is an upgrade tool used by k3s and rke2 clusters. It should always be automatically installed on each provisioned Elemental cluster.
In this context the System Upgrade Controller orchestrates the application of upgrade.cattle.io/v1 Plan to the Elemental nodes.

Versioning and components lifecycle

The Fleet and System Upgrade Controller versions are controlled by Rancher.

On the Rancher cluster, you can run:

  1. kubectl get settings fleet-version system-upgrade-controller-chart-version

For more information about chart versions, you can visit the repository or read the documentation.

The charts version is determined by Rancher.
Depending on how Rancher is installed, the following environment variables can be set, for example when installing the Rancher Helm chart:

  1. CATTLE_SYSTEM_UPGRADE_CONTROLLER_CHART_VERSION
  2. CATTLE_FLEET_VERSION

Manually updating the versions is generally not recommended.
Also beware that a new CATTLE_SYSTEM_UPGRADE_CONTROLLER_CHART_VERSION will trigger a system-upgrade-controller chart update on all provisioned downstream clusters.

Verifying successful upgrades

Verifying the correct execution of the upgrade jobs is a fundamental part of ensuring the Elemental upgrade process is complete.

To troubleshoot each step within the entire upgrade process, please consult the related document.
This section only describes the last step needed to verify the correct application of the upgrade Plan on one targeted cluster.

  • Versions higher or equal to v0.13.4 of the system-upgrade-controller:

    1. kubectl -n cattle-system describe plan os-upgrader-my-upgrade

    Once the Plan is applied to all nodes, it will have a Complete status condition.

  • Older versions:

    Each Plan has a latestHash status value that should match the Plan label on each node.
    A simple script can be used to list all nodes where the plan has not be applied yet:

    1. #!/bin/bash
    3. # This script prints a list of all nodes where an upgrade plan was not applied.  
    4. # To further determine the cause of failures, you can analyze the Plan status and the related jobs.
    5. # For ex: kubectl -n cattle-system get plans,jobs
    7. # Edit this variable according to your ManagedOSImage name.
    8. MANAGED_OS_IMAGE_NAME=my-upgrade
    10. PLAN_NAME=os-upgrader-$MANAGED_OS_IMAGE_NAME
    11. LATEST_HASH=$(kubectl -n cattle-system get plan $PLAN_NAME -o=jsonpath='{.status.latestHash}')
    13. printf "Plan '$PLAN_NAME' latest hash is: '$LATEST_HASH'\n"
    15. PLAN_LABEL=plan.upgrade.cattle.io/$PLAN_NAME
    16. printf "Listing all nodes with mismatching or missing label '$PLAN_LABEL':\n"
    17. kubectl get nodes -l $PLAN_LABEL!=$LATEST_HASH