Upgrade from v1.2.2/v1.3.0 to v1.3.1

General information

An Upgrade button appears on the Dashboard screen whenever a new Harvester version that you can upgrade to becomes available. For more information, see Start an upgrade.

For air-gapped environments, see Prepare an air-gapped upgrade.

Known issues

1. Cluster upgrade stuck after the first node is upgraded

Upgrade from v1.2.2/v1.3.0 to v1.3.1 - 图1tip

To prevent this issue from occurring, label the local-kubeconfig secret before starting the upgrade process. kubectl label secret local-kubeconfig -n fleet-local cluster.x-k8s.io/cluster-name=local

When upgrading a Harvester cluster from v1.2.2 or v1.3.0 to v1.3.1, the upgrade process becomes stuck after the first node is upgraded.

Example:

Upgrade from v1.2.2/v1.3.0 to v1.3.1 - 图2

To resolve this issue, perform the following steps:

  1. Identify the cluster status:

    1. kubectl get clusters.provisioning.cattle.io local -n fleet-local -o yaml

    Example output:

    1. ...
    2.   - lastUpdateTime: "2024-06-18T23:37:39Z"
    3.     message: 'configuring bootstrap node(s) custom-9cb22ccf7984: waiting for kubelet
    4.       to update'
    5.     reason: Waiting
    6.     status: Unknown
    7.     type: Updated
    8.   - lastUpdateTime: "2024-06-18T23:37:39Z"
    9.     message: 'configuring bootstrap node(s) custom-9cb22ccf7984: waiting for kubelet
    10.       to update'
    11.     reason: Waiting
    12.     status: Unknown
    13.     type: Provisioned

    If the output includes the message waiting for kubelet, continue to the next step.

  2. Check the capi-controller-manager pod’s logs:

    1. kubectl logs -n cattle-provisioning-capi-system deployment/capi-controller-manager

    If the output is similar to the following example, the issue likely exists in the cluster.

    1. 2024-06-19T08:54:22.407423986Z W0619 08:54:22.407257       1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Node: Unauthorized
    2. 2024-06-19T08:54:22.407470069Z E0619 08:54:22.407283       1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Node: failed to list *v1.Node: Unauthorized
    3. 2024-06-19T08:55:05.153396619Z W0619 08:55:05.153190       1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Node: Unauthorized
    4. 2024-06-19T08:55:05.153438978Z E0619 08:55:05.153217       1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Node: failed to list *v1.Node: Unauthorized

  3. Apply the following workaround to resume the upgrade:

    Kill and restart the capi-controller-manager pod.

    Example:

    1. kubectl rollout restart deployment/capi-controller-manager -n cattle-provisioning-capi-system

2. Automatic image cleanup is not functioning

Because the published Harvester ISO contains an incomplete image list, automatic image cleanup cannot be performed during an upgrade from v1.2.2 to v1.3.1. This issue does not block the upgrade, and you can use this script to manually clean up container images after the upgrade is completed. For more information, see issue #6620.