Upgrade from v1.0.0 to v1.0.1

This document describes how to upgrade from Harvester v1.0.0 to v1.0.1.

Note we are still working towards zero-downtime upgrade, due to some known issues please follow the steps below before you upgrade your Harvester cluster:

Upgrade from v1.0.0 to v1.0.1 - 图1caution

  • Before you upgrade your Harvester cluster, we highly recommend:
    • Shutting down all your VMs (Harvester GUI -> Virtual Machines -> Select VMs -> Actions -> Stop).
    • Back up your VMs.
  • Do not operate the cluster during an upgrade. For example, creating new VMs, uploading new images, etc.
  • Make sure your hardware meets the preferred hardware requirements. This is due to there will be intermediate resources consumed by an upgrade.
  • Make sure each node has at least 25 GB of free space (df -h /usr/local/).

Upgrade from v1.0.0 to v1.0.1 - 图2caution

  • Make sure all nodes’ times are in sync. Using an NTP server to synchronize time is recommended. If an NTP server is not configured during the installation, you can manually add an NTP server on each node:

    1. $ sudo -i
    3. # Add time servers
    4. $ vim /etc/systemd/timesyncd.conf
    5. [ntp]
    6. NTP=0.pool.ntp.org
    8. # Enable and start the systemd-timesyncd
    9. $ timedatectl set-ntp true
    11. # Check status
    12. $ timedatectl status

Upgrade from v1.0.0 to v1.0.1 - 图3caution

  • NICs that connect to a PCI bridge might be renamed after an upgrade. Please check the knowledge base article for further information.

Create a version

  • Log in to one of your server nodes.

  • Become root and create a version:

    1. [email protected]:~> sudo -i
    3. node1:~ # kubectl create -f https://releases.rancher.com/harvester/v1.0.1/version.yaml
    4. version.harvesterhci.io/1.0.1 created

Upgrade from v1.0.0 to v1.0.1 - 图4note

By default, the ISO image is downloaded from the Harvester release server. To speed up the upgrade and make the upgrade progress smoother, the user can also download the ISO file to a local HTTP server first and substitute the isoURL value in the version.yaml manifest.

e.g.,

  1. # Download the ISO from release server first, assume it's store in http://10.10.0.1/harvester.iso
  2. $ sudo -i
  3. $ curl -fL https://releases.rancher.com/harvester/v1.0.1/version.yaml -o version.yaml
  4. $ vim version.yaml 
  5. apiVersion: harvesterhci.io/v1beta1
  6. kind: Version
  7. metadata:
  8.   name: v1.0.1
  9.   namespace: harvester-system
  10. spec:
  11.   isoChecksum: <SHA-512 checksum of the ISO> 
  12.   isoURL: http://10.10.0.1/harvester.iso
  13.   releaseDate: '20220408'

Start the upgrade

  • Make sure to read the Warning paragraph at the top of this document first.

  • Navigate to Harvester GUI and click the upgrade button on the Dashboard page.

    Upgrade from v1.0.0 to v1.0.1 - 图5

  • Select a version to start upgrading.

    Upgrade from v1.0.0 to v1.0.1 - 图6

  • Click the circle on the top to display the upgrade progress. Upgrade from v1.0.0 to v1.0.1 - 图7

Known issues

Fail to download upgrade image

  • Description

    Downloading the upgrade image can’t complete.

    Upgrade from v1.0.0 to v1.0.1 - 图8

  • Workaround

    We can delete the current upgrade and start over.

    1. # log in to one of the server nodes
    2. $ sudo -i
    4. # list current upgrade, the name changes between deployments
    5. $ kubectl get upgrades.harvesterhci.io -n harvester-system
    6. NAMESPACE          NAME                 AGE
    7. harvester-system   hvst-upgrade-77cks   119m
    9. $ kubectl delete upgrades.harvesterhci.io hvst-upgrade-77cks -n harvester-system

    We recommend mirroring the ISO file to a local webserver, please check the notes in the previous section.

Stuck in Upgrading System Service

  • Description

    • The upgrade is stuck at Upgrading System service.

    • Similar logs are found in rancher pods:

      1. [ERROR] available chart version (100.0.2+up0.3.8) for fleet is less than the min version (100.0.3+up0.3.9-rc1) 
      2. [ERROR] Failed to find system chart fleet will try again in 5 seconds: no chart name found

  • Workaround

    Delete rancher cluster repositories and restart rancher pods.

    1. # login to a server node and become root first
    2. kubectl delete clusterrepos.catalog.cattle.io rancher-charts
    3. kubectl delete clusterrepos.catalog.cattle.io rancher-rke2-charts
    4. kubectl delete clusterrepos.catalog.cattle.io rancher-partner-charts
    5. kubectl delete settings.management.cattle.io chart-default-branch
    6. kubectl rollout restart deployment rancher -n cattle-system

  • Related issues

VMs fail to migrate

  • Description

    • A node keeps waiting in Pre-draining state.
    • There are VMs on that node (checking for virt-launcher-xxx pods) and they can’t be live-migrated out of the node.

  • Workaround

    Shutdown the VMs, you can do this by:

    • Using the GUI.
    • Using the virtctl command.

  • Related issues

fleet-local/local: another operation (install/upgrade/rollback) is in progress

Single node upgrade might fail if node name is too long (>24 characters)