Upgrade Kubeflow

Upgrade Kubeflow

Upgrading your Kubeflow installation on Google Cloud

Before you start

To better understand upgrade process, you should read the following sections first:

This guide assumes the following settings:

The ${MGMT_DIR} and ${MGMT_NAME} environment variables are the same as in Management cluster setup.
The ${KF_DIR}, ${KF_NAME}, ${CLIENT_ID} and ${CLIENT_SECRET} environment variables are the same as in Deploy using kubectl and kpt.

General upgrade instructions

Both management cluster and Kubeflow cluster follow the same instance and upstream folder convention. To upgrade, you’ll typically need to update packages in upstream to the new version and repeat the make apply-<subcommand> commands in their respective deployment process.

However, specific upgrades might need manual actions below.

Upgrading management cluster

Upgrade management cluster from v1.1 to v1.2

Note, there is no change for management cluster from v1.2.0 to v1.2.1. So there is no need to upgrade if you are already on v1.2.0.

The instructions below assume that your current working directory is
```
cd "${MGMT_DIR}"
```

Use your management cluster’s kubectl context:

# Look at all your contexts
kubectl config get-contexts
# Select your management cluster's context
kubectl config use-context "${MGMT_NAME}"
# Verify the context connects to the cluster properly
kubectl get namespace

If you are using a different enviroment, you can always reconfigure the context by:

make create-context

Check your existing config connector version:

# For Kubeflow v1.1, it should be 1.15.1
$ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'
1.15.1

Uninstall the old config connector in the management cluster:

kubectl delete sts,deploy,po,svc,roles,clusterroles,clusterrolebindings --all-namespaces -l cnrm.cloud.google.com/system=true --wait=true
kubectl delete validatingwebhookconfiguration abandon-on-uninstall.cnrm.cloud.google.com --ignore-not-found --wait=true
kubectl delete validatingwebhookconfiguration validating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true
kubectl delete mutatingwebhookconfiguration mutating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true

These commands uninstall the config connector without removing your resources.

Replace your ./Makefile with the version in Kubeflow v1.2.0: https://github.com/kubeflow/gcp-blueprints/blob/v1.2.0/management/Makefile.

If you made any customizations in ./Makefile, you should merge your changes with the upstream version. We’ve refactored the Makefile to move substantial commands into the upstream package, so hopefully future upgrades won’t require a manual merge of the Makefile.
Update ./upstream/management package:
```
make update
```

Use kpt to set user values:

kpt cfg set -R . name ${MGMT_NAME}
kpt cfg set -R . gcloud.core.project ${MGMT_PROJECT}
kpt cfg set -R . location ${LOCATION}

Note, you can find out which setters exist in a package and what there current values are by:

kpt cfg list-setters .

Apply upgraded config connector:
```
make apply-kcc
```
Note, you can optionally also run make apply-cluster, but it should be the same as your existing management cluster.

Check that your config connector upgrade is successful:

# For Kubeflow v1.2, it should be 1.29.0
$ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'
1.29.0

Upgrading Kubeflow cluster

DISCLAIMERS:

The upgrade process depends on each Kubeflow application to handle the upgrade properly. There’s no guarantee on data completeness unless the application provides such a guarantee.
You are recommended to back up your data before an upgrade.
Upgrading Kubeflow cluster can be a disruptive process, please schedule some downtime and communicate with your users.

To upgrade from specific versions of Kubeflow, you may need to take certain manual actions — refer to specific sections in the guidelines below.

General instructions for upgrading Kubeflow:

The instructions below assume that:

Your current working directory is:
```
cd ${KF_DIR}
```

Your kubectl uses a context that connects to your Kubeflow cluster

# List your existing contexts
kubectl config get-contexts
# Use the context that connects to your Kubeflow cluster
kubectl config use-context ${KF_NAME}

Edit the Makefile at ./Makefile and change MANIFESTS_URL to point at the version of Kubeflow manifests you want to use
- Refer to the kpt docs for more info about supported dependencies
Update the local copies:
```
make update
```
Redeploy:
```
make apply
```
To evaluate the changes before deploying them you can run make hydrate and then compare the contents of .build to what is currently deployed.

Upgrade Kubeflow cluster from v1.1 to v1.2

Note, v1.2.1 pins Kubernetes Cluster version to 1.16 to resolve #198. There is no need to upgrade from v1.2.0 to v1.2.1 if your Kubeflow cluster is working properly.

The instructions below assume

Your current working directory is:
```
cd ${KF_DIR}
```

Your kubectl uses a context that connects to your Kubeflow cluster:

# List your existing contexts
kubectl config get-contexts
# Use the context that connects to your Kubeflow cluster
kubectl config use-context ${KF_NAME}

(Recommended) Replace your ./Makefile with the version in Kubeflow v1.2.0: https://github.com/kubeflow/gcp-blueprints/blob/v1.2.0/kubeflow/Makefile.

If you made any customizations in ./Makefile, you should merge your changes with the upstream version.

This step is recommended, because we introduced usability improvements and fixed compatibility for newer Kustomize versions (while still being compatible with Kustomize v3.2.1) to the Makefile. However, the deployment process is backward-compatible, so this is recommended, but not required.
Update ./upstream/manifests package:
```
make update
```
Before applying new resources, you need to delete some immutable resources that were updated in this release:
```
kubectl delete statefulset kfserving-controller-manager -n kubeflow --wait
kubectl delete crds experiments.kubeflow.org suggestions.kubeflow.org trials.kubeflow.org
```
WARNING: This step deletes all Katib running resources.

Refer to a github comment in the v1.2 release issue for more details.
Redeploy:
```
make apply
```
To evaluate the changes before deploying them you can:
1. Run make hydrate.
2. Compare the contents of .build with a historic version with tools like git diff.

Last modified 20.04.2021: Apply Docs Restructure to `v1.2-branch` = update `v1.2-branch` to current `master` v2 (#2612) (4e2602bd)