Installing Kubeflow

Instructions for Kubeflow deployment with Kubeflow Operator

This guide describes how to use the Kubeflow Operator to deploy Kubeflow. As mentioned in the Operator introduction, the Operator also allows you to monitor and manage the Kubeflow installation beyond the initial installation.

Prerequisites

  • Kubeflow Operator needs to be deployed on your cluster for rest of steps to work. Please follow the Install the Kubeflow Operator guide to install the Kubeflow Operator

Deployment Instructions

The Kubeflow Operator uses the KfDef as its custom resource. You can compose a KfDef configuration or pick a default KfDef from the Kubeflow manifests repo. Keep in mind choosing the release that will work with the Kubeflow Operator.

Prepare KfDef configuration

The metadata.name field must be set for the KfDef manifests whether it is downloaded from the Kubeflow manifests repo or is originally written. Following example shows how to prepare the KfDef manifests

  1. # download a default KfDef configuration from remote repo
  2. export KFDEF_URL=https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.yaml
  3. export KFDEF=$(echo "${KFDEF_URL}" | rev | cut -d/ -f1 | rev)
  4. curl -L ${KFDEF_URL} > ${KFDEF}
  6. # add metadata.name field
  7. # Note: yq can be installed from https://github.com/mikefarah/yq
  8. export KUBEFLOW_DEPLOYMENT_NAME=kubeflow
  9. yq w ${KFDEF} 'metadata.name' ${KUBEFLOW_DEPLOYMENT_NAME} > ${KFDEF}.tmp && mv ${KFDEF}.tmp ${KFDEF}

Deploy the Kubeflow with the Kubeflow Operator

Kubeflow Operator is watching on any KfDef resource in the Kubernetes cluster. Depends on how the operator is installed, there are a couple of ways to start the Kubeflow deployment. You can always manually run with following commands

  1. # create the namespace for Kubeflow deployment
  2. KUBEFLOW_NAMESPACE=kubeflow
  3. kubectl create ns ${KUBEFLOW_NAMESPACE}
  5. # create the KfDef custom resource
  6. kubectl create -f ${KFDEF} -n ${KUBEFLOW_NAMESPACE}

Note: in the example above, ${KFDEF} points to a local KfDef configuration file, however, it can also points to a remote URL containing a valid KfDef configuration.

Watch the deployment progress

The Kubeflow deployment is carried on by the operator, you can watch the progress with this command

  1. kubectl logs deployment/kubeflow-operator -n ${OPERATOR_NAMESPACE} -f

Verify the Kubeflow deployment by monitoring the pods in the ${KUBEFLOW_NAMESPACE}

  1. kubectl get pod -n ${KUBEFLOW_NAMESPACE}
  3. NAME                                                     READY   STATUS    RESTARTS   AGE
  4. admission-webhook-bootstrap-stateful-set-0               1/1     Running   3          2m26s
  5. admission-webhook-deployment-5bc5f97cfd-chjnm            1/1     Running   2          46s
  6. application-controller-stateful-set-0                    1/1     Running   0          2m30s
  7. argo-ui-669bcd8bfc-5dk5c                                 1/1     Running   0          45s
  8. cache-deployer-deployment-b75f5c5f6-42n6l                2/2     Running   1          45s
  9. cache-server-85bccd99bd-tfrnr                            2/2     Running   0          44s
  10. centraldashboard-8849f64cf-l45zc                         1/1     Running   0          44s
  11. jupyter-web-app-deployment-6c568f4cbc-pd68m              1/1     Running   0          43s
  12. kubeflow-pipelines-profile-controller-846cc56f44-cmbbf   1/1     Running   0          43s
  13. metacontroller-0                                         1/1     Running   0          96s
  14. metadata-writer-59d755696c-fh6px                         2/2     Running   0          43s
  15. minio-d45d44d4f-rmxft                                    1/1     Running   0          42s
  16. ml-pipeline-6bc56cd86d-kn7zt                             1/2     Running   0          42s
  17. ml-pipeline-persistenceagent-6f99b56974-x2f52            2/2     Running   0          41s
  18. ml-pipeline-scheduledworkflow-d596b8bd-qdz6m             2/2     Running   0          41s
  19. ml-pipeline-ui-8695cc6b46-hr8p5                          2/2     Running   0          40s
  20. ml-pipeline-viewer-crd-5998ff7f56-5rn4s                  2/2     Running   2          40s
  21. ml-pipeline-visualizationserver-cbbb5b5b-w7rbd           2/2     Running   0          39s
  22. mysql-76597cf5b5-jpsrx                                   1/2     Running   0          39s
  23. notebook-controller-deployment-756587d86-fffg8           1/1     Running   0          38s
  24. profiles-deployment-865b78d47f-pbgl4                     2/2     Running   0          38s
  25. workflow-controller-54dccb7dc4-hkg9s                     1/1     Running   0          37s

Last modified 17.03.2021: Create “Distributions” with kfctl + Kubeflow Operator (#2492) (c792fe57)