Kubeflow Deployment with kfctl_k8s_istio

Instructions for installing Kubeflow on your existing Kubernetes cluster using kfctl_k8s_istio config

This config creates a vanilla deployment of Kubeflow with all its core components without any external dependencies. The deployment can be customized based on your environment needs.

Maintainer and supporter: Kubeflow community

Deploy Kubeflow

This Kubeflow deployment requires a default StorageClass with a dynamic volume provisioner. Verify the provisioner field of your default StorageClass definition.If you don’t have a provisioner, ensure that you have configured volume provisioning in your Kubernetes cluster appropriately as mentioned below.

Follow these steps to deploy Kubeflow:

  1. tar -xvf kfctl_<release tag>_<platform>.tar.gz

Script to download and unpack for Linux/MacOS:

  1. opsys=linux # darwin for Mac
  3. curl -s https://api.github.com/repos/kubeflow/kubeflow/releases/latest |\
  4.     grep browser_download |\
  5.     grep $opsys |\
  6.     cut -d '"' -f 4 |\
  7.     xargs curl -O -L && \
  8.     tar -zvxf kfctl_*_${opsys}.tar.gz
  • Run the following commands to set up and deploy Kubeflow. The code below includes an optional command to add the binary kfctl to your path. If you don’t add the binary to your path, you must use the full path to the kfctl binary each time you run it.
  1.    # Add kfctl to PATH, to make the kfctl binary easier to use.
  2.    export PATH=$PATH:"<path to kfctl>"
  3.    export KFAPP="<your choice of application directory name>"
  4.    # Installs Istio by default. Comment out Istio components in the config file to skip Istio installation. See https://github.com/kubeflow/kubeflow/pull/3663
  5.    export CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6-branch/bootstrap/config/kfctl_k8s_istio.0.6.2.yaml"
  7.    kfctl init ${KFAPP} --config=${CONFIG} -V
  8.    cd ${KFAPP}
  9.    kfctl generate all -V
  10.    kfctl apply all -V
  • ${KFAPP} - the name of a directory where you want Kubeflowconfigurations to be stored. This directory is created when you runkfctl init. If you want a custom deployment name, specify that name here.The value of this variable becomes the name of your deployment.The value of this variable cannot be greater than 25 characters. It mustcontain just the directory name, not the full path to the directory.The content of this directory is described in the next section.

Check the resources deployed in namespace kubeflow:

  1.    kubectl -n kubeflow get  all

Access Kubeflow Dashboard

Once Kubeflow is deployed, the Kubeflow Dashboard can be accessed via istio-ingressgateway service. If loadbalancer is not available in your environment, NodePort or Port forwarding can be used to access the Kubeflow Dashboard.Refer Ingress Gateway guide.

Delete Kubeflow

Run the following commands to delete your deployment and reclaim all resources:

  1. cd ${KFAPP}
  2. # If you want to delete all the resources, run:
  3. kfctl delete all -V

Understanding the deployment process

The deployment process is controlled by 4 different commands:

  • init - one time set up.
  • generate - creates config files defining the various resources.
  • apply - creates or updates the resources.
  • delete - deletes the resources.

With the exception of init, all commands take an argument which describes theset of resources to apply the command to; this argument can be one of thefollowing:

  • k8s - all resources that run on Kubernetes.
  • all - platform and Kubernetes resources.

App layout

Your Kubeflow app directory contains the following files and directories:

  • ${KFAPP}/app.yaml defines configurations related to your Kubeflow deployment.
  • ${KFAPP}/kustomize: contains the YAML manifests that will be deployed.

Provisioning of Persistent Volumes in Kubernetes

Note that you can skip this step if you have a dynamic volume provisioner already installed in your cluster.

If you don’t have one:

  • You can choose to create PVs manually after deployment of Kubeflow.
  • Or install a dynamic volume provisioner like Local Path Provisioner. Ensure that the StorageClass used by this provisioner is the default StorageClass.

Troubleshooting

Persistent Volume Claims are in Pending State

Check if PersistentVolumeClaims get Bound to PersistentVolumes.

  1.    kubectl -n kubeflow get pvc

If the PersistentVolumeClaims (PVCs) are in Pending state after deployment and they are not bound to PersistentVolumes (PVs), you may have to either manually create PVs for each PVC in your Kubernetes Cluster or an alternative is to set up dynamic volume provisioning to create PVs on demand and redeploy Kubeflow after deleting existing PVCs.

Next steps