Minikube for Kubeflow

Quickly get Kubeflow running locally

This document outlines the steps you can take to get your local installationof Kubeflow running on top of Minikube. Minikube runs a simple, single-nodeKubernetes cluster inside a virtual machine (VM).

By the end of this document, you’ll have a local installation of Minikube kubernetes cluster along with all the default core components ofKubeflow deployed as services in the pods. You should be able to access Jupyter notebooks and the Kubeflow Dashboard.

Prerequisites

  • Laptop, Desktop or a Workstation
    • >= 12GB RAM
    • >= 8 CPU Cores
    • ~100GB or more Disk Capacity
    • Optional: GPU card
  • Mac OS X or Linux (Ubuntu/RedHat/CentOS)
  • sudo or admin access on the local machine
  • Access to an Internet connection with reasonable bandwidth
  • A hypervisor such as VirtualBox, Vmware Fusion, KVM etc.

If you already have a hypervisor on your system, you can follow the Quick Setup to do a guided Minikube setup.

Install a Hypervisor

If you do not already have a hypervisor or a virtualizer installed, install a new one. Once the hypervisor is installed, you don’t need to start or use it directly. Minikube will automatically invoke the hypervisor to start the VM.

Mac OS X

Install Virtual Box or VMware Fusion.

Ubuntu

Install Virtual Box or KVM.

The KVM2 driver is intended to replace KVM driver. The KVM2 driver is maintained by the minikube team, and is built, tested and released with minikube.For installing KVM:

  1. # Install libvirt and qemu-kvm on your system, e.g.
  2. # Debian/Ubuntu (for older Debian/Ubuntu versions, you may have to use libvirt-bin instead of libvirt-clients and libvirt-daemon-system)
  3. $ sudo apt install libvirt-clients libvirt-daemon-system qemu-kvm
  5. # Add yourself to the libvirt group so you don't need to sudo
  6. # NOTE: For older Debian/Ubuntu versions change the group to `libvirtd`
  7. $ sudo usermod -a -G libvirt $(whoami)
  9. # Update your current session for the group change to take effect
  10. # NOTE: For older Debian/Ubuntu versions change the group to `libvirtd`
  11. $ newgrp libvirt

Then install the driver itself:

  1. curl -Lo docker-machine-driver-kvm2 https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2 \
  2. && chmod +x docker-machine-driver-kvm2 \
  3. && sudo cp docker-machine-driver-kvm2 /usr/local/bin/ \
  4. && rm docker-machine-driver-kvm2
CentOS

Install Virtual Box or KVM.

For installing KVM:

  1. # Install libvirt and qemu-kvm on your system
  2. $ sudo yum install libvirt-daemon-kvm qemu-kvm
  4. # Add yourself to the libvirt group so you don't need to sudo
  5. $ sudo usermod -a -G libvirt $(whoami)
  7. # Update your current session for the group change to take effect
  8. $ newgrp libvirt

Then install the driver itself:

  1. curl -Lo docker-machine-driver-kvm2 https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2 \
  2. && chmod +x docker-machine-driver-kvm2 \
  3. && sudo cp docker-machine-driver-kvm2 /usr/local/bin/ \
  4. && rm docker-machine-driver-kvm2

Quick Setup

The following describes a script driven installation that you can use to deploy allthe necessary components including kubectl, minikube, kfctl along with Kubeflow itself. The script requires input from you on some configuration information and then it drives the rest of the installation. Run the following to start the installation:

  1. export KUBEFLOW_TAG=v0.6.2
  2. curl -O https://raw.githubusercontent.com/kubeflow/kubeflow/${KUBEFLOW_TAG}/scripts/setup-minikube.sh
  3. chmod +x setup-minikube.sh
  4. ./setup-minikube.sh

KUBEFLOW_TAG is a tag corresponding to the version to checkout such as v0.6.2 or master.

The script asks for some config input as shown below:ConfigInputs

Let us consider the example for CPUs configuration. When it asks Assign CPUs between 3..10 [6]: the 3..10 suggest the range of CPU cores available based on your host machine. [6] is the suggested default. You can choose any value within the range and enter the value or just press enter to accept the default value suggested in square brackets. In the image above, we choose the default 6 for CPUs and specified 12GB of memory explicitly. Note that:

  • You will need to specify the virtualizer installed on the system explicitly and it needs to be one of the values provided as options.
  • If you don’t want to mount any local directory into the Jupyter notebook server simply press enter instead of specifying any path. After the configuration is complete, the script will continue execution for the next few minutes and when finished successfully should output some like:LocalDeployment

When the installation finishes successfully, you can access Jupyter notebooks as described in Where to go next. If you have trouble with the installation script or run into errors, you can follow the detailed installation steps manually as described below.

Install Kubectl

GCloud SDK
  1. $ gcloud components install kubectl
Mac OS X
  1. $ brew install kubectl
Ubuntu
  1. $ sudo apt-get update && sudo apt-get install -y apt-transport-https
  2. $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
  3. $ sudo touch /etc/apt/sources.list.d/kubernetes.list
  4. $ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
  5. $ sudo apt-get update
  6. $ sudo apt-get install -y kubectl
CentOS
  1. $ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  2. [kubernetes]
  3. name=Kubernetes
  4. baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
  5. enabled=1
  6. gpgcheck=1
  7. repo_gpgcheck=1
  8. gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
  9. EOF
  11. $ sudo yum install -y kubectl

Verify kubectl installed

Try running

  1. $ kubectl version

This should output something like

  1. Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
  2. Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.7-gke.2", GitCommit:"8d9503f982872112eb283f78cefc6944af640427", GitTreeState:"clean", BuildDate:"2018-09-13T22:19:55Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Install & Start Minikube

Please see detailed instructions for Minikube installation.For quick setup instructions follow along below.

Mac OS X
  1. $ brew cask install minikube

OR

  1. $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.0/minikube-darwin-amd64
  2. $ chmod +x minikube
  3. $ sudo mv minikube /usr/local/bin/
Ubuntu or CentOS
  1. $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.0/minikube-linux-amd64
  2. $ chmod +x minikube
  3. $ sudo mv minikube /usr/local/bin/
Start your minikube cluster
  1. $ minikube start --cpus 4 --memory 8096 --disk-size=40g

This takes a couple minutes as it will talk to the hypervisor and create a VM with the specified configuration.

Notes:

  • These are the minimum recommended settings on the VM created by minikube for kubeflow deployment. You are free to adjust them higher based on your host machinecapabilities and workload requirements.
  • Using certain hypervisors might require you to set –vm-driver option specifying the driveryou want to use. In case, you have the default minikube VM already created (following detailed installation instructions), please use the following to update the VM.
  1. $ minikube stop
  2. $ minikube delete
  3. $ minikube start --cpus 4 --memory 8096 --disk-size=40g

Install Kubeflow on an existing Kubernetes cluster

Now that you have a Kubernetes cluster running - the Minikube cluster - follow theexisting Kubernetes cluster instructions for installingKubeflow.

Where to go next

You can use the following command to set up port forwarding and access the Web UIs:

  1. export NAMESPACE=istio-system
  2. kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80

Now you can access the Kubeflow dashboard at http://localhost:8080/ and Jupyternotebooks at http://localhost:8080/notebooks/.

For Jupyter notebooks, you can use any username and password to log in.Follow the guide to setting up your Jupyter notebooks on Kubeflow.

For further exploration refer to the documentation.