Deploying Kubeflow Pipelines on a local cluster

Deploying Kubeflow Pipelines on a local cluster

How to deploy Kubeflow Pipelines locally with kind, K3s, and K3ai for testing purposes

This guide shows how to deploy Kubeflow Pipelines standalone on a local Kubernetes cluster using:

kind
K3s
K3s on Windows Subsystem for Linux (WSL)
K3ai [alpha]

Such deployment methods can be part of your local environment using the supplied kustomize manifests for test purposes. This guide is an alternative to

Deploying Kubeflow Pipelines (KFP).

Before you get started

You should be familiar with Kubernetes, kubectl, and kustomize.
For native support of kustomize, you will need kubectl v1.14 or higher. You can download and install kubectl by following the kubectl installation guide.

kind

1. Installing kind

kind is a tool for running local Kubernetes clusters using Docker container nodes. kind was primarily designed for testing Kubernetes itself. It can also be used for local development or CI.

You can install and configure kind by following the official quick start.

To get started with kind:

On Linux:

Download and move the kind executable to your directory in your PATH by running the following commands:

curl -Lo ./kind https://kind.sigs.k8s.io/dl/{KIND_VERSION}/kind-linux-amd64 && \
chmod +x ./kind && \
mv ./kind /{YOUR_KIND_DIRECTORY}/kind

Replace the following:

{KIND_VERSION}: the kind version; for example, v0.8.1 as of the date this guide was written
{YOUR_KIND_DIRECTORY}: your directory in PATH

On macOS:

You can use Homebrew to install kind:

brew install kind

On Windows:

You can use the administrative PowerShell console to run the following commands to download and move the kind executable to a directory in your PATH:
PowerShell: Run these commands to download and move the kind executable to a directory in your PATH:
```
curl.exe -Lo kind-windows-amd64.exe https://kind.sigs.k8s.io/dl/{KIND_VERSION}/kind-windows-amd64
Move-Item .\kind-windows-amd64.exe c:\{YOUR_KIND_DIRECTORY}\kind.exe
```
Replace the following:
- {KIND_VERSION}: the kind version - for example, v0.9 (check the latest stable binary versions on the kind releases pages)
- {YOUR_KIND_DIRECTORY}: your directory for kind in PATH
{KIND_VERSION}: the kind version; for example, v0.8.1 as of the date this guide was written
{YOUR_KIND_DIRECTORY}: your directory in PATH
Alternatively, you can use Chocolatey https://chocolatey.org/packages/kind:
```
choco install kind
```

Note: kind uses containerd as a default container-runtime hence you cannot use the standard kubeflow pipeline manifests.

References:

References:

2. Creating a cluster on kind

Having installed kind, you can create a Kubernetes cluster on kind with this command:

kind create cluster

This will bootstrap a Kubernetes cluster using a pre-built node image. You can find that image on the Docker Hub kindest/node here. If you wish to build the node image yourself, you can use the kind build node-image command—see the official building image section for more details. And, to specify another image, use the --image flag.

By default, the cluster will be given the name kind. Use the --name flag to assign the cluster a different context name.

K3s

1. Setting up a cluster on K3s

K3s is a fully compliant Kubernetes distribution with the following enhancements:

Packaged as a single binary.
Lightweight storage backend based on sqlite3 as the default storage mechanism. etcd3, MySQL, Postgres also still available.
Wrapped in simple launcher that handles a lot of the complexity of TLS and options.
Secure by default with reasonable defaults for lightweight environments.
Simple but powerful “batteries-included” features have been added, such as: a local storage provider, a service load balancer, a Helm controller, and the Traefik ingress controller.
Operation of all Kubernetes control plane components is encapsulated in a single binary and process. This allows K3s to automate and manage complex cluster operations like distributing certificates.
External dependencies have been minimized (just a modern kernel and cgroup mounts needed). K3s packages required dependencies, including:
- containerd
- Flannel
- CoreDNS
- CNI
- Host utilities (iptables, socat, etc)
- Ingress controller (traefik)
- Embedded service loadbalancer
- Embedded network policy controller

You can find the the official K3s installation script to install it as a service on systemd- or openrc-based systems on the official K3s website.

To install K3s using that method, run the following command:

curl -sfL https://get.k3s.io | sh -

References:

2. Creating a cluster on K3s

To create a Kubernetes cluster on K3s, use the following command:
```
sudo k3s server &
```
This will bootstrap a Kubernetes cluster kubeconfig is written to /etc/rancher/k3s/k3s.yaml.
```
sudo k3s kubectl get node
```
(Optional) Check your cluster:
```
sudo k3s kubectl get node
```
K3s embeds the popular kubectl command directly in the binaries, so you may immediately interact with the cluster through it.
(Optional) Run the below command on a different node. NODE_TOKEN comes from /var/lib/rancher/k3s/server/node-token on your server:
```
sudo k3s agent --server https://myserver:6443 --token {YOUR_NODE_TOKEN}
```

K3s on Windows Subsystem for Linux (WSL)

1. Setting up a cluster on K3s on Windows Subsystem for Linux (WSL)

The Windows Subsystem for Linux (WSL) lets developers run a GNU/Linux environment—including most command-line tools, utilities, and applications— directly on Windows, unmodified, without the overhead of a traditional virtual machine or dualboot setup.

The full instructions for installing WSL can be found on the official Windows site.

The following steps summarize what you’ll need to set up WSL and then K3s on WSL.

Install [WSL] by following the official docs.
As per the official instructions, update WSL and download your preferred distibution:

References:

K3s on WSL: Quick Start Guide

2. Creating a cluster on K3s on WSL

Below are the steps to create a cluster on K3s in WSL

To create a Kubernetes cluster on K3s on WSL, run the following command:
```
sudo ./k3s server
```
This will bootstrap a Kubernetes cluster but you will cannot yet access from your Windows machine to the cluster itself.

Note: You can’t install K3s using the curl script because there is no supervisor (systemd or openrc) in WSL.
Download the K3s binary from https://github.com/rancher/k3s/releases/latest. Then, inside the directory where you download the K3s binary to, run this command to add execute permission to the K3s binary:
```
chmod +x k3s
```
Start K3s:
```
sudo ./k3s server
```

3. Setting up access to WSL instance

To set up access to your WSL instance:

Copy /etc/rancher/k3s/k3s.yaml from WSL to $HOME/.kube/config.
Edit the copied file by changing the server URL from https://localhost:6443 to the IP of the your WSL instance (ip addr show dev eth0) (For example, https://192.168.170.170:6443.)
Run kubectl in a Windows terminal. If you don’t kubectl installed, follow the official Kubernetes on Windows instructions.

K3ai [alpha]

K3ai is a lightweight “infrastructure in a box” designed specifically to install and configure AI tools and platforms on portable hardware, such as laptops and edge devices. This enables users to perform quick experimentations with Kubeflow on a local cluster.

K3ai’s main goal is to provide a quick way to install Kubernetes (K3s-based) and Kubeflow Pipelines with NVIDIA GPU support and TensorFlow Serving with just one line. (For Kubeflow and other component support, check K3ai’s website for updates.) To install Kubeflow Pipelines using K3ai, run the following commands:

With CPU-only support:

curl -sfL https://get.k3ai.in | bash -s -- --cpu --plugin_kfpipelines

With GPU support:

curl -sfL https://get.k3ai.in | bash -s -- --gpu --plugin_kfpipelines

For more information about K3ai, refer to the official documentation.

Deploying Kubeflow Pipelines

The installation process for Kubeflow Pipelines is the same for all three environments covered in this guide: kind, K3s, and K3ai.

Note: Process Namespace Sharing (PNS) is not mature in Argo yet - for more information go to Argo Executors and reference “pns executors” in any issue you may come across when using PNS.

To deploy the Kubeflow Pipelines, run the following commands:

# env/platform-agnostic-pns hasn't been publically released, so you will install it from master
export PIPELINE_VERSION=1.4.1
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-pns?ref=$PIPELINE_VERSION"

The Kubeflow Pipelines deployment may take several minutes to complete.

Verify that the Kubeflow Pipelines UI is accessible by port-forwarding:
```
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
```
Then, open the Kubeflow Pipelines UI at http://localhost:8080/ or - if you are using kind or K3s within a virtual machine - http://{YOUR_VM_IP_ADDRESS}:8080/

Note that K3ai will automatically print the URL for the web UI at the end of the installation process.

Note: kubectl apply -k accepts local paths and paths that are formatted as hashicorp/go-getter URLs. While the paths in the preceding commands look like URLs, they are not valid URLs.

Uninstalling Kubeflow Pipelines

Below are the steps to remove Kubeflow Pipelines on kind, K3s, or K3ai:

To uninstall Kubeflow Pipelines using your manifest file, run the following command, replacing {YOUR_MANIFEST_FILE} with the name of your manifest file:
```
kubectl delete -k {YOUR_MANIFEST_FILE}`
```

To uninstall Kubeflow Pipelines using manifests from Kubeflow Pipelines’s GitHub repository, run these commands:

export PIPELINE_VERSION=1.4.1
kubectl delete -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-pns?ref=$PIPELINE_VERSION"
kubectl delete -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"

To uninstall Kubeflow Pipelines using manifests from your local repository or file system, run the following commands:

kubectl delete -k manifests/kustomize/env/platform-agnostic-pns
kubectl delete -k manifests/kustomize/cluster-scoped-resources

Last modified 03.03.2021: Move Kubeflow Pipelines under /components (#2505) (c34470b8)