Development

This doc explains how to setup a development environment so you can get started contributing.

Prerequisites

Follow the instructions below to set up your development environment. Once you meet these requirements, you can make changes and deploy your own version of kserve!

Before submitting a PR, see also CONTRIBUTING.md.

Install requirements

You must install these tools:

  1. go: KServe controller is written in Go and requires Go 1.14.0+.
  2. git: For source control.
  3. Go Module: Go’s new dependency management system.
  4. ko: For development.
  5. kubectl: For managing development environments.
  6. kustomize To customize YAMLs for different environments, requires v3.5.4+.
  7. yq yq is used in the project makefiles to parse and display YAML output. Please use yq version 3.*. Latest yq version 4.* has remove -d command so doesn’t work with the scripts.

Install Knative on a Kubernetes cluster

KServe currently requires Knative Serving for auto-scaling, canary rollout, Istio for traffic routing and ingress.

  • To install Knative components on your Kubernetes cluster, follow the installation guide or alternatively, use the Knative Operators to manage your installation. Observability, tracing and logging are optional but are often very valuable tools for troubleshooting difficult issues, they can be installed via the directions here.

  • If you start from scratch, KServe requires Kubernetes 1.17+, Knative 0.19+, Istio 1.9+.

  • If you already have Istio or Knative (e.g. from a Kubeflow install) then you don’t need to install them explictly, as long as version dependencies are satisfied.

Setup your environment

To start your environment you’ll need to set these environment variables (we recommend adding them to your .bashrc):

  1. GOPATH: If you don’t have one, simply pick a directory and add export GOPATH=...
  2. $GOPATH/bin on PATH: This is so that tooling installed via go get will work properly.

  3. KO_DOCKER_REPO: The docker repository to which developer images should be pushed (e.g. docker.io/<username>).

  4. Note: Set up a docker repository for pushing images. You can use any container image registry by adjusting the authentication methods and repository paths mentioned in the sections below.

  5. Google Container Registry quickstart

  6. Docker Hub quickstart
  7. Azure Container Registry quickstart

Note

if you are using docker hub to store your images your KO_DOCKER_REPO variable should be docker.io/<username>. Currently Docker Hub doesn’t let you create subdirs under your username.

.bashrc example:

  1. export GOPATH="$HOME/go"
  2. export PATH="${PATH}:${GOPATH}/bin"
  3. export KO_DOCKER_REPO='docker.io/<username>'

Checkout your fork

The Go tools require that you clone the repository to the src/github.com/kserve/kserve directory in your GOPATH.

To check out this repository:

  1. Create your own fork of this repo
  2. Clone it to your machine:
  1. mkdir -p ${GOPATH}/src/github.com/kubeflow
  2. cd ${GOPATH}/src/github.com/kubeflow
  3. git clone git@github.com:${YOUR_GITHUB_USERNAME}/kserve.git
  4. cd kserve
  5. git remote add upstream git@github.com:kserve/kserve.git
  6. git remote set-url --push upstream no_push

Adding the upstream remote sets you up nicely for regularly syncing your fork.

Once you reach this point you are ready to do a full build and deploy as described below.

Deploy KServe

Check Knative Serving installation

Once you’ve setup your development environment, you can verify the installation with following:

Success

  1. $ kubectl -n knative-serving get pods
  2. NAME                                                     READY   STATUS      RESTARTS   AGE
  3. activator-77784645fc-t2pjf                               1/1     Running     0          11d
  4. autoscaler-6fddf74d5-z2fzf                               1/1     Running     0          11d
  5. autoscaler-hpa-5bf4476cc5-tsbw6                          1/1     Running     0          11d
  6. controller-7b8cd7f95c-6jxxj                              1/1     Running     0          11d
  7. istio-webhook-866c5bc7f8-t5ztb                           1/1     Running     0          11d
  8. networking-istio-54fb8b5d4b-xznwd                        1/1     Running     0          11d
  9. webhook-5f5f7bd9b4-cv27c                                 1/1     Running     0          11d
  11. $ kubectl get gateway -n knative-serving
  12. NAME                      AGE
  13. knative-ingress-gateway   11d
  14. knative-local-gateway     11d
  16. $ kubectl get svc -n istio-system
  17. NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                      AGE
  18. istio-ingressgateway    LoadBalancer   10.101.196.89    X.X.X.X       15021:31101/TCP,80:31781/TCP,443:30372/TCP,15443:31067/TCP   11d
  19. istiod                  ClusterIP      10.101.116.203   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP                11d

Deploy KServe from master branch

We suggest using cert manager for provisioning the certificates for the webhook server. Other solutions should also work as long as they put the certificates in the desired location.

You can follow the cert manager documentation to install it.

If you don’t want to install cert manager, you can set the KSERVE_ENABLE_SELF_SIGNED_CA environment variable to true. KSERVE_ENABLE_SELF_SIGNED_CA will execute a script to create a self-signed CA and patch it to the webhook config.

  1. export KSERVE_ENABLE_SELF_SIGNED_CA=true

After that you can run following command to deploy KServe, you can skip above step if cert manager is already installed.

  1. make deploy

Optional

you can change CPU and memory limits when deploying KServe.

  1. export KSERVE_CONTROLLER_CPU_LIMIT=<cpu_limit>
  2. export KSERVE_CONTROLLER_MEMORY_LIMIT=<memory_limit>
  3. make deploy

Expected Output

  1. $ kubectl get pods -n kserve -l control-plane=kserve-controller-manager
  2. NAME                             READY   STATUS    RESTARTS   AGE
  3. kserve-controller-manager-0      2/2     Running   0          13m

Note

By default it installs to kserve namespace with the published controller manager image from master branch.

Deploy KServe with your own version

Run the following command to deploy KServe controller and model agent with your local change.

  1. make deploy-dev

Note

deploy-dev builds the image from your local code, publishes to KO_DOCKER_REPO and deploys the kserve-controller-manager and model agent with the image digest to your cluster for testing. Please also ensure you are logged in to KO_DOCKER_REPO from your client machine.

Run the following command to deploy model server with your local change.

  1. make deploy-dev-sklearn
  2. make deploy-dev-xgb

Run the following command to deploy explainer with your local change.

  1. make deploy-dev-alibi

Run the following command to deploy storage initializer with your local change.

  1. make deploy-dev-storageInitializer

Warning

The deploy command publishes the image to KO_DOCKER_REPO with the version latest, it changes the InferenceService configmap to point to the newly built image sha. The built image is only for development and testing purpose, the current limitation is that it changes the image impacted and reset all other images including the kserver-controller-manager to use the default ones.

Smoke test after deployment

Run the following command to smoke test the deployment

  1. kubectl apply -f https://github.com/kserve/kserve/tree/master/docs/samples/v1beta1/tensorflow/tensorflow.yaml

You should see model serving deployment running under default or your specified namespace.

  1. $ kubectl get pods -n default -l serving.kserve.io/inferenceservice=flowers-sample

Expected Output 

  1. NAME                                                      READY   STATUS    RESTARTS   AGE
  2. flowers-sample-default-htz8r-deployment-8fd979f9b-w2qbv   3/3     Running   0          10s

Running unit/integration tests

kserver-controller-manager has a few integration tests which requires mock apiserver and etcd, they get installed along with kubebuilder.

To run all unit/integration tests:

  1. make test

Run e2e tests locally

To setup from local code, do:

  1. ./hack/quick_install.sh
  2. make undeploy
  3. make deploy-dev

Install pytest and test deps:

  1. pip3 install pytest==6.0.2 pytest-xdist pytest-rerunfailures
  2. pip3 install --upgrade pytest-tornasync
  3. pip3 install urllib3==1.24.2
  4. pip3 install --upgrade setuptools

Go to python/kserve and install kserve python sdk deps

  1. pip3 install -r requirements.txt
  2. python3 setup.py install --force --user

Then go to test/e2e.

Run kubectl create namespace kserve-ci-e2e-test

For KIND/minikube:

  • Run export KSERVE_INGRESS_HOST_PORT=localhost:8080
  • In a different window run kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80
  • Note that not all tests will pass as the pytorch test requires gpu. These will show as pending pods at the end or you can add marker to skip the test.

Run pytest > testresults.txt

Tests may not clean up. To re-run, first do kubectl delete namespace kserve-ci-e2e-test, recreate namespace and run again.

Iterating

As you make changes to the code-base, there are two special cases to be aware of:

  • If you change an input to generated code, then you must run make manifests. Inputs include:

  • API type definitions in apis/serving/v1beta1,

  • Manifests or kustomize patches stored in config.

  • If you want to add new dependencies, then you add the imports and the specific version of the dependency module in go.mod. When it encounters an import of a package not provided by any module in go.mod, the go command automatically looks up the module containing the package and adds it to go.mod using the latest version.

  • If you want to upgrade the dependency, then you run go get command e.g go get golang.org/x/text to upgrade to the latest version, go get golang.org/x/text@v0.3.0 to upgrade to a specific version.

  1. make deploy-dev