Deploying XGBoost models with InferenceService

This example walks you through how to deploy a xgboost model leveraging the v1beta1 version of the InferenceService CRD. Note that, by default the v1beta1 version will expose your model through an API compatible with the existing V1 Dataplane. However, this example will show you how to serve a model through an API compatible with the new V2 Dataplane.

Training

The first step will be to train a sample xgboost model. We will save this model as model.bst.

  1. import xgboost as xgb
  2. from sklearn.datasets import load_iris
  3. import os
  5. model_dir = "."
  6. BST_FILE = "model.bst"
  8. iris = load_iris()
  9. y = iris['target']
  10. X = iris['data']
  11. dtrain = xgb.DMatrix(X, label=y)
  12. param = {'max_depth': 6,
  13.             'eta': 0.1,
  14.             'silent': 1,
  15.             'nthread': 4,
  16.             'num_class': 10,
  17.             'objective': 'multi:softmax'
  18.             }
  19. xgb_model = xgb.train(params=param, dtrain=dtrain)
  20. model_file = os.path.join((model_dir), BST_FILE)
  21. xgb_model.save_model(model_file)

Testing locally

Once we’ve got our model.bst model serialised, we can then use MLServer to spin up a local server. For more details on MLServer, feel free to check the XGBoost example in their docs.

Note that this step is optional and just meant for testing. Feel free to jump straight to deploying your trained model.

Pre-requisites

Firstly, to use MLServer locally, you will first need to install the mlserver package in your local environment as well as the XGBoost runtime.

  1. pip install mlserver mlserver-xgboost

Model settings

The next step will be providing some model settings so that MLServer knows:

  • The inference runtime that we want our model to use (i.e. mlserver_xgboost.XGBoostModel)
  • Our model’s name and version

These can be specified through environment variables or by creating a local model-settings.json file:

  1. {
  2.   "name": "xgboost-iris",
  3.   "version": "v1.0.0",
  4.   "implementation": "mlserver_xgboost.XGBoostModel"
  5. }

Note that, when we deploy our model, KServe will already inject some sensible defaults so that it runs out-of-the-box without any further configuration. However, you can still override these defaults by providing a model-settings.json file similar to your local one. You can even provide a set of model-settings.json files to load multiple models.

Serving our model locally

With the mlserver package installed locally and a local model-settings.json file, we should now be ready to start our server as:

  1. mlserver start .

Deploy with InferenceService

Lastly, we will use KServe to deploy our trained model. For this, we will just need to use version v1beta1 of the InferenceService CRD and set the the protocolVersion field to v2.

  1. apiVersion: "serving.kserve.io/v1beta1"
  2. kind: "InferenceService"
  3. metadata:
  4.   name: "xgboost-iris"
  5. spec:
  6.   predictor:
  7.     xgboost:
  8.       protocolVersion: "v2"
  9.       storageUri: "gs://kfserving-samples/models/xgboost/iris"

Note that this makes the following assumptions:

  • Your model weights (i.e. your model.bst file) have already been uploaded to a “model repository” (GCS in this example) and can be accessed as gs://kfserving-samples/models/xgboost/iris.
  • There is a K8s cluster available, accessible through kubectl.
  • KServe has already been installed in your cluster.

Assuming that we’ve got a cluster accessible through kubectl with KServe already installed, we can deploy our model as:

  1. kubectl apply -f xgboost.yaml

Testing deployed model

We can now test our deployed model by sending a sample request.

Note that this request needs to follow the V2 Dataplane protocol. You can see an example payload below:

  1. {
  2.   "inputs": [
  3.     {
  4.       "name": "input-0",
  5.       "shape": [2, 4],
  6.       "datatype": "FP32",
  7.       "data": [
  8.         [6.8, 2.8, 4.8, 1.4],
  9.         [6.0, 3.4, 4.5, 1.6]
  10.       ]
  11.     }
  12.   ]
  13. }

Now, assuming that our ingress can be accessed at ${INGRESS_HOST}:${INGRESS_PORT}, we can use curl to send our inference request as:

You can follow these instructions to find out your ingress IP and port.

  1. SERVICE_HOSTNAME=$(kubectl get inferenceservice xgboost-iris -o jsonpath='{.status.url}' | cut -d "/" -f 3)
  3. curl -v \
  4.   -H "Host: ${SERVICE_HOSTNAME}" \
  5.   -d @./iris-input.json \
  6.   http://${INGRESS_HOST}:${INGRESS_PORT}/v2/models/xgboost-iris/infer

The output will be something similar to:

  1. {
  2.   "id": "4e546709-0887-490a-abd6-00cbc4c26cf4",
  3.   "model_name": "xgboost-iris",
  4.   "model_version": "v1.0.0",
  5.   "outputs": [
  6.     {
  7.       "data": [1.0, 1.0],
  8.       "datatype": "FP32",
  9.       "name": "predict",
  10.       "parameters": null,
  11.       "shape": [2]
  12.     }
  13.   ]
  14. }