PyTorch Serving

Instructions for serving a PyTorch model with Seldon

This guide walks you through serving a PyTorch trained model in Kubeflow.

Serving a model

We use seldon-core component deployed following these instructions to serve the model.

See also this Example module which contains the code to wrap the model with Seldon.

We will wrap this class into a seldon-core microservice which we can then deploy as a REST or GRPC API server.

Building a model server

We use the public model server image gcr.io/kubeflow-examples/mnistddpserving as an example

  • This server loads the model from the mount point /mnt/kubeflow-gcfs and includes the supporting assets baked into the container image
  • So you can just run this image to get a pre-trained model from the shared persistent disk
  • Serving your own model using this server, exposing predict service as GRPC API

Building your own model server

You can use the below command to build your own image to wrap your model, also check thisscript example that calls the docker Seldon wrapper to build our server image, exposing the predict service as GRPC API.

  1. docker run -v $(pwd):/my_model seldonio/core-python-wrapper:0.7 /my_model mnistddpserving 0.1 gcr.io --image-name=kubeflow-examples/mnistddpserving --grpc

You can then push the image by running gcloud docker — push gcr.io/kubeflow-examples/mnistddpserving:0.1.

You can find more details about wrapping a model with seldon-core here

Deploying the model to your Kubeflow cluster

This section has not yet been converted to kustomize, please refer to kubeflow/manifests/issues/10.

We need to have seldon component deployed, you can deploy the model once trained using a pre-defined ksonnet component, similar to this example.

Create an environment variable, ${KF_ENV}, to represent a conceptualdeployment environment such as development, test, staging, or production, asdefined by ksonnet. For this example, we use the default environment. You canread more about Kubeflow’s use of ksonnet in the Kubeflowksonnet component guide.

Then modify the Ksonnet componentparameters to use your specific image.

  1. export KF_ENV=default
  2. cd ks_app
  3. ks env add ${KF_ENV}
  4. ks apply ${KF_ENV} -c serving_model

Testing model server

Seldon Core component uses ambassador to route it’s requests to our model server. To send requests to the model, you can port-forward the ambassador container locally:

  1. kubectl port-forward $(kubectl get pods -n ${NAMESPACE} -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n ${NAMESPACE} 8080:80

And send a request, for our example we know is not a torch MNIST image, so it will return an error 500

  1. curl -X POST -H 'Content-Type: application/json' -d '{"data":{"int":"8"}}' http://localhost:8080/seldon/mnist-classifier/api/v0.1/predictions

We should receive an error response as the model server is expecting a 1x786 vector representing a torch image, this will be sufficient to confirm the server model is up and running(This is to avoid having to send manually a vector of 786 pixels, you can interact properly with the model using a web interface if you follow all theinstructions in the example)

  1. {
  2. "timestamp":1540899355053,
  3. "status":500,"error":"Internal Server Error",
  4. "exception":"io.grpc.StatusRuntimeException",
  5. "message":"UNKNOWN: Exception calling application: tensor is not a torch image.",
  6. "path":"/api/v0.1/predictions"
  7. }