Inference Logger

Basic Inference Logger

Create Message Dumper

Create a message dumper Knative Service which will print out the CloudEvents it receives.

yaml

  1. apiVersion: serving.knative.dev/v1
  2. kind: Service
  3. metadata:
  4.   name: message-dumper
  5. spec:
  6.   template:
  7.     spec:
  8.       containers:
  9.       - image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display

kubectl

  1. kubectl create -f message-dumper.yaml

Create an InferenceService with Logger

Create a sklearn predictor with the logger which points at the message dumper url.

yaml

  1. apiVersion: serving.kserve.io/v1beta1
  2. kind: InferenceService
  3. metadata:
  4.   name: sklearn-iris
  5. spec:
  6.   predictor:
  7.     logger:
  8.       mode: all
  9.       url: http://message-dumper.default/
  10.     sklearn:
  11.       storageUri: gs://kfserving-samples/models/sklearn/iris

Note

Here we set the url explicitly, otherwise it defaults to the namespace knative broker or the value of DefaultUrl in the logger section of the inference service configmap.

kubectl

  1. kubectl create -f sklearn-basic-logger.yaml

We can now send a request to the sklearn model. The first step is to determine the ingress IP and ports and set INGRESS_HOST and INGRESS_PORT

  1. MODEL_NAME=sklearn-iris
  2. INPUT_PATH=@./input.json
  3. SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -o jsonpath='{.status.url}' | cut -d "/" -f 3)
  5. curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/$MODEL_NAME:predict -d $INPUT_PATH

Expected Output

  1. {
  2.     "predictions": [1, 1]
  3. }

Check CloudEvents

Check the logs of the message dumper, we can see the CloudEvents associated with our previous curl request.

kubectl

  1. kubectl logs $(kubectl get pod -l serving.knative.dev/service=message-dumper -o jsonpath='{.items[0].metadata.name}') user-container

Expected Output

  1. ☁️  cloudevents.Event
  2. Validation: valid
  3. Context Attributes,
  4.   specversion: 1.0
  5.   type: org.kubeflow.serving.inference.request
  6.   source: http://localhost:9081/
  7.   id: 0009174a-24a8-4603-b098-09c8799950e9
  8.   time: 2021-04-10T00:23:26.0789529Z
  9.   datacontenttype: application/json
  10. Extensions,
  11.   endpoint:
  12.   inferenceservicename: sklearn-iris
  13.   namespace: default
  14.   traceparent: 00-90bdf848647d50283394155d2df58f19-84dacdfdf07cadfc-00
  15. Data,
  16.   {
  17.     "instances": [
  18.       [
  19.         6.8,
  20.         2.8,
  21.         4.8,
  22.         1.4
  23.       ],
  24.       [
  25.         6.0,
  26.         3.4,
  27.         4.5,
  28.         1.6
  29.       ]
  30.     ]
  31.   }
  32. ☁️  cloudevents.Event
  33. Validation: valid
  34. Context Attributes,
  35.   specversion: 1.0
  36.   type: org.kubeflow.serving.inference.response
  37.   source: http://localhost:9081/
  38.   id: 0009174a-24a8-4603-b098-09c8799950e9
  39.   time: 2021-04-10T00:23:26.080736102Z
  40.   datacontenttype: application/json
  41. Extensions,
  42.   endpoint:
  43.   inferenceservicename: sklearn-iris
  44.   namespace: default
  45.   traceparent: 00-55de1514e1d23ee17eb50dda6167bb8c-b6c6e0f6dd8f741d-00
  46. Data,
  47.   {
  48.     "predictions": [
  49.       1,
  50.       1
  51.     ]
  52.   }

Knative Eventing Inference Logger

A cluster running with Knative Eventing installed, along with KServe.

Note

This was tested using Knative Eventing v0.17.

Create Message Dumper

Create a message dumper Knative service which will print out the CloudEvents it receives.

yaml

  1. apiVersion: serving.knative.dev/v1
  2. kind: Service
  3. metadata:
  4.   name: message-dumper
  5. spec:
  6.   template:
  7.     spec:
  8.       containers:
  9.       - image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display

kubectl

  1. kubectl apply -f message-dumper.yaml

Create Channel Broker

Create a Broker which allows you route events to consumers like InferenceService.

yaml

  1. apiVersion: eventing.knative.dev/v1
  2. kind: broker
  3. metadata:
  4.   name: default

kubectl

  1. kubectl apply -f broker.yaml
  3. kubectl get broker default

Take note of the broker URL as that is what we’ll be using in the InferenceService later on.

Create Trigger

We now create a trigger to forward the events to message-dumper service. The trigger can specify a filter that enables selection of relevant events based on the Cloud Event context attributes.

yaml

  1. apiVersion: eventing.knative.dev/v1
  2. kind: Trigger
  3. metadata:
  4.   name: message-dumper-trigger
  5. spec:
  6.   broker: default
  7.   subscriber:
  8.     ref:
  9.       apiVersion: serving.knative.dev/v1
  10.       kind: Service
  11.       name: message-dumper

kubectl

  1. kubectl create -f trigger.yaml

Create an InferenceService with Logger

Create a sklearn predictor with the logger url pointing to the Knative eventing multi-tenant broker in knative-eventing namespace.

yaml

  1. apiVersion: serving.kserve.io/v1beta1
  2. kind: InferenceService
  3. metadata:
  4.   name: sklearn-iris
  5. spec:
  6.   predictor:
  7.     minReplicas: 1
  8.     logger:
  9.       mode: all
  10.       url: http://broker-ingress.knative-eventing.svc.cluster.local/default/default
  11.     sklearn:
  12.       storageUri: gs://kfserving-samples/models/sklearn/iris

kubectl

  1. kubectl create -f sklearn-knative-eventing.yaml

We can now send a request to the sklearn model. The first step is to determine the ingress IP and ports and set INGRESS_HOST and INGRESS_PORT

  1. MODEL_NAME=sklearn-iris
  2. INPUT_PATH=@./input.json
  3. SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -o jsonpath='{.status.url}' | cut -d "/" -f 3)
  5. curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/$MODEL_NAME:predict -d $INPUT_PATH

Expected Output

  1. {
  2.     "predictions": [1, 1]
  3. }

Check CloudEvents

Check the logs of the message dumper, we can see the CloudEvents associated with our previous curl request.

kubectl

  1. kubectl logs $(kubectl get pod -l serving.knative.dev/service=message-dumper -o jsonpath='{.items[0].metadata.name}') user-container

Expected Output

  1. ☁️  cloudevents.Event
  2. Validation: valid
  3. Context Attributes,
  4.   specversion: 1.0
  5.   type: org.kubeflow.serving.inference.request
  6.   source: http://localhost:9081/
  7.   id: defb5816-35f7-4947-a2b1-b9e5d7764ad2
  8.   time: 2021-04-10T01:22:16.498917288Z
  9.   datacontenttype: application/json
  10. Extensions,
  11.   endpoint:
  12.   inferenceservicename: sklearn-iris
  13.   knativearrivaltime: 2021-04-10T01:22:16.500656431Z
  14.   knativehistory: default-kne-trigger-kn-channel.default.svc.cluster.local
  15.   namespace: default
  16.   traceparent: 00-16456300519c5227ffe5f784a88da2f7-2db26af1daae870c-00
  17. Data,
  18.   {
  19.     "instances": [
  20.       [
  21.         6.8,
  22.         2.8,
  23.         4.8,
  24.         1.4
  25.       ],
  26.       [
  27.         6.0,
  28.         3.4,
  29.         4.5,
  30.         1.6
  31.       ]
  32.     ]
  33.   }
  34. ☁️  cloudevents.Event
  35. Validation: valid
  36. Context Attributes,
  37.   specversion: 1.0
  38.   type: org.kubeflow.serving.inference.response
  39.   source: http://localhost:9081/
  40.   id: defb5816-35f7-4947-a2b1-b9e5d7764ad2
  41.   time: 2021-04-10T01:22:16.500492939Z
  42.   datacontenttype: application/json
  43. Extensions,
  44.   endpoint:
  45.   inferenceservicename: sklearn-iris
  46.   knativearrivaltime: 2021-04-10T01:22:16.501931207Z
  47.   knativehistory: default-kne-trigger-kn-channel.default.svc.cluster.local
  48.   namespace: default
  49.   traceparent: 00-2156a24451a4d4ea575fcf6c4f52a672-2b6ea035c83d3200-00
  50. Data,
  51.   {
  52.     "predictions": [
  53.       1,
  54.       1
  55.     ]
  56.   }