Example Anchors Text Explaination for Movie Sentiment

This example uses a movie sentiment dataset to show the explanation on text data, for a more visual walkthrough please try the Jupyter notebook.

Deploy InferenceService with AnchorText Explainer

We can create a InferenceService with a trained sklearn predictor for this dataset and an associated explainer. The black box explainer algorithm we will use is the Text version of Anchors from the Alibi open source library. More details on this algorithm and configuration settings that can be set can be found in the Seldon Alibi documentation.

The InferenceService is shown below:

  1. apiVersion: "serving.kserve.io/v1beta1"
  2. kind: "InferenceService"
  3. metadata:
  4.   name: "moviesentiment"
  5. spec:
  6.   predictor:
  7.     minReplicas: 1
  8.     sklearn:
  9.       storageUri: "gs://seldon-models/sklearn/moviesentiment"
  10.       resources:
  11.         requests:
  12.           cpu: 0.1
  13.           memory: 1Gi       
  14.         limits:
  15.           cpu: 1
  16.           memory: 1Gi         
  17.   explainer:
  18.     minReplicas: 1
  19.     alibi:
  20.       type: AnchorText
  21.       resources:
  22.         requests:
  23.           cpu: 0.1
  24.           memory: 6Gi
  25.         limits:
  26.           memory: 6Gi

Create this InferenceService:

kubectl

  1. kubectl create -f moviesentiment.yaml

Run Inference and Explanation

Set up some environment variables for the model name and cluster entrypoint.

  1. MODEL_NAME=moviesentiment
  2. SERVICE_HOSTNAME=$(kubectl get inferenceservice moviesentiment -o jsonpath='{.status.url}' | cut -d "/" -f 3)

Test the predictor on an example sentence:

  1. curl -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/$MODEL_NAME:predict -d '{"instances":["a visually flashy but narratively opaque and emotionally vapid exercise ."]}'

You should receive the response showing negative sentiment:

  1. {"predictions": [0]}

Test on another sentence:

  1. curl -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/$MODEL_NAME:predict -d '{"instances":["a touching , sophisticated film that almost seems like a documentary in the way it captures an italian immigrant family on the brink of major changes ."]}'

You should receive the response showing positive sentiment:

  1. {"predictions": [1]}

Now lets get an explanation for the first sentence:

  1. curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/$MODEL_NAME:explain -d '{"instances":["a visually flashy but narratively opaque and emotionally vapid exercise ."]}'

Expected Output

  1. {
  2.   "names": [
  3.     "exercise"
  4.   ],
  5.   "precision": 1,
  6.   "coverage": 0.5005,
  7.   "raw": {
  8.     "feature": [
  9.       9
  10.     ],
  11.     "mean": [
  12.       1
  13.     ],
  14.     "precision": [
  15.       1
  16.     ],
  17.     "coverage": [
  18.       0.5005
  19.     ],
  20.     "examples": [
  21.       {
  22.         "covered": [
  23.           [
  24.             "a visually UNK UNK UNK opaque and emotionally vapid exercise UNK"
  25.           ],
  26.           [
  27.             "a visually flashy but UNK UNK and emotionally UNK exercise ."
  28.           ],
  29.           [
  30.             "a visually flashy but narratively UNK UNK UNK UNK exercise ."
  31.           ],
  32.           [
  33.             "UNK UNK flashy UNK narratively opaque UNK UNK vapid exercise ."
  34.           ],
  35.           [
  36.             "UNK visually UNK UNK UNK UNK and UNK vapid exercise UNK"
  37.           ],
  38.           [
  39.             "UNK UNK UNK but UNK opaque UNK emotionally UNK exercise UNK"
  40.           ],
  41.           [
  42.             "a UNK flashy UNK UNK UNK and emotionally vapid exercise ."
  43.           ],
  44.           [
  45.             "UNK UNK flashy UNK narratively opaque UNK emotionally UNK exercise ."
  46.           ],
  47.           [
  48.             "UNK UNK flashy UNK narratively opaque UNK UNK vapid exercise UNK"
  49.           ],
  50.           [
  51.             "a visually UNK but narratively opaque UNK UNK vapid exercise UNK"
  52.           ]
  53.         ],
  54.         "covered_true": [
  55.           [
  56.             "UNK visually flashy but UNK UNK and emotionally vapid exercise ."
  57.           ],
  58.           [
  59.             "UNK visually UNK UNK UNK UNK and UNK UNK exercise ."
  60.           ],
  61.           [
  62.             "a UNK UNK UNK narratively opaque UNK UNK UNK exercise UNK"
  63.           ],
  64.           [
  65.             "a visually UNK UNK narratively opaque UNK UNK UNK exercise UNK"
  66.           ],
  67.           [
  68.             "a UNK UNK UNK UNK UNK and emotionally vapid exercise UNK"
  69.           ],
  70.           [
  71.             "a UNK flashy UNK narratively UNK and UNK vapid exercise UNK"
  72.           ],
  73.           [
  74.             "UNK visually UNK UNK narratively UNK and emotionally UNK exercise ."
  75.           ],
  76.           [
  77.             "UNK visually flashy UNK narratively opaque UNK emotionally UNK exercise UNK"
  78.           ],
  79.           [
  80.             "UNK UNK flashy UNK UNK UNK and UNK vapid exercise UNK"
  81.           ],
  82.           [
  83.             "a UNK flashy UNK UNK UNK and emotionally vapid exercise ."
  84.           ]
  85.         ],
  86.         "covered_false": [],
  87.         "uncovered_true": [],
  88.         "uncovered_false": []
  89.       }
  90.     ],
  91.     "all_precision": 0,
  92.     "num_preds": 1000101,
  93.     "names": [
  94.       "exercise"
  95.     ],
  96.     "positions": [
  97.       63
  98.     ],
  99.     "instance": "a visually flashy but narratively opaque and emotionally vapid exercise .",
  100.     "prediction": 0
  101.   }
  102. }

This shows the key word “bad” was indetified and examples show it in context using the default “UKN” placeholder for surrounding words.

Custom Configuration

You can add custom configuration for the Anchor Text explainer in the ‘config’ section. For example we can change the text explainer to sample from the corpus rather than use UKN placeholders:

  1. apiVersion: "serving.kserve.io/v1beta1"
  2. kind: "InferenceService"
  3. metadata:
  4.   name: "moviesentiment"
  5. spec:
  6.   predictor:
  7.     sklearn:
  8.       storageUri: "gs://seldon-models/sklearn/moviesentiment"
  9.       resources:
  10.         requests:
  11.           cpu: 0.1
  12.   explainer:
  13.     alibi:
  14.       type: AnchorText
  15.       config:
  16.         use_unk: "false"
  17.         sample_proba: "0.5"   
  18.       resources:
  19.         requests:
  20.           cpu: 0.1

If we apply this:

kubectl

  1. kubectl create -f moviesentiment2.yaml

and then ask for an explanation:

  1. curl -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/$MODEL_NAME:explain -d '{"instances":["a visually flashy but narratively opaque and emotionally vapid exercise ."]}'

Expected Output

  1. {
  2.   "names": [
  3.     "exercise"
  4.   ],
  5.   "precision": 0.9918032786885246,
  6.   "coverage": 0.5072,
  7.   "raw": {
  8.     "feature": [
  9.       9
  10.     ],
  11.     "mean": [
  12.       0.9918032786885246
  13.     ],
  14.     "precision": [
  15.       0.9918032786885246
  16.     ],
  17.     "coverage": [
  18.       0.5072
  19.     ],
  20.     "examples": [
  21.       {
  22.         "covered": [
  23.           [
  24.             "each visually playful but enormously opaque and academically vapid exercise ."
  25.           ],
  26.           [
  27.             "each academically trashy but narratively pigmented and profoundly vapid exercise ."
  28.           ],
  29.           [
  30.             "a masterfully flashy but narratively straightforward and verbally disingenuous exercise ."
  31.           ],
  32.           [
  33.             "a visually gaudy but interestingly opaque and emotionally vapid exercise ."
  34.           ],
  35.           [
  36.             "some concurrently flashy but philosophically voxel and emotionally vapid exercise ."
  37.           ],
  38.           [
  39.             "a visually flashy but delightfully sensible and emotionally snobby exercise ."
  40.           ],
  41.           [
  42.             "a surprisingly bland but fantastically seamless and hideously vapid exercise ."
  43.           ],
  44.           [
  45.             "both visually classy but nonetheless robust and musically vapid exercise ."
  46.           ],
  47.           [
  48.             "a visually fancy but narratively robust and emotionally uninformed exercise ."
  49.           ],
  50.           [
  51.             "a visually flashy but tastefully opaque and weirdly vapid exercise ."
  52.           ]
  53.         ],
  54.         "covered_true": [
  55.           [
  56.             "another visually flashy but narratively opaque and emotionally vapid exercise ."
  57.           ],
  58.           [
  59.             "the visually classy but narratively opaque and emotionally vapid exercise ."
  60.           ],
  61.           [
  62.             "the visually arty but overshadow yellowish and emotionally vapid exercise ."
  63.           ],
  64.           [
  65.             "a objectively flashy but genuinely straightforward and emotionally vapid exercise ."
  66.           ],
  67.           [
  68.             "a visually flashy but tastefully opaque and weirdly vapid exercise ."
  69.           ],
  70.           [
  71.             "a emotionally crafty but narratively opaque and emotionally vapid exercise ."
  72.           ],
  73.           [
  74.             "some similarly eclectic but narratively dainty and emotionally illogical exercise ."
  75.           ],
  76.           [
  77.             "a nicely flashy but psychologically opaque and emotionally vapid exercise ."
  78.           ],
  79.           [
  80.             "a visually flashy but narratively colorless and emotionally vapid exercise ."
  81.           ],
  82.           [
  83.             "every properly lavish but logistically opaque and someway incomprehensible exercise ."
  84.           ]
  85.         ],
  86.         "covered_false": [
  87.           [
  88.             "another enormously inventive but socially opaque and somewhat idiotic exercise ."
  89.           ],
  90.           [
  91.             "each visually playful but enormously opaque and academically vapid exercise ."
  92.           ]
  93.         ],
  94.         "uncovered_true": [],
  95.         "uncovered_false": []
  96.       }
  97.     ],
  98.     "all_precision": 0,
  99.     "num_preds": 1000101,
  100.     "names": [
  101.       "exercise"
  102.     ],
  103.     "positions": [
  104.       63
  105.     ],
  106.     "instance": "a visually flashy but narratively opaque and emotionally vapid exercise .",
  107.     "prediction": 0
  108.   }
  109. }

Run on Notebook

You can also run this example on notebook