Predicate script token filter

Predicate script token filter

Removes tokens that don’t match a provided predicate script. The filter supports inline Painless scripts only. Scripts are evaluated in the analysis predicate context.

Example

The following analyze API request uses the predicate_token_filter filter to only output tokens longer than three characters from the fox jumps the lazy dog.

  1. GET /_analyze
  2. {
  3.   "tokenizer": "whitespace",
  4.   "filter": [
  5.     {
  6.       "type": "predicate_token_filter",
  7.       "script": {
  8.         "source": """
  9.           token.term.length() > 3
  10.         """
  11.       }
  12.     }
  13.   ],
  14.   "text": "the fox jumps the lazy dog"
  15. }

The filter produces the following tokens.

  1. [ jumps, lazy ]

The API response contains the position and offsets of each output token. Note the predicate_token_filter filter does not change the tokens’ original positions or offsets.

Response

  1. {
  2.   "tokens" : [
  3.     {
  4.       "token" : "jumps",
  5.       "start_offset" : 8,
  6.       "end_offset" : 13,
  7.       "type" : "word",
  8.       "position" : 2
  9.     },
  10.     {
  11.       "token" : "lazy",
  12.       "start_offset" : 18,
  13.       "end_offset" : 22,
  14.       "type" : "word",
  15.       "position" : 4
  16.     }
  17.   ]
  18. }

Configurable parameters

script

(Required, script object) Script containing a condition used to filter incoming tokens. Only tokens that match this script are included in the output.

This parameter supports inline Painless scripts only. The script is evaluated in the analysis predicate context.

Customize and add to an analyzer

To customize the predicate_token_filter filter, duplicate it to create the basis for a new custom token filter. You can modify the filter using its configurable parameters.

The following create index API request configures a new custom analyzer using a custom predicate_token_filter filter, my_script_filter.

The my_script_filter filter removes tokens with of any type other than ALPHANUM.

  1. PUT /my-index-000001
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "my_analyzer": {
  7.           "tokenizer": "standard",
  8.           "filter": [
  9.             "my_script_filter"
  10.           ]
  11.         }
  12.       },
  13.       "filter": {
  14.         "my_script_filter": {
  15.           "type": "predicate_token_filter",
  16.           "script": {
  17.             "source": """
  18.               token.type.contains("ALPHANUM")
  19.             """
  20.           }
  21.         }
  22.       }
  23.     }
  24.   }
  25. }