Truncate token filter

The truncate token filter is used to shorten tokens exceeding a specified length. It trims tokens to a maximum number of characters, ensuring that tokens exceeding this limit are truncated.

Parameters

The truncate token filter can be configured with the following parameter.

ParameterRequired/OptionalData typeDescription
lengthOptionalIntegerSpecifies the maximum length of the generated token. Default is 10.

Example

The following example request creates a new index named truncate_example and configures an analyzer with a truncate filter:

  1. PUT /truncate_example
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "filter": {
  6.         "truncate_filter": {
  7.           "type": "truncate",
  8.           "length": 5
  9.         }
  10.       },
  11.       "analyzer": {
  12.         "truncate_analyzer": {
  13.           "type": "custom",
  14.           "tokenizer": "standard",
  15.           "filter": [
  16.             "lowercase",
  17.             "truncate_filter"
  18.           ]
  19.         }
  20.       }
  21.     }
  22.   }
  23. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. GET /truncate_example/_analyze
  2. {
  3.   "analyzer": "truncate_analyzer",
  4.   "text": "OpenSearch is powerful and scalable"
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "opens",
  5.       "start_offset": 0,
  6.       "end_offset": 10,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "is",
  12.       "start_offset": 11,
  13.       "end_offset": 13,
  14.       "type": "<ALPHANUM>",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "power",
  19.       "start_offset": 14,
  20.       "end_offset": 22,
  21.       "type": "<ALPHANUM>",
  22.       "position": 2
  23.     },
  24.     {
  25.       "token": "and",
  26.       "start_offset": 23,
  27.       "end_offset": 26,
  28.       "type": "<ALPHANUM>",
  29.       "position": 3
  30.     },
  31.     {
  32.       "token": "scala",
  33.       "start_offset": 27,
  34.       "end_offset": 35,
  35.       "type": "<ALPHANUM>",
  36.       "position": 4
  37.     }
  38.   ]
  39. }