Stemmer override token filter

The stemmer_override token filter allows you to define custom stemming rules that override the behavior of default stemmers like Porter or Snowball. This can be useful when you want to apply specific stemming behavior to certain words that might not be modified correctly by the standard stemming algorithms.

Parameters

The stemmer_override token filter must be configured with exactly one of the following parameters.

ParameterData typeDescription
rulesStringDefines the override rules directly in the settings.
rules_pathStringSpecifies the path to the file containing custom rules (mappings). The path can be either an absolute path or a path relative to the config directory.

Example

The following example request creates a new index named my-index and configures an analyzer with a stemmer_override filter:

  1. PUT /my-index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "filter": {
  6.         "my_stemmer_override_filter": {
  7.           "type": "stemmer_override",
  8.           "rules": [
  9.             "running, runner => run",
  10.             "bought => buy",
  11.             "best => good"
  12.           ]
  13.         }
  14.       },
  15.       "analyzer": {
  16.         "my_custom_analyzer": {
  17.           "type": "custom",
  18.           "tokenizer": "standard",
  19.           "filter": [
  20.             "lowercase",
  21.             "my_stemmer_override_filter"
  22.           ]
  23.         }
  24.       }
  25.     }
  26.   }
  27. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. GET /my-index/_analyze
  2. {
  3.   "analyzer": "my_custom_analyzer",
  4.   "text": "I am a runner and bought the best shoes"
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "i",
  5.       "start_offset": 0,
  6.       "end_offset": 1,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "am",
  12.       "start_offset": 2,
  13.       "end_offset": 4,
  14.       "type": "<ALPHANUM>",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "a",
  19.       "start_offset": 5,
  20.       "end_offset": 6,
  21.       "type": "<ALPHANUM>",
  22.       "position": 2
  23.     },
  24.     {
  25.       "token": "run",
  26.       "start_offset": 7,
  27.       "end_offset": 13,
  28.       "type": "<ALPHANUM>",
  29.       "position": 3
  30.     },
  31.     {
  32.       "token": "and",
  33.       "start_offset": 14,
  34.       "end_offset": 17,
  35.       "type": "<ALPHANUM>",
  36.       "position": 4
  37.     },
  38.     {
  39.       "token": "buy",
  40.       "start_offset": 18,
  41.       "end_offset": 24,
  42.       "type": "<ALPHANUM>",
  43.       "position": 5
  44.     },
  45.     {
  46.       "token": "the",
  47.       "start_offset": 25,
  48.       "end_offset": 28,
  49.       "type": "<ALPHANUM>",
  50.       "position": 6
  51.     },
  52.     {
  53.       "token": "good",
  54.       "start_offset": 29,
  55.       "end_offset": 33,
  56.       "type": "<ALPHANUM>",
  57.       "position": 7
  58.     },
  59.     {
  60.       "token": "shoes",
  61.       "start_offset": 34,
  62.       "end_offset": 39,
  63.       "type": "<ALPHANUM>",
  64.       "position": 8
  65.     }
  66.   ]
  67. }