Remove duplicates token filter

The remove_duplicates token filter is used to remove duplicate tokens that are generated in the same position during analysis.

Example

The following example request creates an index with a keyword_repeat token filter. The filter adds a keyword version of each token in the same position as the token itself and then uses a kstem to create a stemmed version of the token:

  1. PUT /example-index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "custom_analyzer": {
  7.           "type": "custom",
  8.           "tokenizer": "standard",
  9.           "filter": [
  10.             "lowercase",
  11.             "keyword_repeat",
  12.             "kstem"
  13.           ]
  14.         }
  15.       }
  16.     }
  17.   }
  18. }

copy

Use the following request to analyze the string Slower turtle:

  1. GET /example-index/_analyze
  2. {
  3.   "analyzer": "custom_analyzer",
  4.   "text": "Slower turtle"
  5. }

copy

The response contains the token turtle twice in the same position:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "slower",
  5.       "start_offset": 0,
  6.       "end_offset": 6,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "slow",
  12.       "start_offset": 0,
  13.       "end_offset": 6,
  14.       "type": "<ALPHANUM>",
  15.       "position": 0
  16.     },
  17.     {
  18.       "token": "turtle",
  19.       "start_offset": 7,
  20.       "end_offset": 13,
  21.       "type": "<ALPHANUM>",
  22.       "position": 1
  23.     },
  24.     {
  25.       "token": "turtle",
  26.       "start_offset": 7,
  27.       "end_offset": 13,
  28.       "type": "<ALPHANUM>",
  29.       "position": 1
  30.     }
  31.   ]
  32. }

The duplicate token can be removed by adding a remove_duplicates token filter to the index settings:

  1. PUT /index-remove-duplicate
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "custom_analyzer": {
  7.           "type": "custom",
  8.           "tokenizer": "standard",
  9.           "filter": [
  10.             "lowercase",
  11.             "keyword_repeat",
  12.             "kstem",
  13.             "remove_duplicates"
  14.           ]
  15.         }
  16.       }
  17.     }
  18.   }
  19. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. GET /index-remove-duplicate/_analyze
  2. {
  3.   "analyzer": "custom_analyzer",
  4.   "text": "Slower turtle"
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "slower",
  5.       "start_offset": 0,
  6.       "end_offset": 6,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "slow",
  12.       "start_offset": 0,
  13.       "end_offset": 6,
  14.       "type": "<ALPHANUM>",
  15.       "position": 0
  16.     },
  17.     {
  18.       "token": "turtle",
  19.       "start_offset": 7,
  20.       "end_offset": 13,
  21.       "type": "<ALPHANUM>",
  22.       "position": 1
  23.     }
  24.   ]
  25. }