Common grams token filter

The common_grams token filter improves search relevance by keeping commonly occurring phrases (common grams) in the text. This is useful when dealing with languages or datasets in which certain word combinations frequently occur as a unit and can impact search relevance if treated as separate tokens. If any common words are present in the input string, this token filter generates both their unigrams and bigrams.

Using this token filter improves search relevance by keeping common phrases intact. This can help in matching queries more accurately, particularly for frequent word combinations. It also improves search precision by reducing the number of irrelevant matches.

When using this filter, you must carefully select and maintain the common_words list.

Parameters

The common_grams token filter can be configured with the following parameters.

ParameterRequired/OptionalData typeDescription
common_wordsRequiredList of stringsA list of words that should be treated as words that commonly appear together. These words will be used to generate common grams. If the common_words parameter is an empty list, the common_grams token filter becomes a no-op filter, meaning that it doesn’t modify the input tokens at all.
ignore_caseOptionalBooleanIndicates whether the filter should ignore case differences when matching common words. Default is false.
query_modeOptionalBooleanWhen set to true, the following rules are applied:
- Unigrams that are generated from common_words are not included in the output.
- Bigrams in which a non-common word is followed by a common word are retained in the output.
- Unigrams of non-common words are excluded if they are immediately followed by a common word.
- If a non-common word appears at the end of the text and is preceded by a common word, its unigram is not included in the output.

Example

The following example request creates a new index named my_common_grams_index and configures an analyzer with the common_grams filter:

  1. PUT /my_common_grams_index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "filter": {
  6.         "my_common_grams_filter": {
  7.           "type": "common_grams",
  8.           "common_words": ["a", "in", "for"],
  9.           "ignore_case": true,
  10.           "query_mode": true
  11.         }
  12.       },
  13.       "analyzer": {
  14.         "my_analyzer": {
  15.           "type": "custom",
  16.           "tokenizer": "standard",
  17.           "filter": [
  18.             "lowercase",
  19.             "my_common_grams_filter"
  20.           ]
  21.         }
  22.       }
  23.     }
  24.   }
  25. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. GET /my_common_grams_index/_analyze
  2. {
  3.   "analyzer": "my_analyzer",
  4.   "text": "A quick black cat jumps over the lazy dog in the park"
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {"token": "a_quick","start_offset": 0,"end_offset": 7,"type": "gram","position": 0},
  4.     {"token": "quick","start_offset": 2,"end_offset": 7,"type": "<ALPHANUM>","position": 1},
  5.     {"token": "black","start_offset": 8,"end_offset": 13,"type": "<ALPHANUM>","position": 2},
  6.     {"token": "cat","start_offset": 14,"end_offset": 17,"type": "<ALPHANUM>","position": 3},
  7.     {"token": "jumps","start_offset": 18,"end_offset": 23,"type": "<ALPHANUM>","position": 4},
  8.     {"token": "over","start_offset": 24,"end_offset": 28,"type": "<ALPHANUM>","position": 5},
  9.     {"token": "the","start_offset": 29,"end_offset": 32,"type": "<ALPHANUM>","position": 6},
  10.     {"token": "lazy","start_offset": 33,"end_offset": 37,"type": "<ALPHANUM>","position": 7},
  11.     {"token": "dog_in","start_offset": 38,"end_offset": 44,"type": "gram","position": 8},
  12.     {"token": "in_the","start_offset": 42,"end_offset": 48,"type": "gram","position": 9},
  13.     {"token": "the","start_offset": 45,"end_offset": 48,"type": "<ALPHANUM>","position": 10},
  14.     {"token": "park","start_offset": 49,"end_offset": 53,"type": "<ALPHANUM>","position": 11}
  15.   ]
  16. }