Language analyzers

OpenSearch supports the following language analyzers: arabic, armenian, basque, bengali, brazilian, bulgarian, catalan, czech, danish, dutch, english, estonian, finnish, french, galician, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, thai, and turkish.

To use an analyzer when you map an index, specify the value in your query. For example, to map your index with the French language analyzer, specify the french value in the analyzer field:

  1.  "analyzer": "french"

Example request

The following query specifies an index my-index with the content field configured as multi-field, and a sub-field named french is configured with the french language analyzer:

  1. PUT my-index
  2. {
  3.   "mappings": {
  4.     "properties": {
  5.       "content": { 
  6.         "type": "text",
  7.         "fields": {
  8.           "french": { 
  9.             "type": "text",
  10.             "analyzer": "french"
  11.           }
  12.         }
  13.       }
  14.     }
  15.   }
  16. }

copy

The default french analyzer can also be configured for the entire index using the following query:

  1. PUT my-index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "default": {
  7.           "type": "french"
  8.         }
  9.       }
  10.     }
  11.   },
  12.   "mappings": {
  13.     "properties": {
  14.       "content": {
  15.         "type": "text"
  16.       },
  17.       "title": {
  18.         "type": "text"
  19.       },
  20.       "description": {
  21.         "type": "text"
  22.       }
  23.     }
  24.   }
  25. }

copy

Stem exclusion

You can apply stem exclusion to any language analyzer by providing a list of lowercase words that should be excluded from stemming. Internally, OpenSearch uses the keyword_marker token filter to mark these words as keywords, ensuring that they are not stemmed.

Stem exclusion example

Use the following request to configure stem_exclusion:

  1. PUT index_with_stem_exclusion_english_analyzer
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "stem_exclusion_english_analyzer":{
  7.           "type":"english",
  8.           "stem_exclusion": ["manager", "management"]
  9.         }
  10.       }
  11.     }
  12.   }
  13. }

copy

Stem exclusion with custom analyzers

All language analyzers consist of tokenizers and token filters specific to a particular language. If you want to implement a custom version of the language analyzer with stem exclusion, you need to configure the keyword_marker token filter and list the words excluded from stemming in the keywords parameter:

  1. PUT index_with_keyword_marker_analyzer
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "filter": {
  6.         "protected_keywords_filter": {
  7.           "type": "keyword_marker",
  8.           "keywords": ["Apple", "OpenSearch"]
  9.         }
  10.       },
  11.       "analyzer": {
  12.         "custom_english_analyzer": {
  13.           "type": "custom",
  14.           "tokenizer": "standard",
  15.           "filter": [
  16.             "lowercase",
  17.             "protected_keywords_filter",
  18.             "english_stemmer"
  19.           ]
  20.         }
  21.       }
  22.     }
  23.   }
  24. }

copy