Analyzer

The analyzer mapping parameter is used to define the text analysis process that applies to a text field during both index and search operations.

The key functions of the analyzer mapping parameter are:

  1. Tokenization: The analyzer determines how the text is broken down into individual tokens (words, numbers) that can be indexed and searched. Each generated token must not exceed 32,766 bytes in order to avoid indexing failures.

  2. Normalization: The analyzer can apply various normalization techniques, such as converting text to lowercase, removing stop words, and stemming/lemmatizing words.

  3. Consistency: By defining the same analyzer for both index and search operations, you ensure that the text analysis process is consistent, which helps improve the relevance of search results.

  4. Customization: OpenSearch allows you to define custom analyzers by specifying the tokenizer, character filters, and token filters to be used. This gives you fine-grained control over the text analysis process.

For information about specific analyzer parameters, such as analyzer, search_analyzer, or search_quote_analyzer, see Search analyzers.

Example

The following example configuration defines a custom analyzer called my_custom_analyzer:

  1. PUT my_index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "my_custom_analyzer": {
  7.           "type": "custom",
  8.           "tokenizer": "standard",
  9.           "filter": [
  10.             "lowercase",
  11.             "my_stop_filter",
  12.             "my_stemmer"
  13.           ]
  14.         }
  15.       },
  16.       "filter": {
  17.         "my_stop_filter": {
  18.           "type": "stop",
  19.           "stopwords": ["the", "a", "and", "or"]
  20.         },
  21.         "my_stemmer": {
  22.           "type": "stemmer",
  23.           "language": "english"
  24.         }
  25.       }
  26.     }
  27.   },
  28.   "mappings": {
  29.     "properties": {
  30.       "my_text_field": {
  31.         "type": "text",
  32.         "analyzer": "my_custom_analyzer",
  33.         "search_analyzer": "standard",
  34.         "search_quote_analyzer": "my_custom_analyzer"
  35.       }
  36.     }
  37.   }
  38. }

copy

In this example, the my_custom_analyzer uses the standard tokenizer, converts all tokens to lowercase, applies a custom stop word filter, and applies an English stemmer.

You can then map a text field so that it uses this custom analyzer for both index and search operations:

  1. "mappings": {
  2.   "properties": {
  3.     "my_text_field": {
  4.       "type": "text",
  5.       "analyzer": "my_custom_analyzer"
  6.     }
  7.   }
  8. }

copy