Normalization token filter

The normalization token filter is designed to adjust and simplify text in a way that reduces variations, particularly variations in special characters. It is primarily used to handle variations in writing by standardizing characters in specific languages.

The following normalization token filters are available:

Example

The following example request creates a new index named german_normalizer_example and configures an analyzer with a german_normalization filter:

  1. PUT /german_normalizer_example
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "filter": {
  6.         "german_normalizer": {
  7.           "type": "german_normalization"
  8.         }
  9.       },
  10.       "analyzer": {
  11.         "german_normalizer_analyzer": {
  12.           "tokenizer": "standard",
  13.           "filter": [
  14.             "lowercase", 
  15.             "german_normalizer"
  16.           ]
  17.         }
  18.       }
  19.     }
  20.   }
  21. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. POST /german_normalizer_example/_analyze
  2. {
  3.   "text": "Straße München",
  4.   "analyzer": "german_normalizer_analyzer"
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "strasse",
  5.       "start_offset": 0,
  6.       "end_offset": 6,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "munchen",
  12.       "start_offset": 7,
  13.       "end_offset": 14,
  14.       "type": "<ALPHANUM>",
  15.       "position": 1
  16.     }
  17.   ]
  18. }