Test an analyzer

The analyze API is an invaluable tool for viewing the terms produced by an analyzer. A built-in analyzer can be specified inline in the request:

  1. POST _analyze
  2. {
  3.   "analyzer": "whitespace",
  4.   "text":     "The quick brown fox."
  5. }

The API returns the following response:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "The",
  5.       "start_offset": 0,
  6.       "end_offset": 3,
  7.       "type": "word",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "quick",
  12.       "start_offset": 4,
  13.       "end_offset": 9,
  14.       "type": "word",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "brown",
  19.       "start_offset": 10,
  20.       "end_offset": 15,
  21.       "type": "word",
  22.       "position": 2
  23.     },
  24.     {
  25.       "token": "fox.",
  26.       "start_offset": 16,
  27.       "end_offset": 20,
  28.       "type": "word",
  29.       "position": 3
  30.     }
  31.   ]
  32. }

You can also test combinations of:

  • A tokenizer
  • Zero or more token filters
  • Zero or more character filters
  1. POST _analyze
  2. {
  3.   "tokenizer": "standard",
  4.   "filter":  [ "lowercase", "asciifolding" ],
  5.   "text":      "Is this déja vu?"
  6. }

The API returns the following response:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "is",
  5.       "start_offset": 0,
  6.       "end_offset": 2,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "this",
  12.       "start_offset": 3,
  13.       "end_offset": 7,
  14.       "type": "<ALPHANUM>",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "deja",
  19.       "start_offset": 8,
  20.       "end_offset": 12,
  21.       "type": "<ALPHANUM>",
  22.       "position": 2
  23.     },
  24.     {
  25.       "token": "vu",
  26.       "start_offset": 13,
  27.       "end_offset": 15,
  28.       "type": "<ALPHANUM>",
  29.       "position": 3
  30.     }
  31.   ]
  32. }

Positions and character offsets

As can be seen from the output of the analyze API, analyzers not only convert words into terms, they also record the order or relative positions of each term (used for phrase queries or word proximity queries), and the start and end character offsets of each term in the original text (used for highlighting search snippets).

Alternatively, a custom analyzer can be referred to when running the analyze API on a specific index:

  1. PUT my-index-000001
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "std_folded": { 
  7.           "type": "custom",
  8.           "tokenizer": "standard",
  9.           "filter": [
  10.             "lowercase",
  11.             "asciifolding"
  12.           ]
  13.         }
  14.       }
  15.     }
  16.   },
  17.   "mappings": {
  18.     "properties": {
  19.       "my_text": {
  20.         "type": "text",
  21.         "analyzer": "std_folded" 
  22.       }
  23.     }
  24.   }
  25. }
  27. GET my-index-000001/_analyze 
  28. {
  29.   "analyzer": "std_folded", 
  30.   "text":     "Is this déjà vu?"
  31. }
  33. GET my-index-000001/_analyze 
  34. {
  35.   "field": "my_text", 
  36.   "text":  "Is this déjà vu?"
  37. }

The API returns the following response:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "is",
  5.       "start_offset": 0,
  6.       "end_offset": 2,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "this",
  12.       "start_offset": 3,
  13.       "end_offset": 7,
  14.       "type": "<ALPHANUM>",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "deja",
  19.       "start_offset": 8,
  20.       "end_offset": 12,
  21.       "type": "<ALPHANUM>",
  22.       "position": 2
  23.     },
  24.     {
  25.       "token": "vu",
  26.       "start_offset": 13,
  27.       "end_offset": 15,
  28.       "type": "<ALPHANUM>",
  29.       "position": 3
  30.     }
  31.   ]
  32. }

Define a custom analyzer called std_folded.

The field my_text uses the std_folded analyzer.

To refer to this analyzer, the analyze API must specify the index name.

Refer to the analyzer by name.

Refer to the analyzer used by field my_text.