Classic token filter

The primary function of the classic token filter is to work alongside the classic tokenizer. It processes tokens by applying the following common transformations, which aid in text analysis and search:

  • Removal of possessive endings such as ’s. For example, John’s becomes John.
  • Removal of periods from acronyms. For example, D.A.R.P.A. becomes DARPA.

Example

The following example request creates a new index named custom_classic_filter and configures an analyzer with the classic filter:

  1. PUT /custom_classic_filter
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "custom_classic": {
  7.           "type": "custom",
  8.           "tokenizer": "classic",
  9.           "filter": ["classic"]
  10.         }
  11.       }
  12.     }
  13.   }
  14. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. POST /custom_classic_filter/_analyze
  2. {
  3.   "analyzer": "custom_classic",
  4.   "text": "John's co-operate was excellent."
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "John",
  5.       "start_offset": 0,
  6.       "end_offset": 6,
  7.       "type": "<APOSTROPHE>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "co",
  12.       "start_offset": 7,
  13.       "end_offset": 9,
  14.       "type": "<ALPHANUM>",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "operate",
  19.       "start_offset": 10,
  20.       "end_offset": 17,
  21.       "type": "<ALPHANUM>",
  22.       "position": 2
  23.     },
  24.     {
  25.       "token": "was",
  26.       "start_offset": 18,
  27.       "end_offset": 21,
  28.       "type": "<ALPHANUM>",
  29.       "position": 3
  30.     },
  31.     {
  32.       "token": "excellent",
  33.       "start_offset": 22,
  34.       "end_offset": 31,
  35.       "type": "<ALPHANUM>",
  36.       "position": 4
  37.     }
  38.   ]
  39. }