Apostrophe token filter

The apostrophe token filter’s primary function is to remove possessive apostrophes and anything following them. This can be very useful in analyzing text in languages that rely heavily on apostrophes, such as Turkish, in which apostrophes separate the root word from suffixes, including possessive suffixes, case markers, and other grammatical endings.

Example

The following example request creates a new index named custom_text_index with a custom analyzer configured in settings and used in mappings:

  1. PUT /custom_text_index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "custom_analyzer": {
  7.           "type": "custom",
  8.           "tokenizer": "standard",
  9.           "filter": [
  10.             "lowercase",
  11.             "apostrophe"
  12.           ]
  13.         }
  14.       }
  15.     }
  16.   },
  17.   "mappings": {
  18.     "properties": {
  19.       "content": {
  20.         "type": "text",
  21.         "analyzer": "custom_analyzer"
  22.       }
  23.     }
  24.   }
  25. }

copy

Generated tokens

Use the following request to examine the tokens generated using the created analyzer:

  1. POST /custom_text_index/_analyze
  2. {
  3.   "analyzer": "custom_analyzer",
  4.   "text": "John's car is faster than Peter's bike"
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "john",
  5.       "start_offset": 0,
  6.       "end_offset": 6,
  7.       "type": "<ALPHANUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "car",
  12.       "start_offset": 7,
  13.       "end_offset": 10,
  14.       "type": "<ALPHANUM>",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "is",
  19.       "start_offset": 11,
  20.       "end_offset": 13,
  21.       "type": "<ALPHANUM>",
  22.       "position": 2
  23.     },
  24.     {
  25.       "token": "faster",
  26.       "start_offset": 14,
  27.       "end_offset": 20,
  28.       "type": "<ALPHANUM>",
  29.       "position": 3
  30.     },
  31.     {
  32.       "token": "than",
  33.       "start_offset": 21,
  34.       "end_offset": 25,
  35.       "type": "<ALPHANUM>",
  36.       "position": 4
  37.     },
  38.     {
  39.       "token": "peter",
  40.       "start_offset": 26,
  41.       "end_offset": 33,
  42.       "type": "<ALPHANUM>",
  43.       "position": 5
  44.     },
  45.     {
  46.       "token": "bike",
  47.       "start_offset": 34,
  48.       "end_offset": 38,
  49.       "type": "<ALPHANUM>",
  50.       "position": 6
  51.     }
  52.   ]
  53. }

The built-in apostrophe token filter is not suitable for languages such as French, in which apostrophes are used at the beginning of words. For example, "C'est l'amour de l'école" will result in four tokens: “C”, “l”, “de”, and “l”.