Decimal digit token filter

The decimal_digit token filter is used to normalize decimal digit characters (0–9) into their ASCII equivalents in various scripts. This is useful when you want to ensure that all digits are treated uniformly in text analysis, regardless of the script in which they are written.

Example

The following example request creates a new index named my_index and configures an analyzer with a decimal_digit filter:

  1. PUT /my_index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "filter": {
  6.         "my_decimal_digit_filter": {
  7.           "type": "decimal_digit"
  8.         }
  9.       },
  10.       "analyzer": {
  11.         "my_analyzer": {
  12.           "type": "custom",
  13.           "tokenizer": "standard",
  14.           "filter": ["my_decimal_digit_filter"]
  15.         }
  16.       }
  17.     }
  18.   }
  19. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. POST /my_index/_analyze
  2. {
  3.   "analyzer": "my_analyzer",
  4.   "text": "123 ١٢٣ १२३"
  5. }

copy

text breakdown:

  • “123” (ASCII digits)
  • “١٢٣” (Arabic-Indic digits)
  • “१२३” (Devanagari digits)

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "123",
  5.       "start_offset": 0,
  6.       "end_offset": 3,
  7.       "type": "<NUM>",
  8.       "position": 0
  9.     },
  10.     {
  11.       "token": "123",
  12.       "start_offset": 4,
  13.       "end_offset": 7,
  14.       "type": "<NUM>",
  15.       "position": 1
  16.     },
  17.     {
  18.       "token": "123",
  19.       "start_offset": 8,
  20.       "end_offset": 11,
  21.       "type": "<NUM>",
  22.       "position": 2
  23.     }
  24.   ]
  25. }