Length token filter

Removes tokens shorter or longer than specified character lengths. For example, you can use the length filter to exclude tokens shorter than 2 characters and tokens longer than 5 characters.

This filter uses Lucene’s LengthFilter.

The length filter removes entire tokens. If you’d prefer to shorten tokens to a specific length, use the truncate filter.

Example

The following analyze API request uses the length filter to remove tokens longer than 4 characters:

  1. GET _analyze
  2. {
  3.   "tokenizer": "whitespace",
  4.   "filter": [
  5.     {
  6.       "type": "length",
  7.       "min": 0,
  8.       "max": 4
  9.     }
  10.   ],
  11.   "text": "the quick brown fox jumps over the lazy dog"
  12. }

The filter produces the following tokens:

  1. [ the, fox, over, the, lazy, dog ]

Add to an analyzer

The following create index API request uses the length filter to configure a new custom analyzer.

  1. PUT length_example
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "standard_length": {
  7.           "tokenizer": "standard",
  8.           "filter": [ "length" ]
  9.         }
  10.       }
  11.     }
  12.   }
  13. }

Configurable parameters

min

(Optional, integer) Minimum character length of a token. Shorter tokens are excluded from the output. Defaults to 0.

max

(Optional, integer) Maximum character length of a token. Longer tokens are excluded from the output. Defaults to Integer.MAX_VALUE, which is 2^31-1 or 2147483647.

Customize

To customize the length filter, duplicate it to create the basis for a new custom token filter. You can modify the filter using its configurable parameters.

For example, the following request creates a custom length filter that removes tokens shorter than 2 characters and tokens longer than 10 characters:

  1. PUT length_custom_example
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "whitespace_length_2_to_10_char": {
  7.           "tokenizer": "whitespace",
  8.           "filter": [ "length_2_to_10_char" ]
  9.         }
  10.       },
  11.       "filter": {
  12.         "length_2_to_10_char": {
  13.           "type": "length",
  14.           "min": 2,
  15.           "max": 10
  16.         }
  17.       }
  18.     }
  19.   }
  20. }