Fingerprint token filter

The fingerprint token filter is used to standardize and deduplicate text. This is particularly useful when consistency in text processing is crucial. The fingerprint token filter achieves this by processing text using the following steps:

  1. Lowercasing: Converts all text to lowercase.
  2. Splitting: Breaks the text into tokens.
  3. Sorting: Arranges the tokens in alphabetical order.
  4. Removing duplicates: Eliminates repeated tokens.
  5. Joining tokens: Combines the tokens into a single string, typically joined by a space or another specified separator.

Parameters

The fingerprint token filter can be configured with the following two parameters.

ParameterRequired/OptionalData typeDescription
max_output_sizeOptionalIntegerLimits the length of the generated fingerprint string. If the concatenated string exceeds the max_output_size, the filter will not produce any output, resulting in an empty token. Default is 255.
separatorOptionalStringDefines the character(s) used to join the tokens into a single string after they have been sorted and deduplicated. Default is space (“ “).

Example

The following example request creates a new index named my_index and configures an analyzer with a fingerprint token filter:

  1. PUT /my_index
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "filter": {
  6.         "my_fingerprint": {
  7.           "type": "fingerprint",
  8.           "max_output_size": 200,
  9.           "separator": "-"
  10.         }
  11.       },
  12.       "analyzer": {
  13.         "my_analyzer": {
  14.           "type": "custom",
  15.           "tokenizer": "standard",
  16.           "filter": [
  17.             "lowercase",
  18.             "my_fingerprint"
  19.           ]
  20.         }
  21.       }
  22.     }
  23.   }
  24. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. POST /my_index/_analyze
  2. {
  3.   "analyzer": "my_analyzer",
  4.   "text": "OpenSearch is a powerful search engine that scales easily"
  5. }

copy

The response contains the generated tokens:

  1. {
  2.   "tokens": [
  3.     {
  4.       "token": "a-easily-engine-is-opensearch-powerful-scales-search-that",
  5.       "start_offset": 0,
  6.       "end_offset": 57,
  7.       "type": "fingerprint",
  8.       "position": 0
  9.     }
  10.   ]
  11. }