Word delimiter graph token filter

The word_delimiter_graph token filter is used to split tokens at predefined characters and also offers optional token normalization based on customizable rules.

The word_delimiter_graph filter is used to remove punctuation from complex identifiers like part numbers or product IDs. In such cases, it is best used with the keyword tokenizer. For hyphenated words, use the synonym_graph token filter instead of the word_delimiter_graph filter because users frequently search for these terms both with and without hyphens.

By default, the filter applies the following rules.

It’s important not to use tokenizers that strip punctuation, like the standard tokenizer, with this filter. Doing so may prevent proper token splitting and interfere with options like catenate_all or preserve_original. We recommend using this filter with a keyword or whitespace tokenizer.

Parameters

You can configure the word_delimiter_graph token filter using the following parameters.

Example

The following example request creates a new index named my-custom-index and configures an analyzer with a word_delimiter_graph filter:

PUT /my-custom-index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "tokenizer": "keyword",
          "filter": [ "custom_word_delimiter_filter" ]
        }
      },
      "filter": {
        "custom_word_delimiter_filter": {
          "type": "word_delimiter_graph",
          "split_on_case_change": true,
          "split_on_numerics": true,
          "stem_english_possessive": true
        }
      }
    }
  }
}

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

GET /my-custom-index/_analyze
{
  "analyzer": "custom_analyzer",
  "text": "FastCar's Model2023"
}

copy

The response contains the generated tokens:

{
  "tokens": [
    {
      "token": "Fast",
      "start_offset": 0,
      "end_offset": 4,
      "type": "word",
      "position": 0
    },
    {
      "token": "Car",
      "start_offset": 4,
      "end_offset": 7,
      "type": "word",
      "position": 1
    },
    {
      "token": "Model",
      "start_offset": 10,
      "end_offset": 15,
      "type": "word",
      "position": 2
    },
    {
      "token": "2023",
      "start_offset": 15,
      "end_offset": 19,
      "type": "word",
      "position": 3
    }
  ]
}

Differences between the word_delimiter_graph and word_delimiter filters

Both the word_delimiter_graph and word_delimiter token filters generate tokens spanning multiple positions when any of the following parameters are set to true:

• catenate_all
• catenate_numbers
• catenate_words
• preserve_original

To illustrate the differences between these filters, consider the input text Pro-XT500.

word_delimiter_graph

The word_delimiter_graph filter assigns a positionLength attribute to multi-position tokens, indicating how many positions a token spans. This ensures that the filter always generates valid token graphs, making it suitable for use in advanced token graph scenarios. Although token graphs with multi-position tokens are not supported for indexing, they can still be useful in search scenarios. For example, queries like match_phrase can use these graphs to generate multiple subqueries from a single input string. For the example input text, the word_delimiter_graph filter generates the following tokens:

• Pro (position 1)
• XT500 (position 2)
• ProXT500 (position 1, positionLength: 2)

The positionLength attribute the production of a valid graph to be used in advanced queries.

word_delimiter

In contrast, the word_delimiter filter does not assign a positionLength attribute to multi-position tokens, leading to invalid graphs when these tokens are present. For the example input text, the word_delimiter filter generates the following tokens:

• Pro (position 1)
• XT500 (position 2)
• ProXT500 (position 1, no positionLength)

The lack of a positionLength attribute results in a token graph that is invalid for token streams containing multi-position tokens.