Pattern tokenizer

Pattern tokenizer

The pattern tokenizer uses a regular expression to either split text into terms whenever it matches a word separator, or to capture matching text as terms.

The default pattern is \W+, which splits text whenever it encounters non-word characters.

Beware of Pathological Regular Expressions

The pattern tokenizer uses Java Regular Expressions.

A badly written regular expression could run very slowly or even throw a StackOverflowError and cause the node it is running on to exit suddenly.

Read more about pathological regular expressions and how to avoid them.

Example output

  1. POST _analyze
  2. {
  3.   "tokenizer": "pattern",
  4.   "text": "The foo_bar_size's default is 5."
  5. }

The above sentence would produce the following terms:

  1. [ The, foo_bar_size, s, default, is, 5 ]

Configuration

The pattern tokenizer accepts the following parameters:

pattern

A Java regular expression, defaults to \W+.

flags

Java regular expression flags. Flags should be pipe-separated, eg “CASE_INSENSITIVE|COMMENTS”.

group

Which capture group to extract as tokens. Defaults to -1 (split).

Example configuration

In this example, we configure the pattern tokenizer to break text into tokens when it encounters commas:

  1. PUT my-index-000001
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "my_analyzer": {
  7.           "tokenizer": "my_tokenizer"
  8.         }
  9.       },
  10.       "tokenizer": {
  11.         "my_tokenizer": {
  12.           "type": "pattern",
  13.           "pattern": ","
  14.         }
  15.       }
  16.     }
  17.   }
  18. }
  20. POST my-index-000001/_analyze
  21. {
  22.   "analyzer": "my_analyzer",
  23.   "text": "comma,separated,values"
  24. }

The above example produces the following terms:

  1. [ comma, separated, values ]

In the next example, we configure the pattern tokenizer to capture values enclosed in double quotes (ignoring embedded escaped quotes \"). The regex itself looks like this:

  1. "((?:\\"|[^"]|\\")*)"

And reads as follows:

  • A literal "

  • Start capturing:

    • A literal \" OR any character except "
    • Repeat until no more characters match

  • A literal closing "

When the pattern is specified in JSON, the " and \ characters need to be escaped, so the pattern ends up looking like:

  1. \"((?:\\\\\"|[^\"]|\\\\\")+)\"
  1. PUT my-index-000001
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "my_analyzer": {
  7.           "tokenizer": "my_tokenizer"
  8.         }
  9.       },
  10.       "tokenizer": {
  11.         "my_tokenizer": {
  12.           "type": "pattern",
  13.           "pattern": "\"((?:\\\\\"|[^\"]|\\\\\")+)\"",
  14.           "group": 1
  15.         }
  16.       }
  17.     }
  18.   }
  19. }
  21. POST my-index-000001/_analyze
  22. {
  23.   "analyzer": "my_analyzer",
  24.   "text": "\"value\", \"value with embedded \\\" quote\""
  25. }

The above example produces the following two terms:

  1. [ value, value with embedded \" quote ]