Whitespace tokenizer
Whitespace tokenizer
The whitespace
tokenizer breaks text into terms whenever it encounters a whitespace character.
Example output
resp = client.indices.analyze(
tokenizer="whitespace",
text="The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.",
)
print(resp)
response = client.indices.analyze(
body: {
tokenizer: 'whitespace',
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
)
puts response
const response = await client.indices.analyze({
tokenizer: "whitespace",
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.",
});
console.log(response);
POST _analyze
{
"tokenizer": "whitespace",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
The above sentence would produce the following terms:
[ The, 2, QUICK, Brown-Foxes, jumped, over, the, lazy, dog's, bone. ]
Configuration
The whitespace
tokenizer accepts the following parameters:
| The maximum token length. If a token is seen that exceeds this length then it is split at |
当前内容版权归 elasticsearch 或其关联方所有,如需对内容或内容相关联开源项目进行关注与资助,请访问 elasticsearch .