CJK width token filter

CJK width token filter

Normalizes width differences in CJK (Chinese, Japanese, and Korean) characters as follows:

  • Folds full-width ASCII character variants into the equivalent basic Latin characters
  • Folds half-width Katakana character variants into the equivalent Kana characters

This filter is included in Elasticsearch’s built-in CJK language analyzer. It uses Lucene’s CJKWidthFilter.

This token filter can be viewed as a subset of NFKC/NFKD Unicode normalization. See the analysis-icu plugin for full normalization support.

Example

  1. resp = client.indices.analyze(
  2.     tokenizer="standard",
  3.     filter=[
  4.         "cjk_width"
  5.     ],
  6.     text="ｼｰｻｲﾄﾞﾗｲﾅｰ",
  7. )
  8. print(resp)
  1. response = client.indices.analyze(
  2.   body: {
  3.     tokenizer: 'standard',
  4.     filter: [
  5.       'cjk_width'
  6.     ],
  7.     text: 'ｼｰｻｲﾄﾞﾗｲﾅｰ'
  8.   }
  9. )
  10. puts response
  1. const response = await client.indices.analyze({
  2.   tokenizer: "standard",
  3.   filter: ["cjk_width"],
  4.   text: "ｼｰｻｲﾄﾞﾗｲﾅｰ",
  5. });
  6. console.log(response);
  1. GET /_analyze
  2. {
  3.   "tokenizer" : "standard",
  4.   "filter" : ["cjk_width"],
  5.   "text" : "ｼｰｻｲﾄﾞﾗｲﾅｰ"
  6. }

The filter produces the following token:

  1. シーサイドライナー

Add to an analyzer

The following create index API request uses the CJK width token filter to configure a new custom analyzer.

  1. resp = client.indices.create(
  2.     index="cjk_width_example",
  3.     settings={
  4.         "analysis": {
  5.             "analyzer": {
  6.                 "standard_cjk_width": {
  7.                     "tokenizer": "standard",
  8.                     "filter": [
  9.                         "cjk_width"
  10.                     ]
  11.                 }
  12.             }
  13.         }
  14.     },
  15. )
  16. print(resp)
  1. response = client.indices.create(
  2.   index: 'cjk_width_example',
  3.   body: {
  4.     settings: {
  5.       analysis: {
  6.         analyzer: {
  7.           standard_cjk_width: {
  8.             tokenizer: 'standard',
  9.             filter: [
  10.               'cjk_width'
  11.             ]
  12.           }
  13.         }
  14.       }
  15.     }
  16.   }
  17. )
  18. puts response
  1. const response = await client.indices.create({
  2.   index: "cjk_width_example",
  3.   settings: {
  4.     analysis: {
  5.       analyzer: {
  6.         standard_cjk_width: {
  7.           tokenizer: "standard",
  8.           filter: ["cjk_width"],
  9.         },
  10.       },
  11.     },
  12.   },
  13. });
  14. console.log(response);
  1. PUT /cjk_width_example
  2. {
  3.   "settings": {
  4.     "analysis": {
  5.       "analyzer": {
  6.         "standard_cjk_width": {
  7.           "tokenizer": "standard",
  8.           "filter": [ "cjk_width" ]
  9.         }
  10.       }
  11.     }
  12.   }
  13. }