Basque analyzer

The built-in basque analyzer can be applied to a text field using the following command:

  1. PUT /basque-index
  2. {
  3. "mappings": {
  4. "properties": {
  5. "content": {
  6. "type": "text",
  7. "analyzer": "basque"
  8. }
  9. }
  10. }
  11. }

copy

Stem exclusion

You can use stem_exclusion with this language analyzer using the following command:

  1. PUT index_with_stem_exclusion_basque_analyzer
  2. {
  3. "settings": {
  4. "analysis": {
  5. "analyzer": {
  6. "stem_exclusion_basque_analyzer": {
  7. "type": "basque",
  8. "stem_exclusion": ["autoritate", "baldintza"]
  9. }
  10. }
  11. }
  12. }
  13. }

copy

Basque analyzer internals

The basque analyzer is built using the following components:

  • Tokenizer: standard

  • Token filters:

    • lowercase
    • stop (Basque)
    • keyword
    • stemmer (Basque)

Custom Basque analyzer

You can create a custom Basque analyzer using the following command:

  1. PUT /basque-index
  2. {
  3. "settings": {
  4. "analysis": {
  5. "filter": {
  6. "basque_stop": {
  7. "type": "stop",
  8. "stopwords": "_basque_"
  9. },
  10. "basque_stemmer": {
  11. "type": "stemmer",
  12. "language": "basque"
  13. },
  14. "basque_keywords": {
  15. "type": "keyword_marker",
  16. "keywords": []
  17. }
  18. },
  19. "analyzer": {
  20. "basque_analyzer": {
  21. "type": "custom",
  22. "tokenizer": "standard",
  23. "filter": [
  24. "lowercase",
  25. "basque_stop",
  26. "basque_keywords",
  27. "basque_stemmer"
  28. ]
  29. }
  30. }
  31. }
  32. },
  33. "mappings": {
  34. "properties": {
  35. "content": {
  36. "type": "text",
  37. "analyzer": "basque_analyzer"
  38. }
  39. }
  40. }
  41. }

copy

Generated tokens

Use the following request to examine the tokens generated using the analyzer:

  1. POST /basque-index/_analyze
  2. {
  3. "field": "content",
  4. "text": "Ikasleek euskal unibertsitateetan ikasten dute. Haien zenbakiak 123456 dira."
  5. }

copy

The response contains the generated tokens:

  1. {
  2. "tokens": [
  3. {"token": "ikasle","start_offset": 0,"end_offset": 8,"type": "<ALPHANUM>","position": 0},
  4. {"token": "euskal","start_offset": 9,"end_offset": 15,"type": "<ALPHANUM>","position": 1},
  5. {"token": "unibertsi","start_offset": 16,"end_offset": 33,"type": "<ALPHANUM>","position": 2},
  6. {"token": "ikas","start_offset": 34,"end_offset": 41,"type": "<ALPHANUM>","position": 3},
  7. {"token": "haien","start_offset": 48,"end_offset": 53,"type": "<ALPHANUM>","position": 5},
  8. {"token": "zenba","start_offset": 54,"end_offset": 63,"type": "<ALPHANUM>","position": 6},
  9. {"token": "123456","start_offset": 64,"end_offset": 70,"type": "<NUM>","position": 7}
  10. ]
  11. }