Keyword search

By default, OpenSearch calculates document scores using the Okapi BM25 algorithm. BM25 is a keyword-based algorithm that performs lexical search for words that appear in the query.

When determining a document’s relevance, BM25 considers term frequency/inverse document frequency (TF/IDF):

  • Term frequency stipulates that documents in which the search term appears more frequently are more relevant.

  • Inverse document frequency gives less weight to the words that commonly appear in all documents in the corpus (for example, articles like “the”).

Example

The following example query searches for the words long live king in the shakespeare index:

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "match": {
  5.       "text_entry": "long live king"
  6.     }
  7.   }
  8. }

copy

The response contains the matching documents, each with a relevance score in the _score field:

  1. {
  2.   "took": 113,
  3.   "timed_out": false,
  4.   "_shards": {
  5.     "total": 1,
  6.     "successful": 1,
  7.     "skipped": 0,
  8.     "failed": 0
  9.   },
  10.   "hits": {
  11.     "total": {
  12.       "value": 2352,
  13.       "relation": "eq"
  14.     },
  15.     "max_score": 18.781435,
  16.     "hits": [
  17.       {
  18.         "_index": "shakespeare",
  19.         "_id": "32437",
  20.         "_score": 18.781435,
  21.         "_source": {
  22.           "type": "line",
  23.           "line_id": 32438,
  24.           "play_name": "Hamlet",
  25.           "speech_number": 3,
  26.           "line_number": "1.1.3",
  27.           "speaker": "BERNARDO",
  28.           "text_entry": "Long live the king!"
  29.         }
  30.       },
  31.       {
  32.         "_index": "shakespeare",
  33.         "_id": "83798",
  34.         "_score": 16.523308,
  35.         "_source": {
  36.           "type": "line",
  37.           "line_id": 83799,
  38.           "play_name": "Richard III",
  39.           "speech_number": 42,
  40.           "line_number": "3.7.242",
  41.           "speaker": "BUCKINGHAM",
  42.           "text_entry": "Long live Richard, Englands royal king!"
  43.         }
  44.       },
  45.       {
  46.         "_index": "shakespeare",
  47.         "_id": "82994",
  48.         "_score": 15.588365,
  49.         "_source": {
  50.           "type": "line",
  51.           "line_id": 82995,
  52.           "play_name": "Richard III",
  53.           "speech_number": 24,
  54.           "line_number": "3.1.80",
  55.           "speaker": "GLOUCESTER",
  56.           "text_entry": "live long."
  57.         }
  58.       },
  59.       {
  60.         "_index": "shakespeare",
  61.         "_id": "7199",
  62.         "_score": 15.586321,
  63.         "_source": {
  64.           "type": "line",
  65.           "line_id": 7200,
  66.           "play_name": "Henry VI Part 2",
  67.           "speech_number": 12,
  68.           "line_number": "2.2.64",
  69.           "speaker": "BOTH",
  70.           "text_entry": "Long live our sovereign Richard, Englands king!"
  71.         }
  72.       }
  73.       ...
  74.     ]
  75.   }
  76. }

Similarity algorithms

The following table lists the supported similarity algorithms.

AlgorithmDescription
BM25The default OpenSearch Okapi BM25 similarity algorithm.
booleanAssigns terms a score equal to their boost value. Use boolean similarity when you want the document scores to be based on the binary value of whether the terms match.

Specifying similarity

You can specify the similarity algorithm in the similarity parameter when configuring mappings at the field level.

For example, the following query specifies the boolean similarity for the boolean_field. The bm25_field is assigned the default BM25 similarity:

  1. PUT /testindex
  2. {
  3.   "mappings": {
  4.     "properties": {
  5.       "bm25_field": { 
  6.         "type": "text"
  7.       },
  8.       "boolean_field": {
  9.         "type": "text",
  10.         "similarity": "boolean" 
  11.       }
  12.     }
  13.   }
  14. }

copy

Configuring BM25 similarity

You can configure BM25 similarity parameters at the index level as follows:

  1. PUT /testindex
  2. {
  3.   "settings": {
  4.     "index": {
  5.       "similarity": {
  6.         "custom_similarity": {
  7.           "type": "BM25",
  8.           "k1": 1.2,
  9.           "b": 0.75,
  10.           "discount_overlaps": "true"
  11.         }
  12.       }
  13.     }
  14.   }
  15. }

BM25 similarity supports the following parameters.

ParameterData typeDescription
k1FloatDetermines non-linear term frequency normalization (saturation) properties. The default value is 1.2.
bFloatDetermines the degree to which document length normalizes TF values. The default value is 0.75.
discount_overlapsBooleanDetermines whether overlap tokens (tokens with zero position increment) are ignored when computing the norm. Default is true (overlap tokens do not count when computing the norm).

Next steps