Term-level queries

OpenSearch supports two types of queries when you search for data: term-level queries and full-text queries.

The following table describes the differences between them:

 Term-level queriesFull-text queries
DescriptionTerm-level queries answer which documents match a query.Full-text queries answer how well the documents match a query.
AnalyzerThe search term isn’t analyzed. This means that the term query searches for your search term as it is.The search term is analyzed by the same analyzer that was used for the specific field of the document at the time it was indexed. This means that your search term goes through the same analysis process that the document’s field did.
RelevanceTerm-level queries simply return documents that match without sorting them based on the relevance score. They still calculate the relevance score, but this score is the same for all the documents that are returned.Full-text queries calculate a relevance score for each match and sort the results by decreasing order of relevance.
Use CaseUse term-level queries when you want to match exact values such as numbers, dates, tags, and so on, and don’t need the matches to be sorted by relevance.Use full-text queries to match text fields and sort by relevance after taking into account factors like casing and stemming variants.

OpenSearch uses a probabilistic ranking framework called Okapi BM25 to calculate relevance scores. To learn more about Okapi BM25, see Wikipedia.

Assume that you have the complete works of Shakespeare indexed in an OpenSearch cluster. We use a term-level query to search for the phrase “To be, or not to be” in the text_entry field:

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "term": {
  5.       "text_entry": "To be, or not to be"
  6.     }
  7.   }
  8. }

Sample response

  1. {
  2.   "took" : 3,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 0,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : null,
  16.     "hits" : [ ]
  17.   }
  18. }

We don’t get back any matches (hits). This is because the term “To be, or not to be” is searched literally in the inverted index, where only the analyzed values of the text fields are stored. Term-level queries aren’t suited for searching on analyzed text fields because they often yield unexpected results. When working with text data, use term-level queries only for fields mapped as keyword only.

Using a full-text query:

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "match": {
  5.       "text_entry": "To be, or not to be"
  6.     }
  7.   }
  8. }

The search query “To be, or not to be” is analyzed and tokenized into an array of tokens just like the text_entry field of the documents. The full-text query performs an intersection of tokens between our search query and the text_entry fields for all the documents, and then sorts the results by relevance scores:

Sample response

  1. {
  2.   "took" : 19,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 10000,
  13.       "relation" : "gte"
  14.     },
  15.     "max_score" : 17.419369,
  16.     "hits" : [
  17.       {
  18.         "_index" : "shakespeare",
  19.         "_id" : "34229",
  20.         "_score" : 17.419369,
  21.         "_source" : {
  22.           "type" : "line",
  23.           "line_id" : 34230,
  24.           "play_name" : "Hamlet",
  25.           "speech_number" : 19,
  26.           "line_number" : "3.1.64",
  27.           "speaker" : "HAMLET",
  28.           "text_entry" : "To be, or not to be: that is the question:"
  29.         }
  30.       },
  31.       {
  32.         "_index" : "shakespeare",
  33.         "_id" : "109930",
  34.         "_score" : 14.883024,
  35.         "_source" : {
  36.           "type" : "line",
  37.           "line_id" : 109931,
  38.           "play_name" : "A Winters Tale",
  39.           "speech_number" : 23,
  40.           "line_number" : "4.4.153",
  41.           "speaker" : "PERDITA",
  42.           "text_entry" : "Not like a corse; or if, not to be buried,"
  43.         }
  44.       },
  45.       {
  46.         "_index" : "shakespeare",
  47.         "_id" : "103117",
  48.         "_score" : 14.782743,
  49.         "_source" : {
  50.           "type" : "line",
  51.           "line_id" : 103118,
  52.           "play_name" : "Twelfth Night",
  53.           "speech_number" : 53,
  54.           "line_number" : "1.3.95",
  55.           "speaker" : "SIR ANDREW",
  56.           "text_entry" : "will not be seen; or if she be, its four to one"
  57.         }
  58.       }
  59.     ]
  60.   }
  61. }
  62. ...

For a list of all full-text queries, see Full-text queries.

If you want to query for an exact term like “HAMLET” in the speaker field and don’t need the results to be sorted by relevance scores, a term-level query is more efficient:

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "term": {
  5.       "speaker": "HAMLET"
  6.     }
  7.   }
  8. }

Sample response

  1. {
  2.   "took" : 5,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 1582,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 4.2540946,
  16.     "hits" : [
  17.       {
  18.         "_index" : "shakespeare",
  19.         "_id" : "32700",
  20.         "_score" : 4.2540946,
  21.         "_source" : {
  22.           "type" : "line",
  23.           "line_id" : 32701,
  24.           "play_name" : "Hamlet",
  25.           "speech_number" : 9,
  26.           "line_number" : "1.2.66",
  27.           "speaker" : "HAMLET",
  28.           "text_entry" : "[Aside]  A little more than kin, and less than kind."
  29.         }
  30.       },
  31.       {
  32.         "_index" : "shakespeare",
  33.         "_id" : "32702",
  34.         "_score" : 4.2540946,
  35.         "_source" : {
  36.           "type" : "line",
  37.           "line_id" : 32703,
  38.           "play_name" : "Hamlet",
  39.           "speech_number" : 11,
  40.           "line_number" : "1.2.68",
  41.           "speaker" : "HAMLET",
  42.           "text_entry" : "Not so, my lord; I am too much i' the sun."
  43.         }
  44.       },
  45.       {
  46.         "_index" : "shakespeare",
  47.         "_id" : "32709",
  48.         "_score" : 4.2540946,
  49.         "_source" : {
  50.           "type" : "line",
  51.           "line_id" : 32710,
  52.           "play_name" : "Hamlet",
  53.           "speech_number" : 13,
  54.           "line_number" : "1.2.75",
  55.           "speaker" : "HAMLET",
  56.           "text_entry" : "Ay, madam, it is common."
  57.         }
  58.       }
  59.     ]
  60.   }
  61. }
  62. ...

The term-level queries are exact matches. So, if you search for “Hamlet”, you don’t get back any matches, because “HAMLET” is a keyword field and is stored in OpenSearch literally and not in an analyzed form. The search query “HAMLET” is also searched literally. So, to get a match on this field, we need to enter the exact same characters.

Term

Use the term query to search for an exact term in a field.

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "term": {
  5.       "line_id": {
  6.         "value": "61809"
  7.       }
  8.     }
  9.   }
  10. }

Terms

Use the terms query to search for multiple terms in the same field.

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "terms": {
  5.       "line_id": [
  6.         "61809",
  7.         "61810"
  8.       ]
  9.     }
  10.   }
  11. }

You get back documents that match any of the terms.

IDs

Use the ids query to search for one or more document ID values.

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "ids": {
  5.       "values": [
  6.         34229,
  7.         91296
  8.       ]
  9.     }
  10.   }
  11. }

Range

Use the range query to search for a range of values in a field.

To search for documents where the line_id value is >= 10 and <= 20:

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "range": {
  5.       "line_id": {
  6.         "gte": 10,
  7.         "lte": 20
  8.       }
  9.     }
  10.   }
  11. }
ParameterBehavior
gteGreater than or equal to.
gtGreater than.
lteLess than or equal to.
ltLess than.

Assume that you have a products index and you want to find all the products that were added in the year 2019:

  1. GET products/_search
  2. {
  3.   "query": {
  4.     "range": {
  5.       "created": {
  6.         "gte": "2019/01/01",
  7.         "lte": "2019/12/31"
  8.       }
  9.     }
  10.   }
  11. }

Specify relative dates by using basic math expressions.

To subtract 1 year and 1 day from the specified date:

  1. GET products/_search
  2. {
  3.   "query": {
  4.     "range": {
  5.       "created": {
  6.         "gte": "2019/01/01||-1y-1d"
  7.       }
  8.     }
  9.   }
  10. }

The first date that we specify is the anchor date or the starting point for the date math. Add two trailing pipe symbols. You could then add one day (+1d) or subtract two weeks (-2w). This math expression is relative to the anchor date that you specify.

You could also round off dates by adding a forward slash to the date or time unit.

To find products added in the last year and rounded off by month:

  1. GET products/_search
  2. {
  3.   "query": {
  4.     "range": {
  5.       "created": {
  6.         "gte": "now-1y/M"
  7.       }
  8.     }
  9.   }
  10. }

The keyword now refers to the current date and time.

Prefix

Use the prefix query to search for terms that begin with a specific prefix.

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "prefix": {
  5.       "speaker": "KING"
  6.     }
  7.   }
  8. }

Exists

Use the exists query to search for documents that contain a specific field.

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "exists": {
  5.       "field": "speaker"
  6.     }
  7.   }
  8. }

Wildcards

Use wildcard queries to search for terms that match a wildcard pattern.

FeatureBehavior
*Specifies all valid values.
?Specifies a single valid value.

To search for terms that start with H and end with Y:

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "wildcard": {
  5.       "speaker": {
  6.         "value": "H*Y"
  7.       }
  8.     }
  9.   }
  10. }

If we change * to ?, we get no matches, because ? refers to a single character.

Wildcard queries tend to be slow because they need to iterate over a lot of terms. Avoid placing wildcard characters at the beginning of a query because it could be a very expensive operation in terms of both resources and time.

Regex

Use the regexp query to search for terms that match a regular expression.

This regular expression matches any single uppercase or lowercase letter:

  1. GET shakespeare/_search
  2. {
  3.   "query": {
  4.     "regexp": {
  5.       "play_name": "[a-zA-Z]amlet"
  6.     }
  7.   }
  8. }

A few important notes:

  • Regular expressions are applied to the terms in the field (i.e. tokens), not the entire field.
  • Regular expressions use the Lucene syntax, which differs from more standardized implementations. Test thoroughly to ensure that you receive the results you expect. To learn more, see the Lucene documentation.
  • regexp queries can be expensive operations and require the search.allow_expensive_queries setting to be set to true. Before making frequent regexp queries, test their impact on cluster performance and examine alternative queries for achieving similar results.