Hybrid search

Introduced 2.11

Hybrid search combines keyword and neural search to improve search relevance. To implement hybrid search, you need to set up a search pipeline that runs at search time. The search pipeline you’ll configure intercepts search results at an intermediate stage and applies the normalization_processor to them. The normalization_processor normalizes and combines the document scores from multiple query clauses, rescoring the documents according to the chosen normalization and combination techniques.

PREREQUISITE
Before using hybrid search, you must set up a text embedding model. For more information, see Choosing a model.

To use hybrid search, follow these steps:

  1. Create an ingest pipeline.
  2. Create an index for ingestion.
  3. Ingest documents into the index.
  4. Configure a search pipeline.
  5. Search the index using hybrid search.

Step 1: Create an ingest pipeline

To generate vector embeddings, you need to create an ingest pipeline that contains a text_embedding processor, which will convert the text in a document field to vector embeddings. The processor’s field_map determines the input fields from which to generate vector embeddings and the output fields in which to store the embeddings.

The following example request creates an ingest pipeline that converts the text from passage_text to text embeddings and stores the embeddings in passage_embedding:

  1. PUT /_ingest/pipeline/nlp-ingest-pipeline
  2. {
  3.   "description": "A text embedding pipeline",
  4.   "processors": [
  5.     {
  6.       "text_embedding": {
  7.         "model_id": "bQ1J8ooBpBj3wT4HVUsb",
  8.         "field_map": {
  9.           "passage_text": "passage_embedding"
  10.         }
  11.       }
  12.     }
  13.   ]
  14. }

copy

Step 2: Create an index for ingestion

In order to use the text embedding processor defined in your pipeline, create a k-NN index, adding the pipeline created in the previous step as the default pipeline. Ensure that the fields defined in the field_map are mapped as correct types. Continuing with the example, the passage_embedding field must be mapped as a k-NN vector with a dimension that matches the model dimension. Similarly, the passage_text field should be mapped as text.

The following example request creates a k-NN index that is set up with a default ingest pipeline:

  1. PUT /my-nlp-index
  2. {
  3.   "settings": {
  4.     "index.knn": true,
  5.     "default_pipeline": "nlp-ingest-pipeline"
  6.   },
  7.   "mappings": {
  8.     "properties": {
  9.       "id": {
  10.         "type": "text"
  11.       },
  12.       "passage_embedding": {
  13.         "type": "knn_vector",
  14.         "dimension": 768,
  15.         "method": {
  16.           "engine": "lucene",
  17.           "space_type": "l2",
  18.           "name": "hnsw",
  19.           "parameters": {}
  20.         }
  21.       },
  22.       "passage_text": {
  23.         "type": "text"
  24.       }
  25.     }
  26.   }
  27. }

copy

For more information about creating a k-NN index and using supported methods, see k-NN index.

Step 3: Ingest documents into the index

To ingest documents into the index created in the previous step, send the following requests:

  1. PUT /my-nlp-index/_doc/1
  2. {
  3.   "passage_text": "Hello world",
  4.   "id": "s1"
  5. }

copy

  1. PUT /my-nlp-index/_doc/2
  2. {
  3.   "passage_text": "Hi planet",
  4.   "id": "s2"
  5. }

copy

Before the document is ingested into the index, the ingest pipeline runs the text_embedding processor on the document, generating text embeddings for the passage_text field. The indexed document includes the passage_text field, which contains the original text, and the passage_embedding field, which contains the vector embeddings.

Step 4: Configure a search pipeline

To configure a search pipeline with a normalization-processor, use the following request. The normalization technique in the processor is set to min_max, and the combination technique is set to arithmetic_mean. The weights array specifies the weights assigned to each query clause as decimal percentages:

  1. PUT /_search/pipeline/nlp-search-pipeline
  2. {
  3.   "description": "Post processor for hybrid search",
  4.   "phase_results_processors": [
  5.     {
  6.       "normalization-processor": {
  7.         "normalization": {
  8.           "technique": "min_max"
  9.         },
  10.         "combination": {
  11.           "technique": "arithmetic_mean",
  12.           "parameters": {
  13.             "weights": [
  14.               0.3,
  15.               0.7
  16.             ]
  17.           }
  18.         }
  19.       }
  20.     }
  21.   ]
  22. }

copy

To perform hybrid search on your index, use the hybrid query, which combines the results of keyword and semantic search.

Example: Combining a neural query and a match query

The following example request combines two query clauses—a neural query and a match query. It specifies the search pipeline created in the previous step as a query parameter:

  1. GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline
  2. {
  3.   "_source": {
  4.     "exclude": [
  5.       "passage_embedding"
  6.     ]
  7.   },
  8.   "query": {
  9.     "hybrid": {
  10.       "queries": [
  11.         {
  12.           "match": {
  13.             "passage_text": {
  14.               "query": "Hi world"
  15.             }
  16.           }
  17.         },
  18.         {
  19.           "neural": {
  20.             "passage_embedding": {
  21.               "query_text": "Hi world",
  22.               "model_id": "aVeif4oB5Vm0Tdw8zYO2",
  23.               "k": 5
  24.             }
  25.           }
  26.         }
  27.       ]
  28.     }
  29.   }
  30. }

copy

Alternatively, you can set a default search pipeline for the my-nlp-index index. For more information, see Default search pipeline.

The response contains the matching document:

  1. {
  2.   "took" : 36,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 1,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 1.2251667,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my-nlp-index",
  19.         "_id" : "1",
  20.         "_score" : 1.2251667,
  21.         "_source" : {
  22.           "passage_text" : "Hello world",
  23.           "id" : "s1"
  24.         }
  25.       }
  26.     ]
  27.   }
  28. }

copy

Example: Combining a match query and a term query

The following example request combines two query clauses—a match query and a term query. It specifies the search pipeline created in the previous step as a query parameter:

  1. GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline
  2. {
  3.   "_source": {
  4.     "exclude": [
  5.       "passage_embedding"
  6.     ]
  7.   },
  8.   "query": {
  9.     "hybrid": {
  10.       "queries": [
  11.            {
  12.              "match":{
  13.                  "passage_text": "hello"
  14.               }
  15.            },
  16.            {
  17.              "term":{
  18.               "passage_text":{
  19.                  "value":"planet"
  20.               }
  21.              }
  22.            }
  23.       ]
  24.     }
  25.   }
  26. }

copy

The response contains the matching documents:

  1. {
  2.     "took": 11,
  3.     "timed_out": false,
  4.     "_shards": {
  5.         "total": 2,
  6.         "successful": 2,
  7.         "skipped": 0,
  8.         "failed": 0
  9.     },
  10.     "hits": {
  11.         "total": {
  12.             "value": 2,
  13.             "relation": "eq"
  14.         },
  15.         "max_score": 0.7,
  16.         "hits": [
  17.             {
  18.                 "_index": "my-nlp-index",
  19.                 "_id": "2",
  20.                 "_score": 0.7,
  21.                 "_source": {
  22.                     "id": "s2",
  23.                     "passage_text": "Hi planet"
  24.                 }
  25.             },
  26.             {
  27.                 "_index": "my-nlp-index",
  28.                 "_id": "1",
  29.                 "_score": 0.3,
  30.                 "_source": {
  31.                     "id": "s1",
  32.                     "passage_text": "Hello world"
  33.                 }
  34.             }
  35.         ]
  36.     }
  37. }

copy

Hybrid search with post-filtering

Introduced 2.13

You can perform post-filtering on hybrid search results by providing the post_filter parameter in your query.

The post_filter clause is applied after the search results have been retrieved. Post-filtering is useful for applying additional filters to the search results without impacting the scoring or the order of the results.

Post-filtering does not impact document relevance scores or aggregation results.

Example: Post-filtering

The following example request combines two query clauses—a term query and a match query. This is the same query as in the preceding example, but it contains a post_filter:

  1. GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline
  2. {
  3.   "query": {
  4.     "hybrid":{
  5.       "queries":[
  6.         {
  7.           "match":{
  8.             "passage_text": "hello"
  9.           }
  10.         },
  11.         {
  12.           "term":{
  13.             "passage_text":{
  14.               "value":"planet"
  15.             }
  16.           }
  17.         }
  18.       ]
  19.     }
  21.   },
  22.   "post_filter":{
  23.     "match": { "passage_text": "world" }
  24.   }
  25. }

copy

Compare the results to the results without post-filtering in the preceding example. Unlike the preceding example response, which contains two documents, the response in this example contains one document because the second document is filtered using post-filtering:

  1. {
  2.   "took": 18,
  3.   "timed_out": false,
  4.   "_shards": {
  5.     "total": 2,
  6.     "successful": 2,
  7.     "skipped": 0,
  8.     "failed": 0
  9.   },
  10.   "hits": {
  11.     "total": {
  12.       "value": 1,
  13.       "relation": "eq"
  14.     },
  15.     "max_score": 0.3,
  16.     "hits": [
  17.       {
  18.         "_index": "my-nlp-index",
  19.         "_id": "1",
  20.         "_score": 0.3,
  21.         "_source": {
  22.           "id": "s1",
  23.           "passage_text": "Hello world"
  24.         }
  25.       }
  26.     ]
  27.   }
  28. }

Combining hybrid search and aggregations

Introduced 2.13

You can enhance search results by combining a hybrid query clause with any aggregation that OpenSearch supports. Aggregations allow you to use OpenSearch as an analytics engine. For more information about aggregations, see Aggregations.

Most aggregations are performed on the subset of documents that is returned by a hybrid query. The only aggregation that operates on all documents is the global aggregation.

To use aggregations with a hybrid query, first create an index. Aggregations are typically used on fields of special types, like keyword or integer. The following example creates an index with several such fields:

  1. PUT /my-nlp-index
  2. {
  3.   "settings": {
  4.     "number_of_shards": 2
  5.   },
  6.   "mappings": {
  7.     "properties": {
  8.       "doc_index": {
  9.         "type": "integer"
  10.       },
  11.       "doc_keyword": {
  12.         "type": "keyword"
  13.       },
  14.       "category": {
  15.         "type": "keyword"
  16.       }
  17.     }
  18.   }
  19. }

copy

The following request ingests six documents into your new index:

  1. POST /_bulk
  2. { "index": { "_index": "my-nlp-index" } }
  3. { "category": "permission", "doc_keyword": "workable", "doc_index": 4976, "doc_price": 100}
  4. { "index": { "_index": "my-nlp-index" } }
  5. { "category": "sister", "doc_keyword": "angry", "doc_index": 2231, "doc_price": 200 }
  6. { "index": { "_index": "my-nlp-index" } }
  7. { "category": "hair", "doc_keyword": "likeable", "doc_price": 25 }
  8. { "index": { "_index": "my-nlp-index" } }
  9. { "category": "editor", "doc_index": 9871, "doc_price": 30 }
  10. { "index": { "_index": "my-nlp-index" } }
  11. { "category": "statement", "doc_keyword": "entire", "doc_index": 8242, "doc_price": 350  } 
  12. { "index": { "_index": "my-nlp-index" } }
  13. { "category": "statement", "doc_keyword": "idea", "doc_index": 5212, "doc_price": 200  } 
  14. { "index": { "_index": "index-test" } }
  15. { "category": "editor", "doc_keyword": "bubble", "doc_index": 1298, "doc_price": 130 } 
  16. { "index": { "_index": "index-test" } }
  17. { "category": "editor", "doc_keyword": "bubble", "doc_index": 521, "doc_price": 75  }

copy

Now you can combine a hybrid query clause with a min aggregation:

  1. GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline
  2. {
  3.   "query": {
  4.     "hybrid": {
  5.       "queries": [
  6.         {
  7.           "term": {
  8.             "category": "permission"
  9.           }
  10.         },
  11.         {
  12.           "bool": {
  13.             "should": [
  14.               {
  15.                 "term": {
  16.                   "category": "editor"
  17.                 }
  18.               },
  19.               {
  20.                 "term": {
  21.                   "category": "statement"
  22.                 }
  23.               }
  24.             ]
  25.           }
  26.         }
  27.       ]
  28.     }
  29.   },
  30.   "aggs": {
  31.     "total_price": {
  32.       "sum": {
  33.         "field": "doc_price"
  34.       }
  35.     },
  36.     "keywords": {
  37.       "terms": {
  38.         "field": "doc_keyword",
  39.         "size": 10
  40.       }
  41.     }
  42.   }
  43. }

copy

The response contains the matching documents and the aggregation results:

  1. {
  2.   "took": 9,
  3.   "timed_out": false,
  4.   "_shards": {
  5.     "total": 2,
  6.     "successful": 2,
  7.     "skipped": 0,
  8.     "failed": 0
  9.   },
  10.   "hits": {
  11.     "total": {
  12.       "value": 4,
  13.       "relation": "eq"
  14.     },
  15.     "max_score": 0.5,
  16.     "hits": [
  17.       {
  18.         "_index": "my-nlp-index",
  19.         "_id": "mHRPNY4BlN82W_Ar9UMY",
  20.         "_score": 0.5,
  21.         "_source": {
  22.           "doc_price": 100,
  23.           "doc_index": 4976,
  24.           "doc_keyword": "workable",
  25.           "category": "permission"
  26.         }
  27.       },
  28.       {
  29.         "_index": "my-nlp-index",
  30.         "_id": "m3RPNY4BlN82W_Ar9UMY",
  31.         "_score": 0.5,
  32.         "_source": {
  33.           "doc_price": 30,
  34.           "doc_index": 9871,
  35.           "category": "editor"
  36.         }
  37.       },
  38.       {
  39.         "_index": "my-nlp-index",
  40.         "_id": "nXRPNY4BlN82W_Ar9UMY",
  41.         "_score": 0.5,
  42.         "_source": {
  43.           "doc_price": 200,
  44.           "doc_index": 5212,
  45.           "doc_keyword": "idea",
  46.           "category": "statement"
  47.         }
  48.       },
  49.       {
  50.         "_index": "my-nlp-index",
  51.         "_id": "nHRPNY4BlN82W_Ar9UMY",
  52.         "_score": 0.5,
  53.         "_source": {
  54.           "doc_price": 350,
  55.           "doc_index": 8242,
  56.           "doc_keyword": "entire",
  57.           "category": "statement"
  58.         }
  59.       }
  60.     ]
  61.   },
  62.   "aggregations": {
  63.     "total_price": {
  64.       "value": 680
  65.     },
  66.     "doc_keywords": {
  67.       "doc_count_error_upper_bound": 0,
  68.       "sum_other_doc_count": 0,
  69.       "buckets": [
  70.         {
  71.           "key": "entire",
  72.           "doc_count": 1
  73.         },
  74.         {
  75.           "key": "idea",
  76.           "doc_count": 1
  77.         },
  78.         {
  79.           "key": "workable",
  80.           "doc_count": 1
  81.         }
  82.       ]
  83.     }
  84.   }
  85. }