Semantic search

Semantic search considers the context and intent of a query. In OpenSearch, semantic search is facilitated by neural search with text embedding models. Semantic search creates a dense vector (a list of floats) and ingests data into a k-NN index.

PREREQUISITE
Before using semantic search, you must set up a text embedding model. For more information, see Choosing a model.

To use semantic search, follow these steps:

  1. Create an ingest pipeline.
  2. Create an index for ingestion.
  3. Ingest documents into the index.
  4. Search the index using neural search.

Step 1: Create an ingest pipeline

To generate vector embeddings, you need to create an ingest pipeline that contains a text_embedding processor, which will convert the text in a document field to vector embeddings. The processor’s field_map determines the input fields from which to generate vector embeddings and the output fields in which to store the embeddings.

The following example request creates an ingest pipeline where the text from passage_text will be converted into text embeddings and the embeddings will be stored in passage_embedding:

  1. PUT /_ingest/pipeline/nlp-ingest-pipeline
  2. {
  3.   "description": "A text embedding pipeline",
  4.   "processors": [
  5.     {
  6.       "text_embedding": {
  7.         "model_id": "bQ1J8ooBpBj3wT4HVUsb",
  8.         "field_map": {
  9.           "passage_text": "passage_embedding"
  10.         }
  11.       }
  12.     }
  13.   ]
  14. }

copy

Step 2: Create an index for ingestion

In order to use the text embedding processor defined in your pipeline, create a k-NN index, adding the pipeline created in the previous step as the default pipeline. Ensure that the fields defined in the field_map are mapped as correct types. Continuing with the example, the passage_embedding field must be mapped as a k-NN vector with a dimension that matches the model dimension. Similarly, the passage_text field should be mapped as text.

The following example request creates a k-NN index that is set up with a default ingest pipeline:

  1. PUT /my-nlp-index
  2. {
  3.   "settings": {
  4.     "index.knn": true,
  5.     "default_pipeline": "nlp-ingest-pipeline"
  6.   },
  7.   "mappings": {
  8.     "properties": {
  9.       "id": {
  10.         "type": "text"
  11.       },
  12.       "passage_embedding": {
  13.         "type": "knn_vector",
  14.         "dimension": 768,
  15.         "method": {
  16.           "engine": "lucene",
  17.           "space_type": "l2",
  18.           "name": "hnsw",
  19.           "parameters": {}
  20.         }
  21.       },
  22.       "passage_text": {
  23.         "type": "text"
  24.       }
  25.     }
  26.   }
  27. }

copy

For more information about creating a k-NN index and its supported methods, see k-NN index.

Step 3: Ingest documents into the index

To ingest documents into the index created in the previous step, send the following requests:

  1. PUT /my-nlp-index/_doc/1
  2. {
  3.   "passage_text": "Hello world",
  4.   "id": "s1"
  5. }

copy

  1. PUT /my-nlp-index/_doc/2
  2. {
  3.   "passage_text": "Hi planet",
  4.   "id": "s2"
  5. }

copy

Before the document is ingested into the index, the ingest pipeline runs the text_embedding processor on the document, generating text embeddings for the passage_text field. The indexed document includes the passage_text field, which contains the original text, and the passage_embedding field, which contains the vector embeddings.

To perform vector search on your index, use the neural query clause either in the k-NN plugin API or Query DSL queries. You can refine the results by using a k-NN search filter.

The following example request uses a Boolean query to combine a filter clause and two query clauses—a neural query and a match query. The script_score query assigns custom weights to the query clauses:

  1. GET /my-nlp-index/_search
  2. {
  3.   "_source": {
  4.     "excludes": [
  5.       "passage_embedding"
  6.     ]
  7.   },
  8.   "query": {
  9.     "bool": {
  10.       "filter": {
  11.          "wildcard":  { "id": "*1" }
  12.       },
  13.       "should": [
  14.         {
  15.           "script_score": {
  16.             "query": {
  17.               "neural": {
  18.                 "passage_embedding": {
  19.                   "query_text": "Hi world",
  20.                   "model_id": "bQ1J8ooBpBj3wT4HVUsb",
  21.                   "k": 100
  22.                 }
  23.               }
  24.             },
  25.             "script": {
  26.               "source": "_score * 1.5"
  27.             }
  28.           }
  29.         },
  30.         {
  31.           "script_score": {
  32.             "query": {
  33.               "match": {
  34.                 "passage_text": "Hi world"
  35.               }
  36.             },
  37.             "script": {
  38.               "source": "_score * 1.7"
  39.             }
  40.           }
  41.         }
  42.       ]
  43.     }
  44.   }
  45. }

copy

The response contains the matching document:

  1. {
  2.   "took" : 36,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 1,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 1.2251667,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my-nlp-index",
  19.         "_id" : "1",
  20.         "_score" : 1.2251667,
  21.         "_source" : {
  22.           "passage_text" : "Hello world",
  23.           "id" : "s1"
  24.         }
  25.       }
  26.     ]
  27.   }
  28. }

Setting a default model on an index or field

A neural query requires a model ID for generating vector embeddings. To eliminate passing the model ID with each neural query request, you can set a default model on a k-NN index or a field.

First, create a search pipeline with a neural_query_enricher request processor. To set a default model for an index, provide the model ID in the default_model_id parameter. To set a default model for a specific field, provide the field name and the corresponding model ID in the neural_field_default_id map. If you provide both default_model_id and neural_field_default_id, neural_field_default_id takes precedence:

  1. PUT /_search/pipeline/default_model_pipeline 
  2. {
  3.   "request_processors": [
  4.     {
  5.       "neural_query_enricher" : {
  6.         "default_model_id": "bQ1J8ooBpBj3wT4HVUsb",
  7.         "neural_field_default_id": {
  8.            "my_field_1": "uZj0qYoBMtvQlfhaYeud",
  9.            "my_field_2": "upj0qYoBMtvQlfhaZOuM"
  10.         }
  11.       }
  12.     }
  13.   ]
  14. }

copy

Then set the default model for your index:

  1. PUT /my-nlp-index/_settings
  2. {
  3.   "index.search.default_pipeline" : "default_model_pipeline"
  4. }

copy

You can now omit the model ID when searching:

  1. GET /my-nlp-index/_search
  2. {
  3.   "_source": {
  4.     "excludes": [
  5.       "passage_embedding"
  6.     ]
  7.   },
  8.   "query": {
  9.     "neural": {
  10.       "passage_embedding": {
  11.         "query_text": "Hi world",
  12.         "k": 100
  13.       }
  14.     }
  15.   }
  16. }

copy

The response contains both documents:

  1. {
  2.   "took" : 41,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 2,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 1.22762,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my-nlp-index",
  19.         "_id" : "2",
  20.         "_score" : 1.22762,
  21.         "_source" : {
  22.           "passage_text" : "Hi planet",
  23.           "id" : "s2"
  24.         }
  25.       },
  26.       {
  27.         "_index" : "my-nlp-index",
  28.         "_id" : "1",
  29.         "_score" : 1.2251667,
  30.         "_source" : {
  31.           "passage_text" : "Hello world",
  32.           "id" : "s1"
  33.         }
  34.       }
  35.     ]
  36.   }
  37. }