Neural sparse search

Introduced 2.11

Semantic search relies on dense retrieval that is based on text embedding models. However, dense methods use k-NN search, which consumes a large amount of memory and CPU resources. An alternative to semantic search, neural sparse search is implemented using an inverted index and is thus as efficient as BM25. Neural sparse search is facilitated by sparse embedding models. When you perform a neural sparse search, it creates a sparse vector (a list of token: weight key-value pairs representing an entry and its weight) and ingests data into a rank features index.

When selecting a model, choose one of the following options:

  • Use a sparse encoding model at both ingestion time and search time for better search relevance at the expense of relatively high latency.
  • Use a sparse encoding model at ingestion time and a tokenizer at search time for lower search latency at the expense of relatively lower search relevance. Tokenization doesn’t involve model inference, so you can deploy and invoke a tokenizer using the ML Commons Model API for a more streamlined experience.

PREREQUISITE
Before using neural sparse search, make sure to set up a pretrained sparse embedding model or your own sparse embedding model. For more information, see Choosing a model.

To use neural sparse search, follow these steps:

  1. Create an ingest pipeline.
  2. Create an index for ingestion.
  3. Ingest documents into the index.
  4. Search the index using neural search.
  5. Optional Create and enable the two-phase processor.

Step 1: Create an ingest pipeline

To generate vector embeddings, you need to create an ingest pipeline that contains a sparse_encoding processor, which will convert the text in a document field to vector embeddings. The processor’s field_map determines the input fields from which to generate vector embeddings and the output fields in which to store the embeddings.

The following example request creates an ingest pipeline where the text from passage_text will be converted into text embeddings and the embeddings will be stored in passage_embedding:

  1. PUT /_ingest/pipeline/nlp-ingest-pipeline-sparse
  2. {
  3.   "description": "An sparse encoding ingest pipeline",
  4.   "processors": [
  5.     {
  6.       "sparse_encoding": {
  7.         "model_id": "aP2Q8ooBpBj3wT4HVS8a",
  8.         "field_map": {
  9.           "passage_text": "passage_embedding"
  10.         }
  11.       }
  12.     }
  13.   ]
  14. }

copy

To split long text into passages, use the text_chunking ingest processor before the sparse_encoding processor. For more information, see Text chunking.

Step 2: Create an index for ingestion

In order to use the text embedding processor defined in your pipeline, create a rank features index, adding the pipeline created in the previous step as the default pipeline. Ensure that the fields defined in the field_map are mapped as correct types. Continuing with the example, the passage_embedding field must be mapped as rank_features. Similarly, the passage_text field should be mapped as text.

The following example request creates a rank features index that is set up with a default ingest pipeline:

  1. PUT /my-nlp-index
  2. {
  3.   "settings": {
  4.     "default_pipeline": "nlp-ingest-pipeline-sparse"
  5.   },
  6.   "mappings": {
  7.     "properties": {
  8.       "id": {
  9.         "type": "text"
  10.       },
  11.       "passage_embedding": {
  12.         "type": "rank_features"
  13.       },
  14.       "passage_text": {
  15.         "type": "text"
  16.       }
  17.     }
  18.   }
  19. }

copy

To save disk space, you can exclude the embedding vector from the source as follows:

  1. PUT /my-nlp-index
  2. {
  3.   "settings": {
  4.     "default_pipeline": "nlp-ingest-pipeline-sparse"
  5.   },
  6.   "mappings": {
  7.       "_source": {
  8.       "excludes": [
  9.         "passage_embedding"
  10.       ]
  11.     },
  12.     "properties": {
  13.       "id": {
  14.         "type": "text"
  15.       },
  16.       "passage_embedding": {
  17.         "type": "rank_features"
  18.       },
  19.       "passage_text": {
  20.         "type": "text"
  21.       }
  22.     }
  23.   }
  24. }

copy

Once the <token, weight> pairs are excluded from the source, they cannot be recovered. Before applying this optimization, make sure you don’t need the <token, weight> pairs for your application.

Step 3: Ingest documents into the index

To ingest documents into the index created in the previous step, send the following requests:

  1. PUT /my-nlp-index/_doc/1
  2. {
  3.   "passage_text": "Hello world",
  4.   "id": "s1"
  5. }

copy

  1. PUT /my-nlp-index/_doc/2
  2. {
  3.   "passage_text": "Hi planet",
  4.   "id": "s2"
  5. }

copy

Before the document is ingested into the index, the ingest pipeline runs the sparse_encoding processor on the document, generating vector embeddings for the passage_text field. The indexed document includes the passage_text field, which contains the original text, and the passage_embedding field, which contains the vector embeddings.

To perform a neural sparse search on your index, use the neural_sparse query clause in Query DSL queries.

The following example request uses a neural_sparse query to search for relevant documents using a raw text query:

  1. GET my-nlp-index/_search
  2. {
  3.   "query": {
  4.     "neural_sparse": {
  5.       "passage_embedding": {
  6.         "query_text": "Hi world",
  7.         "model_id": "aP2Q8ooBpBj3wT4HVS8a"
  8.       }
  9.     }
  10.   }
  11. }

copy

The response contains the matching documents:

  1. {
  2.   "took" : 688,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 2,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 30.0029,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my-nlp-index",
  19.         "_id" : "1",
  20.         "_score" : 30.0029,
  21.         "_source" : {
  22.           "passage_text" : "Hello world",
  23.           "passage_embedding" : {
  24.             "!" : 0.8708904,
  25.             "door" : 0.8587369,
  26.             "hi" : 2.3929274,
  27.             "worlds" : 2.7839446,
  28.             "yes" : 0.75845814,
  29.             "##world" : 2.5432441,
  30.             "born" : 0.2682308,
  31.             "nothing" : 0.8625516,
  32.             "goodbye" : 0.17146169,
  33.             "greeting" : 0.96817183,
  34.             "birth" : 1.2788506,
  35.             "come" : 0.1623208,
  36.             "global" : 0.4371151,
  37.             "it" : 0.42951578,
  38.             "life" : 1.5750692,
  39.             "thanks" : 0.26481047,
  40.             "world" : 4.7300377,
  41.             "tiny" : 0.5462298,
  42.             "earth" : 2.6555297,
  43.             "universe" : 2.0308156,
  44.             "worldwide" : 1.3903781,
  45.             "hello" : 6.696973,
  46.             "so" : 0.20279501,
  47.             "?" : 0.67785245
  48.           },
  49.           "id" : "s1"
  50.         }
  51.       },
  52.       {
  53.         "_index" : "my-nlp-index",
  54.         "_id" : "2",
  55.         "_score" : 16.480486,
  56.         "_source" : {
  57.           "passage_text" : "Hi planet",
  58.           "passage_embedding" : {
  59.             "hi" : 4.338913,
  60.             "planets" : 2.7755864,
  61.             "planet" : 5.0969057,
  62.             "mars" : 1.7405145,
  63.             "earth" : 2.6087382,
  64.             "hello" : 3.3210192
  65.           },
  66.           "id" : "s2"
  67.         }
  68.       }
  69.     ]
  70.   }
  71. }

You can also use the neural_sparse query with sparse vector embeddings:

  1. GET my-nlp-index/_search
  2. {
  3.   "query": {
  4.     "neural_sparse": {
  5.       "passage_embedding": {
  6.         "query_tokens": {
  7.           "hi" : 4.338913,
  8.           "planets" : 2.7755864,
  9.           "planet" : 5.0969057,
  10.           "mars" : 1.7405145,
  11.           "earth" : 2.6087382,
  12.           "hello" : 3.3210192
  13.         }
  14.       }
  15.     }
  16.   }
  17. }

Step 5: Create and enable the two-phase processor (Optional)

The neural_sparse_two_phase_processor is a new feature introduced in OpenSearch 2.15. Using the two-phase processor can significantly improve the performance of neural sparse queries.

To quickly launch a search pipeline with neural sparse search, use the following example pipeline:

  1. PUT /_search/pipeline/two_phase_search_pipeline
  2. {
  3.   "request_processors": [
  4.     {
  5.       "neural_sparse_two_phase_processor": {
  6.         "tag": "neural-sparse",
  7.         "description": "This processor is making two-phase processor."
  8.       }
  9.     }
  10.   ]
  11. }

copy

Then choose the index you want to configure with the search pipeline and set the index.search.default_pipeline to the pipeline name, as shown in the following example:

  1. PUT /index-name/_settings 
  2. {
  3.   "index.search.default_pipeline" : "two_phase_search_pipeline"
  4. }

copy

Setting a default model on an index or field

A neural_sparse query requires a model ID for generating sparse embeddings. To eliminate passing the model ID with each neural_sparse query request, you can set a default model on index-level or field-level.

First, create a search pipeline with a neural_query_enricher request processor. To set a default model for an index, provide the model ID in the default_model_id parameter. To set a default model for a specific field, provide the field name and the corresponding model ID in the neural_field_default_id map. If you provide both default_model_id and neural_field_default_id, neural_field_default_id takes precedence:

  1. PUT /_search/pipeline/default_model_pipeline 
  2. {
  3.   "request_processors": [
  4.     {
  5.       "neural_query_enricher" : {
  6.         "default_model_id": "bQ1J8ooBpBj3wT4HVUsb",
  7.         "neural_field_default_id": {
  8.            "my_field_1": "uZj0qYoBMtvQlfhaYeud",
  9.            "my_field_2": "upj0qYoBMtvQlfhaZOuM"
  10.         }
  11.       }
  12.     }
  13.   ]
  14. }

copy

Then set the default model for your index:

  1. PUT /my-nlp-index/_settings
  2. {
  3.   "index.search.default_pipeline" : "default_model_pipeline"
  4. }

copy

You can now omit the model ID when searching:

  1. GET /my-nlp-index/_search
  2. {
  3.   "query": {
  4.     "neural_sparse": {
  5.       "passage_embedding": {
  6.         "query_text": "Hi world"
  7.       }
  8.     }
  9.   }
  10. }

copy

The response contains both documents:

  1. {
  2.   "took" : 688,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 2,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 30.0029,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my-nlp-index",
  19.         "_id" : "1",
  20.         "_score" : 30.0029,
  21.         "_source" : {
  22.           "passage_text" : "Hello world",
  23.           "passage_embedding" : {
  24.             "!" : 0.8708904,
  25.             "door" : 0.8587369,
  26.             "hi" : 2.3929274,
  27.             "worlds" : 2.7839446,
  28.             "yes" : 0.75845814,
  29.             "##world" : 2.5432441,
  30.             "born" : 0.2682308,
  31.             "nothing" : 0.8625516,
  32.             "goodbye" : 0.17146169,
  33.             "greeting" : 0.96817183,
  34.             "birth" : 1.2788506,
  35.             "come" : 0.1623208,
  36.             "global" : 0.4371151,
  37.             "it" : 0.42951578,
  38.             "life" : 1.5750692,
  39.             "thanks" : 0.26481047,
  40.             "world" : 4.7300377,
  41.             "tiny" : 0.5462298,
  42.             "earth" : 2.6555297,
  43.             "universe" : 2.0308156,
  44.             "worldwide" : 1.3903781,
  45.             "hello" : 6.696973,
  46.             "so" : 0.20279501,
  47.             "?" : 0.67785245
  48.           },
  49.           "id" : "s1"
  50.         }
  51.       },
  52.       {
  53.         "_index" : "my-nlp-index",
  54.         "_id" : "2",
  55.         "_score" : 16.480486,
  56.         "_source" : {
  57.           "passage_text" : "Hi planet",
  58.           "passage_embedding" : {
  59.             "hi" : 4.338913,
  60.             "planets" : 2.7755864,
  61.             "planet" : 5.0969057,
  62.             "mars" : 1.7405145,
  63.             "earth" : 2.6087382,
  64.             "hello" : 3.3210192
  65.           },
  66.           "id" : "s2"
  67.         }
  68.       }
  69.     ]
  70.   }
  71. }

Next steps

  • To learn more about splitting long text into passages for neural search, see Text chunking.

FAQ

Refer to the following frequently asked questions for more information about neural sparse search.

How do I mitigate remote connector throttling exceptions?

When using connectors to call a remote service like SageMaker, ingestion and search calls sometimes fail due to remote connector throttling exceptions.

To mitigate throttling exceptions, modify the connector’s client_config parameter to decrease the number of maximum connections, using the max_connection setting to prevent the maximum number of concurrent connections from exceeding the threshold of the remote service. You can also modify the retry settings to flatten the request spike during ingestion.

For versions earlier than OpenSearch 2.15, the SageMaker throttling exception will be thrown as the following “error”:

  1.    {
  2.           "type": "status_exception",
  3.           "reason": "Error from remote service: {\"message\":null}"
  4.         }

Next steps

  • To learn more about splitting long text into passages for neural search, see Text chunking.