Text chunking

Introduced 2.13

To split long text into passages, you can use a text_chunking processor as a preprocessing step for a text_embedding or sparse_encoding processor in order to obtain embeddings for each chunked passage. For more information about the processor parameters, see Text chunking processor. Before you start, follow the steps outlined in the pretrained model documentation to register an embedding model. The following example preprocesses text by splitting it into passages and then produces embeddings using the text_embedding processor.

Step 1: Create a pipeline

The following example request creates an ingest pipeline that converts the text in the passage_text field into chunked passages, which will be stored in the passage_chunk field. The text in the passage_chunk field is then converted into text embeddings, and the embeddings are stored in the passage_embedding field:

  1. PUT _ingest/pipeline/text-chunking-embedding-ingest-pipeline
  2. {
  3.   "description": "A text chunking and embedding ingest pipeline",
  4.   "processors": [
  5.     {
  6.       "text_chunking": {
  7.         "algorithm": {
  8.           "fixed_token_length": {
  9.             "token_limit": 10,
  10.             "overlap_rate": 0.2,
  11.             "tokenizer": "standard"
  12.           }
  13.         },
  14.         "field_map": {
  15.           "passage_text": "passage_chunk"
  16.         }
  17.       }
  18.     },
  19.     {
  20.       "text_embedding": {
  21.         "model_id": "LMLPWY4BROvhdbtgETaI",
  22.         "field_map": {
  23.           "passage_chunk": "passage_chunk_embedding"
  24.         }
  25.       }
  26.     }
  27.   ]
  28. }

copy

Step 2: Create an index for ingestion

In order to use the ingest pipeline, you need to create a k-NN index. The passage_chunk_embedding field must be of the nested type. The knn.dimension field must contain the number of dimensions for your model:

  1. PUT testindex
  2. {
  3.   "settings": {
  4.     "index": {
  5.       "knn": true
  6.     }
  7.   },
  8.   "mappings": {
  9.     "properties": {
  10.       "text": {
  11.         "type": "text"
  12.       },
  13.       "passage_chunk_embedding": {
  14.         "type": "nested",
  15.         "properties": {
  16.           "knn": {
  17.             "type": "knn_vector",
  18.             "dimension": 768
  19.           }
  20.         }
  21.       }
  22.     }
  23.   }
  24. }

copy

Step 3: Ingest documents into the index

To ingest a document into the index created in the previous step, send the following request:

  1. POST testindex/_doc?pipeline=text-chunking-embedding-ingest-pipeline
  2. {
  3.   "passage_text": "This is an example document to be chunked. The document contains a single paragraph, two sentences and 24 tokens by standard tokenizer in OpenSearch."
  4. }

copy

You can use a nested query to perform vector search on your index. We recommend setting score_mode to max, where the document score is set to the highest score out of all passage embeddings:

  1. GET testindex/_search
  2. {
  3.   "query": {
  4.     "nested": {
  5.       "score_mode": "max",
  6.       "path": "passage_chunk_embedding",
  7.       "query": {
  8.         "neural": {
  9.           "passage_chunk_embedding.knn": {
  10.             "query_text": "document",
  11.             "model_id": "-tHZeI4BdQKclr136Wl7"
  12.           }
  13.         }
  14.       }
  15.     }
  16.   }
  17. }

copy