Truncate hits processor

Introduced 2.12

The truncate_hits response processor discards returned search hits after a given hit count is reached. The truncate_hits processor is designed to work with the oversample request processor but may be used on its own.

The target_size parameter (which specifies where to truncate) is optional. If it is not specified, then OpenSearch uses the original_size variable set by the oversample processor (if available).

The following is a common usage pattern:

  1. Add the oversample processor to a request pipeline to fetch a larger set of results.
  2. In the response pipeline, apply a reranking processor (which may promote results from beyond the originally requested top N) or the collapse processor (which may discard results after deduplication).
  3. Apply the truncate processor to return (at most) the originally requested number of hits.

Request fields

The following table lists all request fields.

FieldData typeDescription
target_sizeIntegerThe maximum number of search hits to return (>=0). If not specified, the processor will try to read the original_size variable and will fail if it is not available. Optional.
context_prefixStringMay be used to read the original_size variable from a specific scope in order to avoid collisions. Optional.
tagStringThe processor’s identifier. Optional.
descriptionStringA description of the processor. Optional.
ignore_failureBooleanIf true, OpenSearch ignores any failure of this processor and continues to run the remaining processors in the search pipeline. Optional. Default is false.

Example

The following example demonstrates using a search pipeline with a truncate processor.

Setup

Create an index named my_index containing many documents:

  1. POST /_bulk
  2. { "create":{"_index":"my_index","_id":1}}
  3. { "doc": { "title" : "document 1" }}
  4. { "create":{"_index":"my_index","_id":2}}
  5. { "doc": { "title" : "document 2" }}
  6. { "create":{"_index":"my_index","_id":3}}
  7. { "doc": { "title" : "document 3" }}
  8. { "create":{"_index":"my_index","_id":4}}
  9. { "doc": { "title" : "document 4" }}
  10. { "create":{"_index":"my_index","_id":5}}
  11. { "doc": { "title" : "document 5" }}
  12. { "create":{"_index":"my_index","_id":6}}
  13. { "doc": { "title" : "document 6" }}
  14. { "create":{"_index":"my_index","_id":7}}
  15. { "doc": { "title" : "document 7" }}
  16. { "create":{"_index":"my_index","_id":8}}
  17. { "doc": { "title" : "document 8" }}
  18. { "create":{"_index":"my_index","_id":9}}
  19. { "doc": { "title" : "document 9" }}
  20. { "create":{"_index":"my_index","_id":10}}
  21. { "doc": { "title" : "document 10" }}

copy

Creating a search pipeline

The following request creates a search pipeline named my_pipeline with a truncate_hits response processor that discards hits after the first five:

  1. PUT /_search/pipeline/my_pipeline 
  2. {
  3.   "response_processors": [
  4.     {
  5.       "truncate_hits" : {
  6.         "tag" : "truncate_1",
  7.         "description" : "This processor will discard results after the first 5.",
  8.         "target_size" : 5
  9.       }
  10.     }
  11.   ]
  12. }

copy

Using a search pipeline

Search for documents in my_index without a search pipeline:

  1. POST /my_index/_search
  2. {
  3.   "size": 8
  4. }

copy

The response contains eight hits:

Response

  1. {
  2.   "took" : 13,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 10,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 1.0,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my_index",
  19.         "_id" : "1",
  20.         "_score" : 1.0,
  21.         "_source" : {
  22.           "doc" : {
  23.             "title" : "document 1"
  24.           }
  25.         }
  26.       },
  27.       {
  28.         "_index" : "my_index",
  29.         "_id" : "2",
  30.         "_score" : 1.0,
  31.         "_source" : {
  32.           "doc" : {
  33.             "title" : "document 2"
  34.           }
  35.         }
  36.       },
  37.       {
  38.         "_index" : "my_index",
  39.         "_id" : "3",
  40.         "_score" : 1.0,
  41.         "_source" : {
  42.           "doc" : {
  43.             "title" : "document 3"
  44.           }
  45.         }
  46.       },
  47.       {
  48.         "_index" : "my_index",
  49.         "_id" : "4",
  50.         "_score" : 1.0,
  51.         "_source" : {
  52.           "doc" : {
  53.             "title" : "document 4"
  54.           }
  55.         }
  56.       },
  57.       {
  58.         "_index" : "my_index",
  59.         "_id" : "5",
  60.         "_score" : 1.0,
  61.         "_source" : {
  62.           "doc" : {
  63.             "title" : "document 5"
  64.           }
  65.         }
  66.       },
  67.       {
  68.         "_index" : "my_index",
  69.         "_id" : "6",
  70.         "_score" : 1.0,
  71.         "_source" : {
  72.           "doc" : {
  73.             "title" : "document 6"
  74.           }
  75.         }
  76.       },
  77.       {
  78.         "_index" : "my_index",
  79.         "_id" : "7",
  80.         "_score" : 1.0,
  81.         "_source" : {
  82.           "doc" : {
  83.             "title" : "document 7"
  84.           }
  85.         }
  86.       },
  87.       {
  88.         "_index" : "my_index",
  89.         "_id" : "8",
  90.         "_score" : 1.0,
  91.         "_source" : {
  92.           "doc" : {
  93.             "title" : "document 8"
  94.           }
  95.         }
  96.       }
  97.     ]
  98.   }
  99. }

To search with a pipeline, specify the pipeline name in the search_pipeline query parameter:

  1. POST /my_index/_search?search_pipeline=my_pipeline
  2. {
  3.   "size": 8
  4. }

copy

The response contains only 5 hits, even though 8 were requested and 10 were available:

Response

  1. {
  2.   "took" : 3,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 10,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 1.0,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my_index",
  19.         "_id" : "1",
  20.         "_score" : 1.0,
  21.         "_source" : {
  22.           "doc" : {
  23.             "title" : "document 1"
  24.           }
  25.         }
  26.       },
  27.       {
  28.         "_index" : "my_index",
  29.         "_id" : "2",
  30.         "_score" : 1.0,
  31.         "_source" : {
  32.           "doc" : {
  33.             "title" : "document 2"
  34.           }
  35.         }
  36.       },
  37.       {
  38.         "_index" : "my_index",
  39.         "_id" : "3",
  40.         "_score" : 1.0,
  41.         "_source" : {
  42.           "doc" : {
  43.             "title" : "document 3"
  44.           }
  45.         }
  46.       },
  47.       {
  48.         "_index" : "my_index",
  49.         "_id" : "4",
  50.         "_score" : 1.0,
  51.         "_source" : {
  52.           "doc" : {
  53.             "title" : "document 4"
  54.           }
  55.         }
  56.       },
  57.       {
  58.         "_index" : "my_index",
  59.         "_id" : "5",
  60.         "_score" : 1.0,
  61.         "_source" : {
  62.           "doc" : {
  63.             "title" : "document 5"
  64.           }
  65.         }
  66.       }
  67.     ]
  68.   }
  69. }

Oversample, collapse, and truncate hits

The following is a more realistic example in which you use oversample to request many candidate documents, use collapse to remove documents that duplicate a particular field (to get more diverse results), and then use truncate to return the originally requested document count (to avoid returning a large result payload from the cluster).

Setup

Create many documents containing a field that you’ll use for collapsing:

  1. POST /_bulk
  2. { "create":{"_index":"my_index","_id":1}}
  3. { "title" : "document 1", "color":"blue" }
  4. { "create":{"_index":"my_index","_id":2}}
  5. { "title" : "document 2", "color":"blue" }
  6. { "create":{"_index":"my_index","_id":3}}
  7. { "title" : "document 3", "color":"red" }
  8. { "create":{"_index":"my_index","_id":4}}
  9. { "title" : "document 4", "color":"red" }
  10. { "create":{"_index":"my_index","_id":5}}
  11. { "title" : "document 5", "color":"yellow" }
  12. { "create":{"_index":"my_index","_id":6}}
  13. { "title" : "document 6", "color":"yellow" }
  14. { "create":{"_index":"my_index","_id":7}}
  15. { "title" : "document 7", "color":"orange" }
  16. { "create":{"_index":"my_index","_id":8}}
  17. { "title" : "document 8", "color":"orange" }
  18. { "create":{"_index":"my_index","_id":9}}
  19. { "title" : "document 9", "color":"green" }
  20. { "create":{"_index":"my_index","_id":10}}
  21. { "title" : "document 10", "color":"green" }

copy

Create a pipeline that collapses only on the color field:

  1. PUT /_search/pipeline/collapse_pipeline
  2. {
  3.   "response_processors": [
  4.     {
  5.       "collapse" : {
  6.         "field": "color"
  7.       }
  8.     }
  9.   ]
  10. }

copy

Create another pipeline that oversamples, collapses, and then truncates results:

  1. PUT /_search/pipeline/oversampling_collapse_pipeline
  2. {
  3.   "request_processors": [
  4.     {
  5.       "oversample": {
  6.         "sample_factor": 3
  7.       }
  8.     }
  9.   ],
  10.   "response_processors": [
  11.     {
  12.       "collapse" : {
  13.         "field": "color"
  14.       }
  15.     },
  16.     {
  17.       "truncate_hits": {
  18.         "description": "Truncates back to the original size before oversample increased it."
  19.       }
  20.     }
  21.   ]
  22. }

copy

Collapse without oversample

In this example, you request the top three documents before collapsing on the color field. Because the first two documents have the same color, the second one is discarded, and the request returns the first and third documents:

  1. POST /my_index/_search?search_pipeline=collapse_pipeline
  2. {
  3.   "size": 3
  4. }

copy

Response

  1. {
  2.   "took" : 2,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 10,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 1.0,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my_index",
  19.         "_id" : "1",
  20.         "_score" : 1.0,
  21.         "_source" : {
  22.           "title" : "document 1",
  23.           "color" : "blue"
  24.         }
  25.       },
  26.       {
  27.         "_index" : "my_index",
  28.         "_id" : "3",
  29.         "_score" : 1.0,
  30.         "_source" : {
  31.           "title" : "document 3",
  32.           "color" : "red"
  33.         }
  34.       }
  35.     ]
  36.   },
  37.   "profile" : {
  38.     "shards" : [ ]
  39.   }
  40. }

Oversample, collapse, and truncate

Now you will use the oversampling_collapse_pipeline, which requests the top 9 documents (multiplying the size by 3), deduplicates by color, and then returns the top 3 hits:

  1. POST /my_index/_search?search_pipeline=oversampling_collapse_pipeline
  2. {
  3.   "size": 3
  4. }

copy

Response

  1. {
  2.   "took" : 2,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 10,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 1.0,
  16.     "hits" : [
  17.       {
  18.         "_index" : "my_index",
  19.         "_id" : "1",
  20.         "_score" : 1.0,
  21.         "_source" : {
  22.           "title" : "document 1",
  23.           "color" : "blue"
  24.         }
  25.       },
  26.       {
  27.         "_index" : "my_index",
  28.         "_id" : "3",
  29.         "_score" : 1.0,
  30.         "_source" : {
  31.           "title" : "document 3",
  32.           "color" : "red"
  33.         }
  34.       },
  35.       {
  36.         "_index" : "my_index",
  37.         "_id" : "5",
  38.         "_score" : 1.0,
  39.         "_source" : {
  40.           "title" : "document 5",
  41.           "color" : "yellow"
  42.         }
  43.       }
  44.     ]
  45.   },
  46.   "profile" : {
  47.     "shards" : [ ]
  48.   }
  49. }