Search processors

Search processors can be of the following types:

Search request processors

A search request processor intercepts a search request (the query and the metadata passed in the request), performs an operation with or on the search request, and submits the search request to the index.

The following table lists all supported search request processors.

ProcessorDescriptionEarliest available version
filter_queryAdds a filtering query that is used to filter requests.2.8
ml_inferenceInvokes registered machine learning (ML) models in order to rewrite queries.2.16
neural_query_enricherSets a default model for neural search and neural sparse search at the index or field level.2.11 (neural), 2.13 (neural sparse)
neural_sparse_two_phase_processorAccelerates the neural sparse query.2.15
oversampleIncreases the search request size parameter, storing the original value in the pipeline state.2.12
scriptAdds a script that is run on newly indexed documents.2.8

Search response processors

A search response processor intercepts a search response and search request (the query, results, and metadata passed in the request), performs an operation with or on the search response, and returns the search response.

The following table lists all supported search response processors.

ProcessorDescriptionEarliest available version
collapseDeduplicates search hits based on a field value, similarly to collapse in a search request.2.12
ml_inferenceInvokes registered machine learning (ML) models in order to incorporate model output as additional search response fields.2.16
personalize_search_rankingUses Amazon Personalize to rerank search results (requires setting up the Amazon Personalize service).2.9
rename_fieldRenames an existing field.2.8
rerankReranks search results using a cross-encoder model.2.12
retrieval_augmented_generationUsed for retrieval-augmented generation (RAG) in conversational search.2.10 (generally available in 2.12)
sortSorts an array of items in either ascending or descending order.2.16
splitSplits a string field into an array of substrings based on a specified delimiter.2.16
truncate_hitsDiscards search hits after a specified target count is reached. Can undo the effect of the oversample request processor.2.12

Search phase results processors

A search phase results processor runs between search phases at the coordinating node level. It intercepts the results retrieved from one search phase and transforms them before passing them to the next search phase.

The following table lists all supported search phase results processors.

ProcessorDescriptionEarliest available version
normalization-processorIntercepts the query phase results and normalizes and combines the document scores before passing the documents to the fetch phase.2.10

Viewing available processor types

You can use the Nodes Search Pipelines API to view the available processor types:

  1. GET /_nodes/search_pipelines

copy

The response contains the search_pipelines object that lists the available request and response processors:

Response

  1. {
  2.   "_nodes" : {
  3.     "total" : 1,
  4.     "successful" : 1,
  5.     "failed" : 0
  6.   },
  7.   "cluster_name" : "runTask",
  8.   "nodes" : {
  9.     "36FHvCwHT6Srbm2ZniEPhA" : {
  10.       "name" : "runTask-0",
  11.       "transport_address" : "127.0.0.1:9300",
  12.       "host" : "127.0.0.1",
  13.       "ip" : "127.0.0.1",
  14.       "version" : "3.0.0",
  15.       "build_type" : "tar",
  16.       "build_hash" : "unknown",
  17.       "roles" : [
  18.         "cluster_manager",
  19.         "data",
  20.         "ingest",
  21.         "remote_cluster_client"
  22.       ],
  23.       "attributes" : {
  24.         "testattr" : "test",
  25.         "shard_indexing_pressure_enabled" : "true"
  26.       },
  27.       "search_pipelines" : {
  28.         "request_processors" : [
  29.           {
  30.             "type" : "filter_query"
  31.           },
  32.           {
  33.             "type" : "script"
  34.           }
  35.         ],
  36.         "response_processors" : [
  37.           {
  38.             "type" : "rename_field"
  39.           }
  40.         ]
  41.       }
  42.     }
  43.   }
  44. }

In addition to the processors provided by OpenSearch, additional processors may be provided by plugins.

Selectively enabling processors

Processors defined by the search-pipeline-common module are selectively enabled through the following cluster settings: search.pipeline.common.request.processors.allowed, search.pipeline.common.response.processors.allowed, or search.pipeline.common.search.phase.results.processors.allowed. If unspecified, then all processors are enabled. An empty list disables all processors. Removing enabled processors causes pipelines using them to fail after a node restart.