Pipeline processor

The pipeline processor allows a pipeline to reference and include another predefined pipeline. This can be useful when you have a set of common processors that need to be shared across multiple pipelines. Instead of redefining those common processors in each pipeline, you can create a separate base pipeline containing the shared processors and then reference that base pipeline from other pipelines using the pipeline processor.

The following is the syntax for the pipeline processor:

  1. {
  2.   "pipeline": {
  3.     "name": "general-pipeline"
  4.   }
  5. }

copy

Configuration parameters

The following table lists the required and optional parameters for the pipeline processor.

ParameterRequired/OptionalDescription
nameRequiredThe name of the pipeline to execute.
descriptionOptionalA description of the processor’s purpose or configuration.
ifOptionalSpecifies to conditionally execute the processor.
ignore_failureOptionalSpecifies to ignore processor failures. See Handling pipeline failures.
on_failureOptionalSpecifies to handle processor failures. See Handling pipeline failures.
tagOptionalAn identifier for the processor. Useful for debugging and metrics.

Using the processor

Follow these steps to use the processor in a pipeline.

Step 1: Create a pipeline

The following query creates a general pipeline named general-pipeline and then creates a new pipeline named outer-pipeline, which references the general-pipeline:

  1. PUT _ingest/pipeline/general_pipeline  
  2. {  
  3.   "description": "a general pipeline",  
  4.   "processors": [  
  5.     {  
  6.       "uppercase": {  
  7.         "field": "protocol"  
  8.       },  
  9.       "remove": {  
  10.         "field": "name"  
  11.       }  
  12.     }  
  13.   ]  
  14. }

copy

  1. PUT _ingest/pipeline/outer-pipeline  
  2. {  
  3.   "description": "an outer pipeline referencing the general pipeline",  
  4.   "processors": [  
  5.     {  
  6.       "pipeline": {  
  7.         "name": "general-pipeline"  
  8.       }  
  9.     }  
  10.   ]  
  11. }

copy

Step 2 (Optional): Test the pipeline

It is recommended that you test your pipeline before you ingest documents.

To test the pipeline, run the following query:

  1. POST _ingest/pipeline/outer-pipeline/_simulate
  2. {  
  3.   "docs": [  
  4.     {  
  5.       "_source": {  
  6.         "protocol": "https",  
  7.         "name":"test"  
  8.       }  
  9.     }  
  10.   ]  
  11. }

copy

Response

The following example response confirms that the pipeline is working as expected:

  1. {  
  2.   "docs": [  
  3.     {  
  4.       "doc": {  
  5.         "_index": "_index",  
  6.         "_id": "_id",  
  7.         "_source": {  
  8.           "protocol": "HTTPS"  
  9.         },  
  10.         "_ingest": {  
  11.           "timestamp": "2024-05-24T02:43:43.700735801Z"  
  12.         }  
  13.       }  
  14.     }  
  15.   ]  
  16. }

copy

Step 3: Ingest a document

The following query ingests a document into an index named testindex1:

  1. POST testindex1/_doc/1?pipeline=outer-pipeline  
  2. {  
  3.   "protocol": "https",  
  4.   "name": "test"  
  5. }

copy

Response

The request indexes the document with the protocol field converted to uppercase and the field name removed from the index testindex1, as shown in the following response:

  1. {  
  2.   "_index": "testindex1",  
  3.   "_id": "1",  
  4.   "_version": 2,  
  5.   "result": "created",  
  6.   "_shards": {  
  7.     "total": 2,  
  8.     "successful": 2,  
  9.     "failed": 0  
  10.   },  
  11.   "_seq_no": 1,  
  12.   "_primary_term": 1  
  13. }

copy

Step 4 (Optional): Retrieve the document

To retrieve the document, run the following query:

  1. GET testindex1/_doc/1

copy

Response

The response shows the document with the protocol field converted to uppercase and the field name removed:

  1. {  
  2.   "_index": "testindex1",  
  3.   "_id": "1",  
  4.   "_version": 2,  
  5.   "_seq_no": 1,  
  6.   "_primary_term": 1,  
  7.   "found": true,  
  8.   "_source": {  
  9.     "protocol": "HTTPS"  
  10.   }  
  11. }