Fail processor

The fail processor is useful for performing data transformation and enrichment during the indexing process. The primary use case for the fail processor is to fail an indexing operation when certain conditions are met.

The following is the syntax for the fail processor:

  1. "fail": { 
  2.   "if": "ctx.foo == 'bar'", 
  3.   "message": "Custom error message" 
  4.   }

copy

Configuration parameters

The following table lists the required and optional parameters for the fail processor.

ParameterRequired/OptionalDescription
messageRequiredA custom error message to be included in the failure response.
descriptionOptionalA brief description of the processor.
ifOptionalA condition for running the processor.
ignore_failureOptionalSpecifies whether the processor continues execution even if it encounters an error. If set to true, then failures are ignored. Default is false.
on_failureOptionalA list of processors to run if the processor fails.
tagOptionalAn identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type.

Using the processor

Follow these steps to use the processor in a pipeline.

Step 1: Create a pipeline

The following query creates a pipeline, named fail-log-pipeline, that uses the fail processor to intentionally fail the pipeline execution for log events:

  1. PUT _ingest/pipeline/fail-log-pipeline  
  2. {  
  3.   "description": "A pipeline to test the fail processor for log events",  
  4.   "processors": [  
  5.     {  
  6.       "fail": {  
  7.         "if": "ctx.user_info.contains('password') || ctx.user_info.contains('credit card')",  
  8.         "message": "Document containing personally identifiable information (PII) cannot be indexed!"  
  9.       }  
  10.     }  
  11.   ]  
  12. }

copy

Step 2 (Optional): Test the pipeline

It is recommended that you test your pipeline before you ingest documents.

To test the pipeline, run the following query:

  1. POST _ingest/pipeline/fail-log-pipeline/_simulate  
  2. {  
  3.   "docs": [  
  4.     {  
  5.       "_source": {  
  6.         "user_info": "Sensitive information including credit card"  
  7.       }  
  8.     }  
  9.   ]  
  10. }

copy

Response

The following example response confirms that the pipeline is working as expected:

  1. {
  2.   "docs": [
  3.     {
  4.       "error": {
  5.         "root_cause": [
  6.           {
  7.             "type": "fail_processor_exception",
  8.             "reason": "Document containing personally identifiable information (PII) cannot be indexed!"
  9.           }
  10.         ],
  11.         "type": "fail_processor_exception",
  12.         "reason": "Document containing personally identifiable information (PII) cannot be indexed!"
  13.       }
  14.     }
  15.   ]
  16. }

copy

Step 3: Ingest a document

The following query ingests a document into an index named testindex1:

  1. PUT testindex1/_doc/1?pipeline=fail-log-pipeline  
  2. {  
  3.   "user_info": "Sensitive information including credit card"  
  4. }

copy

Response

The request fails to index the log event into the index testindex1 due to the string credit card being present in user_info. The following response includes the custom error message specified in the fail processor:

  2.   "error": {
  3.     "root_cause": [
  4.       {
  5.         "type": "fail_processor_exception",
  6.         "reason": "Document containing personally identifiable information (PII) cannot be indexed!"
  7.       }
  8.     ],
  9.     "type": "fail_processor_exception",
  10.     "reason": "Document containing personally identifiable information (PII) cannot be indexed!"
  11.   },
  12.   "status": 500
  13. }

copy

Step 4 (Optional): Retrieve the document

Because the log event was not indexed due to the pipeline failure, attempting to retrieve it results in the document not found error "found": false:

  1. GET testindex1/_doc/1

copy

Document error example

  1. {  
  2.   "_index": "testindex1",  
  3.   "_id": "1",  
  4.   "found": false  
  5. }