Handling pipeline failures

Introduced 1.0

Each ingest pipeline consists of a series of processors that are applied to the documents in sequence. If a processor fails, the entire pipeline will fail. You have two options for handling failures:

  • Fail the entire pipeline: If a processor fails, the entire pipeline will fail and the document will not be indexed.
  • Fail the current processor and continue with the next processor: This can be useful if you want to continue processing the document even if one of the processors fails.

By default, an ingest pipeline stops if one of its processors fails. If you want the pipeline to continue running when a processor fails, you can set the ignore_failure parameter for that processor to true when creating the pipeline:

  1. PUT _ingest/pipeline/my-pipeline/
  2. {
  3.   "description": "Rename 'provider' field to 'cloud.provider'",
  4.   "processors": [
  5.     {
  6.       "rename": {
  7.         "field": "provider",
  8.         "target_field": "cloud.provider",
  9.         "ignore_failure": true
  10.       }
  11.     }
  12.   ]
  13. }

copy

You can specify the on_failure parameter to run immediately after a processor fails. If you have specified on_failure, OpenSearch will run the other processors in the pipeline even if the on_failure configuration is empty:

  1. PUT _ingest/pipeline/my-pipeline/
  2. {
  3.   "description": "Add timestamp to the document",
  4.   "processors": [
  5.     {
  6.       "date": {
  7.         "field": "timestamp_field",
  8.         "formats": ["yyyy-MM-dd HH:mm:ss"],
  9.         "target_field": "@timestamp",
  10.         "on_failure": [
  11.           {
  12.             "set": {
  13.               "field": "ingest_error",
  14.               "value": "failed"
  15.             }
  16.           }
  17.         ]
  18.       }
  19.     }
  20.   ]
  21. }

copy

If the processor fails, OpenSearch logs the failure and continues to run all remaining processors in the search pipeline. To check whether there were any failures, you can use ingest pipeline metrics.

Ingest pipeline metrics

To view ingest pipeline metrics, use the Nodes Stats API:

  1. GET /_nodes/stats/ingest?filter_path=nodes.*.ingest

copy

The response contains statistics for all ingest pipelines, for example:

  1.  {
  2.   "nodes": {
  3.     "iFPgpdjPQ-uzTdyPLwQVnQ": {
  4.       "ingest": {
  5.         "total": {
  6.           "count": 28,
  7.           "time_in_millis": 82,
  8.           "current": 0,
  9.           "failed": 9
  10.         },
  11.         "pipelines": {
  12.           "user-behavior": {
  13.             "count": 5,
  14.             "time_in_millis": 0,
  15.             "current": 0,
  16.             "failed": 0,
  17.             "processors": [
  18.               {
  19.                 "append": {
  20.                   "type": "append",
  21.                   "stats": {
  22.                     "count": 5,
  23.                     "time_in_millis": 0,
  24.                     "current": 0,
  25.                     "failed": 0
  26.                   }
  27.                 }
  28.               }
  29.             ]
  30.           },
  31.            "remove_ip": {
  32.             "count": 5,
  33.             "time_in_millis": 9,
  34.             "current": 0,
  35.             "failed": 2,
  36.             "processors": [
  37.               {
  38.                 "remove": {
  39.                   "type": "remove",
  40.                   "stats": {
  41.                     "count": 5,
  42.                     "time_in_millis": 8,
  43.                     "current": 0,
  44.                     "failed": 2
  45.                   }
  46.                 }
  47.               }
  48.             ]
  49.           }
  50.         }
  51.       }
  52.     }
  53.   }
  54. }

Troubleshooting ingest pipeline failures: The first thing you should do is check the logs to see whether there are any errors or warnings that can help you identify the cause of the failure. OpenSearch logs contain information about the ingest pipeline that failed, including the processor that failed and the reason for the failure.