Conditionals with the Pipeline Processor

The combination of the if conditional and the Pipeline can result in a simple, yet powerful means to process heterogeneous input. For example, you can define a single pipeline that delegates to other pipelines based on some criteria.

  1. PUT _ingest/pipeline/logs_pipeline
  2. {
  3.   "description": "A pipeline of pipelines for log files",
  4.   "version": 1,
  5.   "processors": [
  6.     {
  7.       "pipeline": {
  8.         "if": "ctx.service?.name == 'apache_httpd'",
  9.         "name": "httpd_pipeline"
  10.       }
  11.     },
  12.     {
  13.       "pipeline": {
  14.         "if": "ctx.service?.name == 'syslog'",
  15.         "name": "syslog_pipeline"
  16.       }
  17.     },
  18.     {
  19.       "fail": {
  20.         "if": "ctx.service?.name != 'apache_httpd' && ctx.service?.name != 'syslog'",
  21.         "message": "This pipeline requires service.name to be either `syslog` or `apache_httpd`"
  22.       }
  23.     }
  24.   ]
  25. }

The above example allows consumers to point to a single pipeline for all log based index requests. Based on the conditional, the correct pipeline will be called to process that type of data.

This pattern works well with a default pipeline defined in an index mapping template for all indexes that hold data that needs pre-index processing.