Index rollups API

Use the index rollup operations to programmatically work with index rollup jobs.

Create or update an index rollup job

Introduced 1.0

Creates or updates an index rollup job. You must provide the seq_no and primary_term parameters.

Request

  1. PUT _plugins/_rollup/jobs/<rollup_id> // Create
  2. PUT _plugins/_rollup/jobs/<rollup_id>?if_seq_no=1&if_primary_term=1 // Update
  3. {
  4.   "rollup": {
  5.     "source_index": "nyc-taxi-data",
  6.     "target_index": "rollup-nyc-taxi-data",
  7.     "schedule": {
  8.       "interval": {
  9.         "period": 1,
  10.         "unit": "Days"
  11.       }
  12.     },
  13.     "description": "Example rollup job",
  14.     "enabled": true,
  15.     "page_size": 200,
  16.     "delay": 0,
  17.     "roles": [
  18.       "rollup_all",
  19.       "nyc_taxi_all",
  20.       "example_rollup_index_all"
  21.     ],
  22.     "continuous": false,
  23.     "dimensions": [
  24.       {
  25.         "date_histogram": {
  26.           "source_field": "tpep_pickup_datetime",
  27.           "fixed_interval": "1h",
  28.           "timezone": "America/Los_Angeles"
  29.         }
  30.       },
  31.       {
  32.         "terms": {
  33.           "source_field": "PULocationID"
  34.         }
  35.       }
  36.     ],
  37.     "metrics": [
  38.       {
  39.         "source_field": "passenger_count",
  40.         "metrics": [
  41.           {
  42.             "avg": {}
  43.           },
  44.           {
  45.             "sum": {}
  46.           },
  47.           {
  48.             "max": {}
  49.           },
  50.           {
  51.             "min": {}
  52.           },
  53.           {
  54.             "value_count": {}
  55.           }
  56.         ]
  57.       }
  58.     ]
  59.   }
  60. }

You can specify the following options.

OptionsDescriptionTypeRequired
sourceindexThe name of the detector.StringYes
target_indexSpecify the target index that the rolled up data is ingested into. You can either create a new target index or use an existing index. The target index cannot be a combination of raw and rolled up data. This field supports dynamically generated index names like rollup{{ctx.source_index}}, where source_index cannot contain wildcards.StringYes
scheduleSchedule of the index rollup job which can be an interval or a cron expression.ObjectYes
schedule.intervalSpecify the frequency of execution of the rollup job.ObjectNo
schedule.interval.start_timeStart time of the interval.TimestampYes
schedule.interval.periodDefine the interval period.StringYes
schedule.interval.unitSpecify the time unit of the interval.StringYes
schedule.interval.cronOptionally, specify a cron expression to define therollup frequency.ListNo
schedule.interval.cron.expressionSpecify a Unix cron expression.StringYes
schedule.interval.cron.timezoneSpecify timezones as defined by the IANA Time Zone Database. Defaults to UTC.StringNo
descriptionOptionally, describe the rollup job.StringNo
enabledWhen true, the index rollup job is scheduled. Default is true.BooleanYes
continuousSpecify whether or not the index rollup job continuously rolls up data forever or just executes over the current data set once and stops. Default is false.BooleanYes
error_notificationSet up a Mustache message template for error notifications. For example, if an index rollup job fails, the system sends a message to a Slack channel.ObjectNo
page_sizeSpecify the number of buckets to paginate at a time during rollup.NumberYes
delayThe number of milliseconds to delay execution of the index rollup job.LongNo
dimensionsSpecify aggregations to create dimensions for the roll up time window. Supported groups are terms, histogram, and date_histogram. For more information, see Bucket Aggregations.ArrayYes
metricsSpecify a list of objects that represent the fields and metrics that you want to calculate. Supported metrics are sum, max, min, value_count and avg. For more information, see Metric Aggregations.ArrayNo

Example response

  1. {
  2.   "_id": "<rollup_id>",
  3.   "_version": 3,
  4.   "_seq_no": 1,
  5.   "_primary_term": 1,
  6.   "rollup": {
  7.     "rollup_id": "<rollup_id>",
  8.     "enabled": true,
  9.     "schedule": {
  10.       "interval": {
  11.         "start_time": 1680159934649,
  12.         "period": 1,
  13.         "unit": "Days",
  14.         "schedule_delay": 0
  15.       }
  16.     },
  17.     "last_updated_time": 1680159934649,
  18.     "enabled_time": 1680159934649,
  19.     "description": "Example rollup job",
  20.     "schema_version": 17,
  21.     "source_index": "nyc-taxi-data",
  22.     "target_index": "rollup-nyc-taxi-data",
  23.     "metadata_id": null,
  24.     "page_size": 200,
  25.     "delay": 0,
  26.     "continuous": false,
  27.     "dimensions": [
  28.       {
  29.         "date_histogram": {
  30.           "fixed_interval": "1h",
  31.           "source_field": "tpep_pickup_datetime",
  32.           "target_field": "tpep_pickup_datetime",
  33.           "timezone": "America/Los_Angeles"
  34.         }
  35.       },
  36.       {
  37.         "terms": {
  38.           "source_field": "PULocationID",
  39.           "target_field": "PULocationID"
  40.         }
  41.       }
  42.     ],
  43.     "metrics": [
  44.       {
  45.         "source_field": "passenger_count",
  46.         "metrics": [
  47.           {
  48.             "avg": {}
  49.           },
  50.           {
  51.             "sum": {}
  52.           },
  53.           {
  54.             "max": {}
  55.           },
  56.           {
  57.             "min": {}
  58.           },
  59.           {
  60.             "value_count": {}
  61.           }
  62.         ]
  63.       }
  64.     ]
  65.   }
  66. }

Get an index rollup job

Introduced 1.0

Returns all information about an index rollup job based on the rollup_id.

Request

  1. GET _plugins/_rollup/jobs/<rollup_id>

Example response

  1. {
  2.   "_id": "my_rollup",
  3.   "_seqNo": 1,
  4.   "_primaryTerm": 1,
  5.   "rollup": { ... }
  6. }

Delete an index rollup job

Introduced 1.0

Deletes an index rollup job based on the rollup_id.

Request

  1. DELETE _plugins/_rollup/jobs/<rollup_id>

Example response

  1. 200 OK

Start or stop an index rollup job

Introduced 1.0

Start or stop an index rollup job.

Request

  1. POST _plugins/_rollup/jobs/<rollup_id>/_start
  2. POST _plugins/_rollup/jobs/<rollup_id>/_stop

Example response

  1. 200 OK

Explain an index rollup job

Introduced 1.0

Returns detailed metadata information about the index rollup job and its current progress.

Request

  1. GET _plugins/_rollup/jobs/<rollup_id>/_explain

Example response

  1. {
  2.   "example_rollup": {
  3.     "rollup_id": "example_rollup",
  4.     "last_updated_time": 1602014281,
  5.     "continuous": {
  6.       "next_window_start_time": 1602055591,
  7.       "next_window_end_time": 1602075591
  8.     },
  9.     "status": "running",
  10.     "failure_reason": null,
  11.     "stats": {
  12.       "pages_processed": 342,
  13.       "documents_processed": 489359,
  14.       "rollups_indexed": 3420,
  15.       "index_time_in_ms": 30495,
  16.       "search_time_in_ms": 584922
  17.     }
  18.   }
  19. }