Index rollups API

Use the index rollup operations to programmatically work with index rollup jobs.

Table of contents

Create or update an index rollup job

Introduced 1.0

Creates or updates an index rollup job. You must provide the seq_no and primary_term parameters.

Request

  1. PUT _plugins/_rollup/jobs/<rollup_id> // Create
  2. PUT _plugins/_rollup/jobs/<rollup_id>?if_seq_no=1&if_primary_term=1 // Update
  3. {
  4.   "rollup": {
  5.     "source_index": "nyc-taxi-data",
  6.     "target_index": "rollup-nyc-taxi-data",
  7.     "schedule": {
  8.       "interval": {
  9.         "period": 1,
  10.         "unit": "Days"
  11.       }
  12.     },
  13.     "description": "Example rollup job",
  14.     "enabled": true,
  15.     "page_size": 200,
  16.     "delay": 0,
  17.     "roles": [
  18.       "rollup_all",
  19.       "nyc_taxi_all",
  20.       "example_rollup_index_all"
  21.     ],
  22.     "continuous": false,
  23.     "dimensions": {
  24.       "date_histogram": {
  25.         "source_field": "tpep_pickup_datetime",
  26.         "fixed_interval": "1h",
  27.         "timezone": "America/Los_Angeles"
  28.       },
  29.       "terms": {
  30.         "source_field": "PULocationID"
  31.       },
  32.       "metrics": [
  33.         {
  34.           "source_field": "passenger_count",
  35.           "metrics": [
  36.             {
  37.               "avg": {}
  38.             },
  39.             {
  40.               "sum": {}
  41.             },
  42.             {
  43.               "max": {}
  44.             },
  45.             {
  46.               "min": {}
  47.             },
  48.             {
  49.               "value_count": {}
  50.             }
  51.           ]
  52.         }
  53.       ]
  54.     }
  55.   }
  56. }

You can specify the following options.

OptionsDescriptionTypeRequired
source_indexThe name of the detector.StringYes
target_indexSpecify the target index that the rolled up data is ingested into. You could either create a new target index or use an existing index. The target index cannot be a combination of raw and rolled up data.StringYes
scheduleSchedule of the index rollup job which can be an interval or a cron expression.ObjectYes
schedule.intervalSpecify the frequency of execution of the rollup job.ObjectNo
schedule.interval.start_timeStart time of the interval.TimestampYes
schedule.interval.periodDefine the interval period.StringYes
schedule.interval.unitSpecify the time unit of the interval.StringYes
schedule.interval.cronOptionally, specify a cron expression to define therollup frequency.ListNo
schedule.interval.cron.expressionSpecify a Unix cron expression.StringYes
schedule.interval.cron.timezoneSpecify timezones as defined by the IANA Time Zone Database. Defaults to UTC.StringNo
descriptionOptionally, describe the rollup job.StringNo
enabledWhen true, the index rollup job is scheduled. Default is true.BooleanYes
continuousSpecify whether or not the index rollup job continuously rolls up data forever or just executes over the current data set once and stops. Default is false.BooleanYes
error_notificationSet up a Mustache message template sent for error notifications. For example, if an index rollup job fails, the system sends a message to a Slack channel.ObjectNo
page_sizeSpecify the number of buckets to paginate through at a time while rolling up.NumberYes
delayThe number of milliseconds to delay execution of the index rollup job.LongNo
dimensionsSpecify aggregations to create dimensions for the roll up time window.ObjectYes
dimensions.date_histogramSpecify either fixed_interval or calendar_interval, but not both. Either one limits what you can query in the target index.ObjectNo
dimensions.date_histogram.fixed_intervalSpecify the fixed interval for aggregations in milliseconds, seconds, minutes, hours, or days.StringNo
dimensions.date_histogram.calendar_intervalSpecify the calendar interval for aggregations in minutes, hours, days, weeks, months, quarters, or years.StringNo
dimensions.date_histogram.fieldSpecify the date field used in date histogram aggregation.StringNo
dimensions.date_histogram.timezoneSpecify the timezones as defined by the IANA Time Zone Database. The default is UTC.StringNo
dimensions.termsSpecify the term aggregations that you want to roll up.ObjectNo
dimensions.terms.fieldsSpecify terms aggregation for compatible fields.ObjectNo
dimensions.histogramSpecify the histogram aggregations that you want to roll up.ObjectNo
dimensions.histogram.fieldAdd a field for histogram aggregations.StringYes
dimensions.histogram.intervalSpecify the histogram aggregation interval for the field.LongYes
dimensions.metricsSpecify a list of objects that represent the fields and metrics that you want to calculate.Nested objectNo
dimensions.metrics.fieldSpecify the field that you want to perform metric aggregations on.StringNo
dimensions.metrics.field.metricsSpecify the metric aggregations you want to calculate for the field.Multiple stringsNo

Sample response

  1. {
  2.   "_id": "rollup_id",
  3.   "_seqNo": 1,
  4.   "_primaryTerm": 1,
  5.   "rollup": { ... }
  6. }

Get an index rollup job

Introduced 1.0

Returns all information about an index rollup job based on the rollup_id.

Request

  1. GET _plugins/_rollup/jobs/<rollup_id>

Sample response

  1. {
  2.   "_id": "my_rollup",
  3.   "_seqNo": 1,
  4.   "_primaryTerm": 1,
  5.   "rollup": { ... }
  6. }

Delete an index rollup job

Introduced 1.0

Deletes an index rollup job based on the rollup_id.

Request

  1. DELETE _plugins/_rollup/jobs/<rollup_id>

Sample response

  1. 200 OK

Start or stop an index rollup job

Introduced 1.0

Start or stop an index rollup job.

Request

  1. POST _plugins/_rollup/jobs/<rollup_id>/_start
  2. POST _plugins/_rollup/jobs/<rollup_id>/_stop

Sample response

  1. 200 OK

Explain an index rollup job

Introduced 1.0

Returns detailed metadata information about the index rollup job and its current progress.

Request

  1. GET _plugins/_rollup/jobs/<rollup_id>/_explain

Sample response

  1. {
  2.   "example_rollup": {
  3.     "rollup_id": "example_rollup",
  4.     "last_updated_time": 1602014281,
  5.     "continuous": {
  6.       "next_window_start_time": 1602055591,
  7.       "next_window_end_time": 1602075591
  8.     },
  9.     "status": "running",
  10.     "failure_reason": null,
  11.     "stats": {
  12.       "pages_processed": 342,
  13.       "documents_processed": 489359,
  14.       "rollups_indexed": 3420,
  15.       "index_time_in_ms": 30495,
  16.       "search_time_in_ms": 584922
  17.     }
  18.   }
  19. }