Date histogram aggregation

This multi-bucket aggregation is similar to the normal histogram, but it can only be used with date or date range values. Because dates are represented internally in Elasticsearch as long values, it is possible, but not as accurate, to use the normal histogram on dates as well. The main difference in the two APIs is that here the interval can be specified using date/time expressions. Time-based data requires special support because time-based intervals are not always a fixed length.

Like the histogram, values are rounded down into the closest bucket. For example, if the interval is a calendar day, 2020-01-03T07:00:01Z is rounded to 2020-01-03T00:00:00Z. Values are rounded as follows:

  1. bucket_key = Math.floor(value / interval) * interval)

Calendar and fixed intervals

When configuring a date histogram aggregation, the interval can be specified in two manners: calendar-aware time intervals, and fixed time intervals.

Calendar-aware intervals understand that daylight savings changes the length of specific days, months have different amounts of days, and leap seconds can be tacked onto a particular year.

Fixed intervals are, by contrast, always multiples of SI units and do not change based on calendaring context.

Combined interval field is deprecated

[7.2] Deprecated in 7.2. interval field is deprecated Historically both calendar and fixed intervals were configured in a single interval field, which led to confusing semantics. Specifying 1d would be assumed as a calendar-aware time, whereas 2d would be interpreted as fixed time. To get “one day” of fixed time, the user would need to specify the next smaller unit (in this case, 24h).

This combined behavior was often unknown to users, and even when knowledgeable about the behavior it was difficult to use and confusing.

This behavior has been deprecated in favor of two new, explicit fields: calendar_interval and fixed_interval.

By forcing a choice between calendar and intervals up front, the semantics of the interval are clear to the user immediately and there is no ambiguity. The old interval field will be removed in the future.

Calendar intervals

Calendar-aware intervals are configured with the calendar_interval parameter. You can specify calendar intervals using the unit name, such as month, or as a single unit quantity, such as 1M. For example, day and 1d are equivalent. Multiple quantities, such as 2d, are not supported.

The accepted calendar intervals are:

minute, 1m

All minutes begin at 00 seconds. One minute is the interval between 00 seconds of the first minute and 00 seconds of the following minute in the specified time zone, compensating for any intervening leap seconds, so that the number of minutes and seconds past the hour is the same at the start and end.

hour, 1h

All hours begin at 00 minutes and 00 seconds. One hour (1h) is the interval between 00:00 minutes of the first hour and 00:00 minutes of the following hour in the specified time zone, compensating for any intervening leap seconds, so that the number of minutes and seconds past the hour is the same at the start and end.

day, 1d

All days begin at the earliest possible time, which is usually 00:00:00 (midnight). One day (1d) is the interval between the start of the day and the start of of the following day in the specified time zone, compensating for any intervening time changes.

week, 1w

One week is the interval between the start day_of_week:hour:minute:second and the same day of the week and time of the following week in the specified time zone.

month, 1M

One month is the interval between the start day of the month and time of day and the same day of the month and time of the following month in the specified time zone, so that the day of the month and time of day are the same at the start and end.

quarter, 1q

One quarter is the interval between the start day of the month and time of day and the same day of the month and time of day three months later, so that the day of the month and time of day are the same at the start and end.

year, 1y

One year is the interval between the start day of the month and time of day and the same day of the month and time of day the following year in the specified time zone, so that the date and time are the same at the start and end.

Calendar interval examples

As an example, here is an aggregation requesting bucket intervals of a month in calendar time:

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "sales_over_time": {
  5.       "date_histogram": {
  6.         "field": "date",
  7.         "calendar_interval": "month"
  8.       }
  9.     }
  10.   }
  11. }

If you attempt to use multiples of calendar units, the aggregation will fail because only singular calendar units are supported:

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "sales_over_time": {
  5.       "date_histogram": {
  6.         "field": "date",
  7.         "calendar_interval": "2d"
  8.       }
  9.     }
  10.   }
  11. }
  1. {
  2.   "error" : {
  3.     "root_cause" : [...],
  4.     "type" : "x_content_parse_exception",
  5.     "reason" : "[1:82] [date_histogram] failed to parse field [calendar_interval]",
  6.     "caused_by" : {
  7.       "type" : "illegal_argument_exception",
  8.       "reason" : "The supplied interval [2d] could not be parsed as a calendar interval.",
  9.       "stack_trace" : "java.lang.IllegalArgumentException: The supplied interval [2d] could not be parsed as a calendar interval."
  10.     }
  11.   }
  12. }

Fixed intervals

Fixed intervals are configured with the fixed_interval parameter.

In contrast to calendar-aware intervals, fixed intervals are a fixed number of SI units and never deviate, regardless of where they fall on the calendar. One second is always composed of 1000ms. This allows fixed intervals to be specified in any multiple of the supported units.

However, it means fixed intervals cannot express other units such as months, since the duration of a month is not a fixed quantity. Attempting to specify a calendar interval like month or quarter will throw an exception.

The accepted units for fixed intervals are:

milliseconds (ms)

A single millisecond. This is a very, very small interval.

seconds (s)

Defined as 1000 milliseconds each.

minutes (m)

Defined as 60 seconds each (60,000 milliseconds). All minutes begin at 00 seconds.

hours (h)

Defined as 60 minutes each (3,600,000 milliseconds). All hours begin at 00 minutes and 00 seconds.

days (d)

Defined as 24 hours (86,400,000 milliseconds). All days begin at the earliest possible time, which is usually 00:00:00 (midnight).

Fixed interval examples

If we try to recreate the “month” calendar_interval from earlier, we can approximate that with 30 fixed days:

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "sales_over_time": {
  5.       "date_histogram": {
  6.         "field": "date",
  7.         "fixed_interval": "30d"
  8.       }
  9.     }
  10.   }
  11. }

But if we try to use a calendar unit that is not supported, such as weeks, we’ll get an exception:

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "sales_over_time": {
  5.       "date_histogram": {
  6.         "field": "date",
  7.         "fixed_interval": "2w"
  8.       }
  9.     }
  10.   }
  11. }
  1. {
  2.   "error" : {
  3.     "root_cause" : [...],
  4.     "type" : "x_content_parse_exception",
  5.     "reason" : "[1:82] [date_histogram] failed to parse field [fixed_interval]",
  6.     "caused_by" : {
  7.       "type" : "illegal_argument_exception",
  8.       "reason" : "failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized",
  9.       "stack_trace" : "java.lang.IllegalArgumentException: failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized"
  10.     }
  11.   }
  12. }

Notes

In all cases, when the specified end time does not exist, the actual end time is the closest available time after the specified end.

Widely distributed applications must also consider vagaries such as countries that start and stop daylight savings time at 12:01 A.M., so end up with one minute of Sunday followed by an additional 59 minutes of Saturday once a year, and countries that decide to move across the international date line. Situations like that can make irregular time zone offsets seem easy.

As always, rigorous testing, especially around time-change events, will ensure that your time interval specification is what you intend it to be.

WARNING: To avoid unexpected results, all connected servers and clients must sync to a reliable network time service.

fractional time values are not supported, but you can address this by shifting to another time unit (e.g., 1.5h could instead be specified as 90m).

You can also specify time values using abbreviations supported by time units parsing.

Keys

Internally, a date is represented as a 64 bit number representing a timestamp in milliseconds-since-the-epoch (01/01/1970 midnight UTC). These timestamps are returned as the key name of the bucket. The key_as_string is the same timestamp converted to a formatted date string using the format parameter specification:

If you don’t specify format, the first date format specified in the field mapping is used.

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "sales_over_time": {
  5.       "date_histogram": {
  6.         "field": "date",
  7.         "calendar_interval": "1M",
  8.         "format": "yyyy-MM-dd" 
  9.       }
  10.     }
  11.   }
  12. }

Supports expressive date format pattern

Response:

  1. {
  2.   ...
  3.   "aggregations": {
  4.     "sales_over_time": {
  5.       "buckets": [
  6.         {
  7.           "key_as_string": "2015-01-01",
  8.           "key": 1420070400000,
  9.           "doc_count": 3
  10.         },
  11.         {
  12.           "key_as_string": "2015-02-01",
  13.           "key": 1422748800000,
  14.           "doc_count": 2
  15.         },
  16.         {
  17.           "key_as_string": "2015-03-01",
  18.           "key": 1425168000000,
  19.           "doc_count": 2
  20.         }
  21.       ]
  22.     }
  23.   }
  24. }

Time zone

Elasticsearch stores date-times in Coordinated Universal Time (UTC). By default, all bucketing and rounding is also done in UTC. Use the time_zone parameter to indicate that bucketing should use a different time zone.

For example, if the interval is a calendar day and the time zone is America/New_York then 2020-01-03T01:00:01Z is : # Converted to 2020-01-02T18:00:01 # Rounded down to 2020-01-02T00:00:00 # Then converted back to UTC to produce 2020-01-02T05:00:00:00Z # Finally, when the bucket is turned into a string key it is printed in America/New_York so it’ll display as "2020-01-02T00:00:00".

It looks like:

  1. bucket_key = localToUtc(Math.floor(utcToLocal(value) / interval) * interval))

You can specify time zones as an ISO 8601 UTC offset (e.g. +01:00 or -08:00) or as an IANA time zone ID, such as America/Los_Angeles.

Consider the following example:

  1. PUT my-index-000001/_doc/1?refresh
  2. {
  3.   "date": "2015-10-01T00:30:00Z"
  4. }
  6. PUT my-index-000001/_doc/2?refresh
  7. {
  8.   "date": "2015-10-01T01:30:00Z"
  9. }
  11. GET my-index-000001/_search?size=0
  12. {
  13.   "aggs": {
  14.     "by_day": {
  15.       "date_histogram": {
  16.         "field":     "date",
  17.         "calendar_interval":  "day"
  18.       }
  19.     }
  20.   }
  21. }

If you don’t specify a time zone, UTC is used. This would result in both of these documents being placed into the same day bucket, which starts at midnight UTC on 1 October 2015:

  1. {
  2.   ...
  3.   "aggregations": {
  4.     "by_day": {
  5.       "buckets": [
  6.         {
  7.           "key_as_string": "2015-10-01T00:00:00.000Z",
  8.           "key":           1443657600000,
  9.           "doc_count":     2
  10.         }
  11.       ]
  12.     }
  13.   }
  14. }

If you specify a time_zone of -01:00, midnight in that time zone is one hour before midnight UTC:

  1. GET my-index-000001/_search?size=0
  2. {
  3.   "aggs": {
  4.     "by_day": {
  5.       "date_histogram": {
  6.         "field":     "date",
  7.         "calendar_interval":  "day",
  8.         "time_zone": "-01:00"
  9.       }
  10.     }
  11.   }
  12. }

Now the first document falls into the bucket for 30 September 2015, while the second document falls into the bucket for 1 October 2015:

  1. {
  2.   ...
  3.   "aggregations": {
  4.     "by_day": {
  5.       "buckets": [
  6.         {
  7.           "key_as_string": "2015-09-30T00:00:00.000-01:00", 
  8.           "key": 1443574800000,
  9.           "doc_count": 1
  10.         },
  11.         {
  12.           "key_as_string": "2015-10-01T00:00:00.000-01:00", 
  13.           "key": 1443661200000,
  14.           "doc_count": 1
  15.         }
  16.       ]
  17.     }
  18.   }
  19. }

The key_as_string value represents midnight on each day in the specified time zone.

Many time zones shift their clocks for daylight savings time. Buckets close to the moment when those changes happen can have slightly different sizes than you would expect from the calendar_interval or fixed_interval. For example, consider a DST start in the CET time zone: on 27 March 2016 at 2am, clocks were turned forward 1 hour to 3am local time. If you use day as the calendar_interval, the bucket covering that day will only hold data for 23 hours instead of the usual 24 hours for other buckets. The same is true for shorter intervals, like a fixed_interval of 12h, where you’ll have only a 11h bucket on the morning of 27 March when the DST shift happens.

Offset

Use the offset parameter to change the start value of each bucket by the specified positive (+) or negative offset (-) duration, such as 1h for an hour, or 1d for a day. See Time units for more possible time duration options.

For example, when using an interval of day, each bucket runs from midnight to midnight. Setting the offset parameter to +6h changes each bucket to run from 6am to 6am:

  1. PUT my-index-000001/_doc/1?refresh
  2. {
  3.   "date": "2015-10-01T05:30:00Z"
  4. }
  6. PUT my-index-000001/_doc/2?refresh
  7. {
  8.   "date": "2015-10-01T06:30:00Z"
  9. }
  11. GET my-index-000001/_search?size=0
  12. {
  13.   "aggs": {
  14.     "by_day": {
  15.       "date_histogram": {
  16.         "field":     "date",
  17.         "calendar_interval":  "day",
  18.         "offset":    "+6h"
  19.       }
  20.     }
  21.   }
  22. }

Instead of a single bucket starting at midnight, the above request groups the documents into buckets starting at 6am:

  1. {
  2.   ...
  3.   "aggregations": {
  4.     "by_day": {
  5.       "buckets": [
  6.         {
  7.           "key_as_string": "2015-09-30T06:00:00.000Z",
  8.           "key": 1443592800000,
  9.           "doc_count": 1
  10.         },
  11.         {
  12.           "key_as_string": "2015-10-01T06:00:00.000Z",
  13.           "key": 1443679200000,
  14.           "doc_count": 1
  15.         }
  16.       ]
  17.     }
  18.   }
  19. }

The start offset of each bucket is calculated after time_zone adjustments have been made.

Keyed Response

Setting the keyed flag to true associates a unique string key with each bucket and returns the ranges as a hash rather than an array:

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "sales_over_time": {
  5.       "date_histogram": {
  6.         "field": "date",
  7.         "calendar_interval": "1M",
  8.         "format": "yyyy-MM-dd",
  9.         "keyed": true
  10.       }
  11.     }
  12.   }
  13. }

Response:

  1. {
  2.   ...
  3.   "aggregations": {
  4.     "sales_over_time": {
  5.       "buckets": {
  6.         "2015-01-01": {
  7.           "key_as_string": "2015-01-01",
  8.           "key": 1420070400000,
  9.           "doc_count": 3
  10.         },
  11.         "2015-02-01": {
  12.           "key_as_string": "2015-02-01",
  13.           "key": 1422748800000,
  14.           "doc_count": 2
  15.         },
  16.         "2015-03-01": {
  17.           "key_as_string": "2015-03-01",
  18.           "key": 1425168000000,
  19.           "doc_count": 2
  20.         }
  21.       }
  22.     }
  23.   }
  24. }

Scripts

As with the normal histogram, both document-level scripts and value-level scripts are supported. You can control the order of the returned buckets using the order settings and filter the returned buckets based on a min_doc_count setting (by default all buckets between the first bucket that matches documents and the last one are returned). This histogram also supports the extended_bounds setting, which enables extending the bounds of the histogram beyond the data itself. For more information, see Extended Bounds.

Missing value

The missing parameter defines how to treat documents that are missing a value. By default, they are ignored, but it is also possible to treat them as if they have a value.

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "sale_date": {
  5.       "date_histogram": {
  6.         "field": "date",
  7.         "calendar_interval": "year",
  8.         "missing": "2000/01/01" 
  9.       }
  10.     }
  11.   }
  12. }

Documents without a value in the publish_date field will fall into the same bucket as documents that have the value 2000-01-01.

Order

By default the returned buckets are sorted by their key ascending, but you can control the order using the order setting. This setting supports the same order functionality as Terms Aggregation.

Using a script to aggregate by day of the week

When you need to aggregate the results by day of the week, use a script that returns the day of the week:

  1. POST /sales/_search?size=0
  2. {
  3.   "aggs": {
  4.     "dayOfWeek": {
  5.       "terms": {
  6.         "script": {
  7.           "lang": "painless",
  8.           "source": "doc['date'].value.dayOfWeekEnum.value"
  9.         }
  10.       }
  11.     }
  12.   }
  13. }

Response:

  1. {
  2.   ...
  3.   "aggregations": {
  4.     "dayOfWeek": {
  5.       "doc_count_error_upper_bound": 0,
  6.       "sum_other_doc_count": 0,
  7.       "buckets": [
  8.         {
  9.           "key": "7",
  10.           "doc_count": 4
  11.         },
  12.         {
  13.           "key": "4",
  14.           "doc_count": 3
  15.         }
  16.       ]
  17.     }
  18.   }
  19. }

The response will contain all the buckets having the relative day of the week as key : 1 for Monday, 2 for Tuesday…​ 7 for Sunday.