Boxplot Aggregation

A boxplot metrics aggregation that computes boxplot of numeric values extracted from the aggregated documents. These values can be generated by a provided script or extracted from specific numeric or histogram fields in the documents.

The boxplot aggregation returns essential information for making a box plot: minimum, maximum median, first quartile (25th percentile) and third quartile (75th percentile) values.

Syntax

A boxplot aggregation looks like this in isolation:

  1. {
  2.   "boxplot": {
  3.     "field": "load_time"
  4.   }
  5. }

Let’s look at a boxplot representing load time:

  1. GET latency/_search
  2. {
  3.   "size": 0,
  4.   "aggs": {
  5.     "load_time_boxplot": {
  6.       "boxplot": {
  7.         "field": "load_time" 
  8.       }
  9.     }
  10.   }
  11. }

The field load_time must be a numeric field

The response will look like this:

  1. {
  2.   ...
  4.  "aggregations": {
  5.     "load_time_boxplot": {
  6.       "min": 0.0,
  7.       "max": 990.0,
  8.       "q1": 165.0,
  9.       "q2": 445.0,
  10.       "q3": 725.0
  11.     }
  12.   }
  13. }

Script

The boxplot metric supports scripting. For example, if our load times are in milliseconds but we want values calculated in seconds, we could use a script to convert them on-the-fly:

  1. GET latency/_search
  2. {
  3.   "size": 0,
  4.   "aggs": {
  5.     "load_time_boxplot": {
  6.       "boxplot": {
  7.         "script": {
  8.           "lang": "painless",
  9.           "source": "doc['load_time'].value / params.timeUnit", 
  10.           "params": {
  11.             "timeUnit": 1000                                    
  12.           }
  13.         }
  14.       }
  15.     }
  16.   }
  17. }

The field parameter is replaced with a script parameter, which uses the script to generate values which percentiles are calculated on

Scripting supports parameterized input just like any other script

This will interpret the script parameter as an inline script with the painless script language and no script parameters. To use a stored script use the following syntax:

  1. GET latency/_search
  2. {
  3.   "size": 0,
  4.   "aggs": {
  5.     "load_time_boxplot": {
  6.       "boxplot": {
  7.         "script": {
  8.           "id": "my_script",
  9.           "params": {
  10.             "field": "load_time"
  11.           }
  12.         }
  13.       }
  14.     }
  15.   }
  16. }

Boxplot values are (usually) approximate

The algorithm used by the boxplot metric is called TDigest (introduced by Ted Dunning in Computing Accurate Quantiles using T-Digests).

Boxplot as other percentile aggregations are also non-deterministic. This means you can get slightly different results using the same data.

Compression

Approximate algorithms must balance memory utilization with estimation accuracy. This balance can be controlled using a compression parameter:

  1. GET latency/_search
  2. {
  3.   "size": 0,
  4.   "aggs": {
  5.     "load_time_boxplot": {
  6.       "boxplot": {
  7.         "field": "load_time",
  8.         "compression": 200    
  9.       }
  10.     }
  11.   }
  12. }

Compression controls memory usage and approximation error

The TDigest algorithm uses a number of “nodes” to approximate percentiles — the more nodes available, the higher the accuracy (and large memory footprint) proportional to the volume of data. The compression parameter limits the maximum number of nodes to 20 * compression.

Therefore, by increasing the compression value, you can increase the accuracy of your percentiles at the cost of more memory. Larger compression values also make the algorithm slower since the underlying tree data structure grows in size, resulting in more expensive operations. The default compression value is 100.

A “node” uses roughly 32 bytes of memory, so under worst-case scenarios (large amount of data which arrives sorted and in-order) the default settings will produce a TDigest roughly 64KB in size. In practice data tends to be more random and the TDigest will use less memory.

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

  1. GET latency/_search
  2. {
  3.   "size": 0,
  4.   "aggs": {
  5.     "grade_boxplot": {
  6.       "boxplot": {
  7.         "field": "grade",
  8.         "missing": 10     
  9.       }
  10.     }
  11.   }
  12. }

Documents without a value in the grade field will fall into the same bucket as documents that have the value 10.