Bucket correlation aggregation

Bucket correlation aggregation

This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

A sibling pipeline aggregation which executes a correlation function on the configured sibling multi-bucket aggregation.

Parameters

buckets_path

(Required, string) Path to the buckets that contain one set of values to correlate. For syntax, see buckets_path Syntax.

function

(Required, object) The correlation function to execute.

Properties of function

  • count_correlation

    (Required*, object) The configuration to calculate a count correlation. This function is designed for determining the correlation of a term value and a given metric. Consequently, it needs to meet the following requirements.

    • The buckets_path must point to a _count metric.
    • The total count of all the bucket_path count values must be less than or equal to indicator.doc_count.
    • When utilizing this function, an initial calculation to gather the required indicator values is required.

    Properties of count_correlation

    • indicator

      (Required, object) The indicator with which to correlate the configured bucket_path values.

      Properties of indicator

      doc_count

      (Required, integer) The total number of documents that initially created the expectations. It’s required to be greater than or equal to the sum of all values in the buckets_path as this is the originating superset of data to which the term values are correlated.

      expectations

      (Required, array) An array of numbers with which to correlate the configured bucket_path values. The length of this value must always equal the number of buckets returned by the bucket_path.

      fractions

      (Optional, array) An array of fractions to use when averaging and calculating variance. This should be used if the pre-calculated data and the buckets_path have known gaps. The length of fractions, if provided, must equal expectations.

Syntax

A bucket_correlation aggregation looks like this in isolation:

  1. {
  2.   "bucket_correlation": {
  3.     "buckets_path": "range_values>_count", 
  4.     "function": {
  5.       "count_correlation": { 
  6.         "indicator": {
  7.           "expectations": [...],
  8.           "doc_count": 10000
  9.         }
  10.       }
  11.     }
  12.   }
  13. }

The buckets containing the values to correlate against.

The correlation function definition.

Example

The following snippet correlates the individual terms in the field version with the latency metric. Not shown is the pre-calculation of the latency indicator values, which was done utilizing the percentiles aggregation.

This example is only using the 10s percentiles.

  1. POST correlate_latency/_search?size=0&filter_path=aggregations
  2. {
  3.   "aggs": {
  4.     "buckets": {
  5.       "terms": { 
  6.         "field": "version",
  7.         "size": 2
  8.       },
  9.       "aggs": {
  10.         "latency_ranges": {
  11.           "range": { 
  12.             "field": "latency",
  13.             "ranges": [
  14.               { "to": 0.0 },
  15.               { "from": 0, "to": 105 },
  16.               { "from": 105, "to": 225 },
  17.               { "from": 225, "to": 445 },
  18.               { "from": 445, "to": 665 },
  19.               { "from": 665, "to": 885 },
  20.               { "from": 885, "to": 1115 },
  21.               { "from": 1115, "to": 1335 },
  22.               { "from": 1335, "to": 1555 },
  23.               { "from": 1555, "to": 1775 },
  24.               { "from": 1775 }
  25.             ]
  26.           }
  27.         },
  28.         "bucket_correlation": { 
  29.           "bucket_correlation": {
  30.             "buckets_path": "latency_ranges>_count",
  31.             "function": {
  32.               "count_correlation": {
  33.                 "indicator": {
  34.                    "expectations": [0, 52.5, 165, 335, 555, 775, 1000, 1225, 1445, 1665, 1775],
  35.                    "doc_count": 200
  36.                 }
  37.               }
  38.             }
  39.           }
  40.         }
  41.       }
  42.     }
  43.   }
  44. }

The term buckets containing a range aggregation and the bucket correlation aggregation. Both are utilized to calculate the correlation of the term values with the latency.

The range aggregation on the latency field. The ranges were created referencing the percentiles of the latency field.

The bucket correlation aggregation that calculates the correlation of the number of term values within each range and the previously calculated indicator values.

And the following may be the response:

  1. {
  2.   "aggregations" : {
  3.     "buckets" : {
  4.       "doc_count_error_upper_bound" : 0,
  5.       "sum_other_doc_count" : 0,
  6.       "buckets" : [
  7.         {
  8.           "key" : "1.0",
  9.           "doc_count" : 100,
  10.           "latency_ranges" : {
  11.             "buckets" : [
  12.               {
  13.                 "key" : "*-0.0",
  14.                 "to" : 0.0,
  15.                 "doc_count" : 0
  16.               },
  17.               {
  18.                 "key" : "0.0-105.0",
  19.                 "from" : 0.0,
  20.                 "to" : 105.0,
  21.                 "doc_count" : 1
  22.               },
  23.               {
  24.                 "key" : "105.0-225.0",
  25.                 "from" : 105.0,
  26.                 "to" : 225.0,
  27.                 "doc_count" : 9
  28.               },
  29.               {
  30.                 "key" : "225.0-445.0",
  31.                 "from" : 225.0,
  32.                 "to" : 445.0,
  33.                 "doc_count" : 0
  34.               },
  35.               {
  36.                 "key" : "445.0-665.0",
  37.                 "from" : 445.0,
  38.                 "to" : 665.0,
  39.                 "doc_count" : 0
  40.               },
  41.               {
  42.                 "key" : "665.0-885.0",
  43.                 "from" : 665.0,
  44.                 "to" : 885.0,
  45.                 "doc_count" : 0
  46.               },
  47.               {
  48.                 "key" : "885.0-1115.0",
  49.                 "from" : 885.0,
  50.                 "to" : 1115.0,
  51.                 "doc_count" : 10
  52.               },
  53.               {
  54.                 "key" : "1115.0-1335.0",
  55.                 "from" : 1115.0,
  56.                 "to" : 1335.0,
  57.                 "doc_count" : 20
  58.               },
  59.               {
  60.                 "key" : "1335.0-1555.0",
  61.                 "from" : 1335.0,
  62.                 "to" : 1555.0,
  63.                 "doc_count" : 20
  64.               },
  65.               {
  66.                 "key" : "1555.0-1775.0",
  67.                 "from" : 1555.0,
  68.                 "to" : 1775.0,
  69.                 "doc_count" : 20
  70.               },
  71.               {
  72.                 "key" : "1775.0-*",
  73.                 "from" : 1775.0,
  74.                 "doc_count" : 20
  75.               }
  76.             ]
  77.           },
  78.           "bucket_correlation" : {
  79.             "value" : 0.8402398981360937
  80.           }
  81.         },
  82.         {
  83.           "key" : "2.0",
  84.           "doc_count" : 100,
  85.           "latency_ranges" : {
  86.             "buckets" : [
  87.               {
  88.                 "key" : "*-0.0",
  89.                 "to" : 0.0,
  90.                 "doc_count" : 0
  91.               },
  92.               {
  93.                 "key" : "0.0-105.0",
  94.                 "from" : 0.0,
  95.                 "to" : 105.0,
  96.                 "doc_count" : 19
  97.               },
  98.               {
  99.                 "key" : "105.0-225.0",
  100.                 "from" : 105.0,
  101.                 "to" : 225.0,
  102.                 "doc_count" : 11
  103.               },
  104.               {
  105.                 "key" : "225.0-445.0",
  106.                 "from" : 225.0,
  107.                 "to" : 445.0,
  108.                 "doc_count" : 20
  109.               },
  110.               {
  111.                 "key" : "445.0-665.0",
  112.                 "from" : 445.0,
  113.                 "to" : 665.0,
  114.                 "doc_count" : 20
  115.               },
  116.               {
  117.                 "key" : "665.0-885.0",
  118.                 "from" : 665.0,
  119.                 "to" : 885.0,
  120.                 "doc_count" : 20
  121.               },
  122.               {
  123.                 "key" : "885.0-1115.0",
  124.                 "from" : 885.0,
  125.                 "to" : 1115.0,
  126.                 "doc_count" : 10
  127.               },
  128.               {
  129.                 "key" : "1115.0-1335.0",
  130.                 "from" : 1115.0,
  131.                 "to" : 1335.0,
  132.                 "doc_count" : 0
  133.               },
  134.               {
  135.                 "key" : "1335.0-1555.0",
  136.                 "from" : 1335.0,
  137.                 "to" : 1555.0,
  138.                 "doc_count" : 0
  139.               },
  140.               {
  141.                 "key" : "1555.0-1775.0",
  142.                 "from" : 1555.0,
  143.                 "to" : 1775.0,
  144.                 "doc_count" : 0
  145.               },
  146.               {
  147.                 "key" : "1775.0-*",
  148.                 "from" : 1775.0,
  149.                 "doc_count" : 0
  150.               }
  151.             ]
  152.           },
  153.           "bucket_correlation" : {
  154.             "value" : -0.5759855613334943
  155.           }
  156.         }
  157.       ]
  158.     }
  159.   }
  160. }