Matrix stats aggregation

Matrix stats aggregation

The matrix_stats aggregation is a numeric aggregation that computes the following statistics over a set of document fields:

count

Number of per field samples included in the calculation.

mean

The average value for each field.

variance

Per field Measurement for how spread out the samples are from the mean.

skewness

Per field measurement quantifying the asymmetric distribution around the mean.

kurtosis

Per field measurement quantifying the shape of the distribution.

covariance

A matrix that quantitatively describes how changes in one field are associated with another.

correlation

The covariance matrix scaled to a range of -1 to 1, inclusive. Describes the relationship between field distributions.

Unlike other metric aggregations, the matrix_stats aggregation does not support scripting.

The following example demonstrates the use of matrix stats to describe the relationship between income and poverty.

  1. GET /_search
  2. {
  3.   "aggs": {
  4.     "statistics": {
  5.       "matrix_stats": {
  6.         "fields": [ "poverty", "income" ]
  7.       }
  8.     }
  9.   }
  10. }

The aggregation type is matrix_stats and the fields setting defines the set of fields (as an array) for computing the statistics. The above request returns the following response:

  1. {
  2.   ...
  3.   "aggregations": {
  4.     "statistics": {
  5.       "doc_count": 50,
  6.       "fields": [ {
  7.           "name": "income",
  8.           "count": 50,
  9.           "mean": 51985.1,
  10.           "variance": 7.383377037755103E7,
  11.           "skewness": 0.5595114003506483,
  12.           "kurtosis": 2.5692365287787124,
  13.           "covariance": {
  14.             "income": 7.383377037755103E7,
  15.             "poverty": -21093.65836734694
  16.           },
  17.           "correlation": {
  18.             "income": 1.0,
  19.             "poverty": -0.8352655256272504
  20.           }
  21.         }, {
  22.           "name": "poverty",
  23.           "count": 50,
  24.           "mean": 12.732000000000001,
  25.           "variance": 8.637730612244896,
  26.           "skewness": 0.4516049811903419,
  27.           "kurtosis": 2.8615929677997767,
  28.           "covariance": {
  29.             "income": -21093.65836734694,
  30.             "poverty": 8.637730612244896
  31.           },
  32.           "correlation": {
  33.             "income": -0.8352655256272504,
  34.             "poverty": 1.0
  35.           }
  36.         } ]
  37.     }
  38.   }
  39. }

The doc_count field indicates the number of documents involved in the computation of the statistics.

Multi Value Fields

The matrix_stats aggregation treats each document field as an independent sample. The mode parameter controls what array value the aggregation will use for array or multi-valued fields. This parameter can take one of the following:

avg

(default) Use the average of all values.

min

Pick the lowest value.

max

Pick the highest value.

sum

Use the sum of all values.

median

Use the median of all values.

Missing Values

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value. This is done by adding a set of fieldname : value mappings to specify default values per field.

  1. GET /_search
  2. {
  3.   "aggs": {
  4.     "matrixstats": {
  5.       "matrix_stats": {
  6.         "fields": [ "poverty", "income" ],
  7.         "missing": { "income": 50000 }      
  8.       }
  9.     }
  10.   }
  11. }

Documents without a value in the income field will have the default value 50000.