Significant terms aggregations

The significant_terms aggregation lets you spot unusual or interesting term occurrences in a filtered subset relative to the rest of the data in an index.

A foreground set is the set of documents that you filter. A background set is a set of all documents in an index. The significant_terms aggregation examines all documents in the foreground set and finds a score for significant occurrences in contrast to the documents in the background set.

In the sample web log data, each document has a field containing the user-agent of the visitor. This example searches for all requests from an iOS operating system. A regular terms aggregation on this foreground set returns Firefox because it has the most number of documents within this bucket. On the other hand, a significant_terms aggregation returns Internet Explorer (IE) because IE has a significantly higher appearance in the foreground set as compared to the background set.

  1. GET opensearch_dashboards_sample_data_logs/_search
  2. {
  3.   "size": 0,
  4.   "query": {
  5.     "terms": {
  6.       "machine.os.keyword": [
  7.         "ios"
  8.       ]
  9.     }
  10.   },
  11.   "aggs": {
  12.     "significant_response_codes": {
  13.       "significant_terms": {
  14.         "field": "agent.keyword"
  15.       }
  16.     }
  17.   }
  18. }

copy

Example response

  1. ...
  2. "aggregations" : {
  3.   "significant_response_codes" : {
  4.     "doc_count" : 2737,
  5.     "bg_count" : 14074,
  6.     "buckets" : [
  7.       {
  8.         "key" : "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
  9.         "doc_count" : 818,
  10.         "score" : 0.01462731514608217,
  11.         "bg_count" : 4010
  12.       },
  13.       {
  14.         "key" : "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1",
  15.         "doc_count" : 1067,
  16.         "score" : 0.009062566630410223,
  17.         "bg_count" : 5362
  18.       }
  19.     ]
  20.   }
  21.  }
  22. }

If the significant_terms aggregation doesn’t return any result, you might have not filtered the results with a query. Alternatively, the distribution of terms in the foreground set might be the same as the background set, implying that there isn’t anything unusual in the foreground set.

The default source of statistical information for background term frequencies is the entire index. You can narrow this scope with a background filter for more focus