Frequent item sets aggregation

Frequent item sets aggregation

A bucket aggregation which finds frequent item sets. It is a form of association rules mining that identifies items that often occur together. Items that are frequently purchased together or log events that tend to co-occur are examples of frequent item sets. Finding frequent item sets helps to discover relationships between different data points (items).

The aggregation reports closed item sets. A frequent item set is called closed if no superset exists with the same ratio of documents (also known as its support value). For example, we have the two following candidates for a frequent item set, which have the same support value: 1. apple, orange, banana 2. apple, orange, banana, tomato. Only the second item set (apple, orange, banana, tomato) is returned, and the first set – which is a subset of the second one – is skipped. Both item sets might be returned if their support values are different.

The runtime of the aggregation depends on the data and the provided parameters. It might take a significant time for the aggregation to complete. For this reason, it is recommended to use async search to run your requests asynchronously.

Syntax

A frequent_item_sets aggregation looks like this in isolation:

  1. "frequent_item_sets": {
  2.   "minimum_set_size": 3,
  3.   "fields": [
  4.     {"field": "my_field_1"},
  5.     {"field": "my_field_2"}
  6.   ]
  7. }

Table 51. frequent_item_sets Parameters

Parameter Name

Description

Required

Default Value

fields

(array) Fields to analyze.

Required

minimum_set_size

(integer) The minimum size of one item set.

Optional

1

minimum_support

(integer) The minimum support of one item set.

Optional

0.1

size

(integer) The number of top item sets to return.

Optional

10

filter

(object) Query that filters documents from the analysis

Optional

match_all

Fields

Supported field types for the analyzed fields are keyword, numeric, ip, date, and arrays of these types. You can also add runtime fields to your analyzed fields.

If the combined cardinality of the analyzed fields are high, the aggregation might require a significant amount of system resources.

You can filter the values for each field by using the include and exclude parameters. The parameters can be regular expression strings or arrays of strings of exact terms. The filtered values are removed from the analysis and therefore reduce the runtime. If both include and exclude are defined, exclude takes precedence; it means include is evaluated first and then exclude.

Minimum set size

The minimum set size is the minimum number of items the set needs to contain. A value of 1 returns the frequency of single items. Only item sets that contain at least the number of minimum_set_size items are returned. For example, the item set orange, banana, apple is returned only if the minimum set size is 3 or lower.

Minimum support

The minimum support value is the ratio of documents that an item set must exist in to be considered “frequent”. In particular, it is a normalized value between 0 and 1. It is calculated by dividing the number of documents containing the item set by the total number of documents.

For example, if a given item set is contained by five documents and the total number of documents is 20, then the support of the item set is 5/20 = 0.25. Therefore, this set is returned only if the minimum support is 0.25 or lower. As a higher minimum support prunes more items, the calculation is less resource intensive. The minimum_support parameter has an effect on the required memory and the runtime of the aggregation.

Size

This parameter defines the maximum number of item sets to return. The result contains top-k item sets; the item sets with the highest support values. This parameter has a significant effect on the required memory and the runtime of the aggregation.

Filter

A query to filter documents to use as part of the analysis. Documents that don’t match the filter are ignored when generating the item sets, however still count when calculating the support of an item set.

Use the filter if you want to narrow the item set analysis to fields of interest. Use a top-level query to filter the data set.

Examples

In the following examples, we use the e-commerce Kibana sample data set.

Aggregation with two analyzed fields and an exclude parameter

In the first example, the goal is to find out based on transaction data (1.) from what product categories the customers purchase products frequently together and (2.) from which cities they make those purchases. We want to exclude results where location information is not available (where the city name is other). Finally, we are interested in sets with three or more items, and want to see the first three frequent item sets with the highest support.

Note that we use the async search endpoint in this first example.

  1. resp = client.async_search.submit(
  2.     index="kibana_sample_data_ecommerce",
  3.     size=0,
  4.     aggs={
  5.         "my_agg": {
  6.             "frequent_item_sets": {
  7.                 "minimum_set_size": 3,
  8.                 "fields": [
  9.                     {
  10.                         "field": "category.keyword"
  11.                     },
  12.                     {
  13.                         "field": "geoip.city_name",
  14.                         "exclude": "other"
  15.                     }
  16.                 ],
  17.                 "size": 3
  18.             }
  19.         }
  20.     },
  21. )
  22. print(resp)
  1. const response = await client.asyncSearch.submit({
  2.   index: "kibana_sample_data_ecommerce",
  3.   size: 0,
  4.   aggs: {
  5.     my_agg: {
  6.       frequent_item_sets: {
  7.         minimum_set_size: 3,
  8.         fields: [
  9.           {
  10.             field: "category.keyword",
  11.           },
  12.           {
  13.             field: "geoip.city_name",
  14.             exclude: "other",
  15.           },
  16.         ],
  17.         size: 3,
  18.       },
  19.     },
  20.   },
  21. });
  22. console.log(response);
  1. POST /kibana_sample_data_ecommerce/_async_search
  2. {
  3.    "size":0,
  4.    "aggs":{
  5.       "my_agg":{
  6.          "frequent_item_sets":{
  7.             "minimum_set_size":3,
  8.             "fields":[
  9.                {
  10.                   "field":"category.keyword"
  11.                },
  12.                {
  13.                   "field":"geoip.city_name",
  14.                   "exclude":"other"
  15.                }
  16.             ],
  17.             "size":3
  18.          }
  19.       }
  20.    }
  21. }

The response of the API call above contains an identifier (id) of the async search request. You can use the identifier to retrieve the search results:

  1. resp = client.async_search.get(
  2.     id="<id>",
  3. )
  4. print(resp)
  1. const response = await client.asyncSearch.get({
  2.   id: "<id>",
  3. });
  4. console.log(response);
  1. GET /_async_search/<id>

The API returns a response similar to the following one:

  1. (...)
  2. "aggregations" : {
  3.     "my_agg" : {
  4.       "buckets" : [ 
  5.         {
  6.           "key" : { 
  7.             "category.keyword" : [
  8.               "Women's Clothing",
  9.               "Women's Shoes"
  10.             ],
  11.             "geoip.city_name" : [
  12.               "New York"
  13.             ]
  14.           },
  15.           "doc_count" : 217, 
  16.           "support" : 0.04641711229946524 
  17.         },
  18.         {
  19.           "key" : {
  20.             "category.keyword" : [
  21.               "Women's Clothing",
  22.               "Women's Accessories"
  23.             ],
  24.             "geoip.city_name" : [
  25.               "New York"
  26.             ]
  27.           },
  28.           "doc_count" : 135,
  29.           "support" : 0.028877005347593583
  30.         },
  31.         {
  32.           "key" : {
  33.             "category.keyword" : [
  34.               "Men's Clothing",
  35.               "Men's Shoes"
  36.             ],
  37.             "geoip.city_name" : [
  38.               "Cairo"
  39.             ]
  40.           },
  41.           "doc_count" : 123,
  42.           "support" : 0.026310160427807486
  43.         }
  44.       ],
  45.     (...)
  46.   }
  47. }

The array of returned item sets.

The key object contains one item set. In this case, it consists of two values of the category.keyword field and one value of the geoip.city_name.

The number of documents that contain the item set.

The support value of the item set. It is calculated by dividing the number of documents containing the item set by the total number of documents.

The response shows that the categories customers purchase from most frequently together are Women's Clothing and Women's Shoes and customers from New York tend to buy items from these categories frequently together. In other words, customers who buy products labelled Women's Clothing more likely buy products also from the Women's Shoes category and customers from New York most likely buy products from these categories together. The item set with the second highest support is Women's Clothing and Women's Accessories with customers mostly from New York. Finally, the item set with the third highest support is Men's Clothing and Men's Shoes with customers mostly from Cairo.

Aggregation with two analyzed fields and a filter

We take the first example, but want to narrow the item sets to places in Europe. For that, we add a filter, and this time, we don’t use the exclude parameter:

  1. resp = client.async_search.submit(
  2.     index="kibana_sample_data_ecommerce",
  3.     size=0,
  4.     aggs={
  5.         "my_agg": {
  6.             "frequent_item_sets": {
  7.                 "minimum_set_size": 3,
  8.                 "fields": [
  9.                     {
  10.                         "field": "category.keyword"
  11.                     },
  12.                     {
  13.                         "field": "geoip.city_name"
  14.                     }
  15.                 ],
  16.                 "size": 3,
  17.                 "filter": {
  18.                     "term": {
  19.                         "geoip.continent_name": "Europe"
  20.                     }
  21.                 }
  22.             }
  23.         }
  24.     },
  25. )
  26. print(resp)
  1. const response = await client.asyncSearch.submit({
  2.   index: "kibana_sample_data_ecommerce",
  3.   size: 0,
  4.   aggs: {
  5.     my_agg: {
  6.       frequent_item_sets: {
  7.         minimum_set_size: 3,
  8.         fields: [
  9.           {
  10.             field: "category.keyword",
  11.           },
  12.           {
  13.             field: "geoip.city_name",
  14.           },
  15.         ],
  16.         size: 3,
  17.         filter: {
  18.           term: {
  19.             "geoip.continent_name": "Europe",
  20.           },
  21.         },
  22.       },
  23.     },
  24.   },
  25. });
  26. console.log(response);
  1. POST /kibana_sample_data_ecommerce/_async_search
  2. {
  3.   "size": 0,
  4.   "aggs": {
  5.     "my_agg": {
  6.       "frequent_item_sets": {
  7.         "minimum_set_size": 3,
  8.         "fields": [
  9.           { "field": "category.keyword" },
  10.           { "field": "geoip.city_name" }
  11.         ],
  12.         "size": 3,
  13.         "filter": {
  14.           "term": {
  15.             "geoip.continent_name": "Europe"
  16.           }
  17.         }
  18.       }
  19.     }
  20.   }
  21. }

The result will only show item sets that created from documents matching the filter, namely purchases in Europe. Using filter, the calculated support still takes all purchases into acount. That’s different than specifying a query at the top-level, in which case support gets calculated only from purchases in Europe.

Analyzing numeric values by using a runtime field

The frequent items aggregation enables you to bucket numeric values by using runtime fields. The next example demonstrates how to use a script to add a runtime field to your documents called price_range, which is calculated from the taxful total price of the individual transactions. The runtime field then can be used in the frequent items aggregation as a field to analyze.

  1. resp = client.search(
  2.     index="kibana_sample_data_ecommerce",
  3.     runtime_mappings={
  4.         "price_range": {
  5.             "type": "keyword",
  6.             "script": {
  7.                 "source": "\n           def bucket_start = (long) Math.floor(doc['taxful_total_price'].value / 50) * 50;\n           def bucket_end = bucket_start + 50;\n           emit(bucket_start.toString() + \"-\" + bucket_end.toString());\n        "
  8.             }
  9.         }
  10.     },
  11.     size=0,
  12.     aggs={
  13.         "my_agg": {
  14.             "frequent_item_sets": {
  15.                 "minimum_set_size": 4,
  16.                 "fields": [
  17.                     {
  18.                         "field": "category.keyword"
  19.                     },
  20.                     {
  21.                         "field": "price_range"
  22.                     },
  23.                     {
  24.                         "field": "geoip.city_name"
  25.                     }
  26.                 ],
  27.                 "size": 3
  28.             }
  29.         }
  30.     },
  31. )
  32. print(resp)
  1. const response = await client.search({
  2.   index: "kibana_sample_data_ecommerce",
  3.   runtime_mappings: {
  4.     price_range: {
  5.       type: "keyword",
  6.       script: {
  7.         source:
  8.           "\n           def bucket_start = (long) Math.floor(doc['taxful_total_price'].value / 50) * 50;\n           def bucket_end = bucket_start + 50;\n           emit(bucket_start.toString() + \"-\" + bucket_end.toString());\n        ",
  9.       },
  10.     },
  11.   },
  12.   size: 0,
  13.   aggs: {
  14.     my_agg: {
  15.       frequent_item_sets: {
  16.         minimum_set_size: 4,
  17.         fields: [
  18.           {
  19.             field: "category.keyword",
  20.           },
  21.           {
  22.             field: "price_range",
  23.           },
  24.           {
  25.             field: "geoip.city_name",
  26.           },
  27.         ],
  28.         size: 3,
  29.       },
  30.     },
  31.   },
  32. });
  33. console.log(response);
  1. GET kibana_sample_data_ecommerce/_search
  2. {
  3.   "runtime_mappings": {
  4.     "price_range": {
  5.       "type": "keyword",
  6.       "script": {
  7.         "source": """
  8.            def bucket_start = (long) Math.floor(doc['taxful_total_price'].value / 50) * 50;
  9.            def bucket_end = bucket_start + 50;
  10.            emit(bucket_start.toString() + "-" + bucket_end.toString());
  11.         """
  12.       }
  13.     }
  14.   },
  15.   "size": 0,
  16.   "aggs": {
  17.     "my_agg": {
  18.       "frequent_item_sets": {
  19.         "minimum_set_size": 4,
  20.         "fields": [
  21.           {
  22.             "field": "category.keyword"
  23.           },
  24.           {
  25.             "field": "price_range"
  26.           },
  27.           {
  28.             "field": "geoip.city_name"
  29.           }
  30.         ],
  31.         "size": 3
  32.       }
  33.     }
  34.   }
  35. }

The API returns a response similar to the following one:

  1. (...)
  2. "aggregations" : {
  3.     "my_agg" : {
  4.       "buckets" : [
  5.         {
  6.           "key" : {
  7.             "category.keyword" : [
  8.               "Women's Clothing",
  9.               "Women's Shoes"
  10.             ],
  11.             "price_range" : [
  12.               "50-100"
  13.             ],
  14.             "geoip.city_name" : [
  15.               "New York"
  16.             ]
  17.           },
  18.           "doc_count" : 100,
  19.           "support" : 0.0213903743315508
  20.         },
  21.         {
  22.           "key" : {
  23.             "category.keyword" : [
  24.               "Women's Clothing",
  25.               "Women's Shoes"
  26.             ],
  27.             "price_range" : [
  28.               "50-100"
  29.             ],
  30.             "geoip.city_name" : [
  31.               "Dubai"
  32.             ]
  33.           },
  34.           "doc_count" : 59,
  35.           "support" : 0.012620320855614974
  36.         },
  37.         {
  38.           "key" : {
  39.             "category.keyword" : [
  40.               "Men's Clothing",
  41.               "Men's Shoes"
  42.             ],
  43.             "price_range" : [
  44.               "50-100"
  45.             ],
  46.             "geoip.city_name" : [
  47.               "Marrakesh"
  48.             ]
  49.           },
  50.           "doc_count" : 53,
  51.           "support" : 0.011336898395721925
  52.         }
  53.       ],
  54.     (...)
  55.     }
  56.   }

The response shows the categories that customers purchase from most frequently together, the location of the customers who tend to buy items from these categories, and the most frequent price ranges of these purchases.