Inference bucket aggregation

Inference bucket aggregation

A parent pipeline aggregation which loads a pre-trained model and performs inference on the collated result fields from the parent bucket aggregation.

To use the inference bucket aggregation, you need to have the same security privileges that are required for using the get trained models API.

Syntax

A inference aggregation looks like this in isolation:

  1. {
  2.   "inference": {
  3.     "model_id": "a_model_for_inference", 
  4.     "inference_config": { 
  5.       "regression_config": {
  6.         "num_top_feature_importance_values": 2
  7.       }
  8.     },
  9.     "buckets_path": {
  10.       "avg_cost": "avg_agg", 
  11.       "max_cost": "max_agg"
  12.     }
  13.   }
  14. }

The unique identifier or alias for the trained model.

The optional inference config which overrides the model’s default settings

Map the value of avg_agg to the model’s input field avg_cost

Table 63. inference Parameters

Parameter NameDescriptionRequiredDefault Value

model_id

The ID or alias for the trained model.

Required

-

inference_config

Contains the inference type and its options. There are two types: regression and classification

Optional

-

buckets_path

Defines the paths to the input aggregations and maps the aggregation names to the field names expected by the model. See buckets_path Syntax for more details

Required

-

Configuration options for inference models

The inference_config setting is optional and usually isn’t required as the pre-trained models come equipped with sensible defaults. In the context of aggregations some options can be overridden for each of the two types of model.

Configuration options for regression models

num_top_feature_importance_values

(Optional, integer) Specifies the maximum number of feature importance values per document. By default, it is zero and no feature importance calculation occurs.

Configuration options for classification models

num_top_classes

(Optional, integer) Specifies the number of top class predictions to return. Defaults to 0.

num_top_feature_importance_values

(Optional, integer) Specifies the maximum number of feature importance values per document. Defaults to 0 which means no feature importance calculation occurs.

prediction_field_type

(Optional, string) Specifies the type of the predicted field to write. Valid values are: string, number, boolean. When boolean is provided 1.0 is transformed to true and 0.0 to false.

Example

The following snippet aggregates a web log by client_ip and extracts a number of features via metric and bucket sub-aggregations as input to the inference aggregation configured with a model trained to identify suspicious client IPs:

  1. resp = client.search(
  2.     index="kibana_sample_data_logs",
  3.     size=0,
  4.     aggs={
  5.         "client_ip": {
  6.             "composite": {
  7.                 "sources": [
  8.                     {
  9.                         "client_ip": {
  10.                             "terms": {
  11.                                 "field": "clientip"
  12.                             }
  13.                         }
  14.                     }
  15.                 ]
  16.             },
  17.             "aggs": {
  18.                 "url_dc": {
  19.                     "cardinality": {
  20.                         "field": "url.keyword"
  21.                     }
  22.                 },
  23.                 "bytes_sum": {
  24.                     "sum": {
  25.                         "field": "bytes"
  26.                     }
  27.                 },
  28.                 "geo_src_dc": {
  29.                     "cardinality": {
  30.                         "field": "geo.src"
  31.                     }
  32.                 },
  33.                 "geo_dest_dc": {
  34.                     "cardinality": {
  35.                         "field": "geo.dest"
  36.                     }
  37.                 },
  38.                 "responses_total": {
  39.                     "value_count": {
  40.                         "field": "timestamp"
  41.                     }
  42.                 },
  43.                 "success": {
  44.                     "filter": {
  45.                         "term": {
  46.                             "response": "200"
  47.                         }
  48.                     }
  49.                 },
  50.                 "error404": {
  51.                     "filter": {
  52.                         "term": {
  53.                             "response": "404"
  54.                         }
  55.                     }
  56.                 },
  57.                 "error503": {
  58.                     "filter": {
  59.                         "term": {
  60.                             "response": "503"
  61.                         }
  62.                     }
  63.                 },
  64.                 "malicious_client_ip": {
  65.                     "inference": {
  66.                         "model_id": "malicious_clients_model",
  67.                         "buckets_path": {
  68.                             "response_count": "responses_total",
  69.                             "url_dc": "url_dc",
  70.                             "bytes_sum": "bytes_sum",
  71.                             "geo_src_dc": "geo_src_dc",
  72.                             "geo_dest_dc": "geo_dest_dc",
  73.                             "success": "success._count",
  74.                             "error404": "error404._count",
  75.                             "error503": "error503._count"
  76.                         }
  77.                     }
  78.                 }
  79.             }
  80.         }
  81.     },
  82. )
  83. print(resp)
  1. response = client.search(
  2.   index: 'kibana_sample_data_logs',
  3.   body: {
  4.     size: 0,
  5.     aggregations: {
  6.       client_ip: {
  7.         composite: {
  8.           sources: [
  9.             {
  10.               client_ip: {
  11.                 terms: {
  12.                   field: 'clientip'
  13.                 }
  14.               }
  15.             }
  16.           ]
  17.         },
  18.         aggregations: {
  19.           url_dc: {
  20.             cardinality: {
  21.               field: 'url.keyword'
  22.             }
  23.           },
  24.           bytes_sum: {
  25.             sum: {
  26.               field: 'bytes'
  27.             }
  28.           },
  29.           geo_src_dc: {
  30.             cardinality: {
  31.               field: 'geo.src'
  32.             }
  33.           },
  34.           geo_dest_dc: {
  35.             cardinality: {
  36.               field: 'geo.dest'
  37.             }
  38.           },
  39.           responses_total: {
  40.             value_count: {
  41.               field: 'timestamp'
  42.             }
  43.           },
  44.           success: {
  45.             filter: {
  46.               term: {
  47.                 response: '200'
  48.               }
  49.             }
  50.           },
  51.           "error404": {
  52.             filter: {
  53.               term: {
  54.                 response: '404'
  55.               }
  56.             }
  57.           },
  58.           "error503": {
  59.             filter: {
  60.               term: {
  61.                 response: '503'
  62.               }
  63.             }
  64.           },
  65.           malicious_client_ip: {
  66.             inference: {
  67.               model_id: 'malicious_clients_model',
  68.               buckets_path: {
  69.                 response_count: 'responses_total',
  70.                 url_dc: 'url_dc',
  71.                 bytes_sum: 'bytes_sum',
  72.                 geo_src_dc: 'geo_src_dc',
  73.                 geo_dest_dc: 'geo_dest_dc',
  74.                 success: 'success._count',
  75.                 "error404": 'error404._count',
  76.                 "error503": 'error503._count'
  77.               }
  78.             }
  79.           }
  80.         }
  81.       }
  82.     }
  83.   }
  84. )
  85. puts response
  1. const response = await client.search({
  2.   index: "kibana_sample_data_logs",
  3.   size: 0,
  4.   aggs: {
  5.     client_ip: {
  6.       composite: {
  7.         sources: [
  8.           {
  9.             client_ip: {
  10.               terms: {
  11.                 field: "clientip",
  12.               },
  13.             },
  14.           },
  15.         ],
  16.       },
  17.       aggs: {
  18.         url_dc: {
  19.           cardinality: {
  20.             field: "url.keyword",
  21.           },
  22.         },
  23.         bytes_sum: {
  24.           sum: {
  25.             field: "bytes",
  26.           },
  27.         },
  28.         geo_src_dc: {
  29.           cardinality: {
  30.             field: "geo.src",
  31.           },
  32.         },
  33.         geo_dest_dc: {
  34.           cardinality: {
  35.             field: "geo.dest",
  36.           },
  37.         },
  38.         responses_total: {
  39.           value_count: {
  40.             field: "timestamp",
  41.           },
  42.         },
  43.         success: {
  44.           filter: {
  45.             term: {
  46.               response: "200",
  47.             },
  48.           },
  49.         },
  50.         error404: {
  51.           filter: {
  52.             term: {
  53.               response: "404",
  54.             },
  55.           },
  56.         },
  57.         error503: {
  58.           filter: {
  59.             term: {
  60.               response: "503",
  61.             },
  62.           },
  63.         },
  64.         malicious_client_ip: {
  65.           inference: {
  66.             model_id: "malicious_clients_model",
  67.             buckets_path: {
  68.               response_count: "responses_total",
  69.               url_dc: "url_dc",
  70.               bytes_sum: "bytes_sum",
  71.               geo_src_dc: "geo_src_dc",
  72.               geo_dest_dc: "geo_dest_dc",
  73.               success: "success._count",
  74.               error404: "error404._count",
  75.               error503: "error503._count",
  76.             },
  77.           },
  78.         },
  79.       },
  80.     },
  81.   },
  82. });
  83. console.log(response);
  1. GET kibana_sample_data_logs/_search
  2. {
  3.   "size": 0,
  4.   "aggs": {
  5.     "client_ip": { 
  6.       "composite": {
  7.         "sources": [
  8.           {
  9.             "client_ip": {
  10.               "terms": {
  11.                 "field": "clientip"
  12.               }
  13.             }
  14.           }
  15.         ]
  16.       },
  17.       "aggs": { 
  18.         "url_dc": {
  19.           "cardinality": {
  20.             "field": "url.keyword"
  21.           }
  22.         },
  23.         "bytes_sum": {
  24.           "sum": {
  25.             "field": "bytes"
  26.           }
  27.         },
  28.         "geo_src_dc": {
  29.           "cardinality": {
  30.             "field": "geo.src"
  31.           }
  32.         },
  33.         "geo_dest_dc": {
  34.           "cardinality": {
  35.             "field": "geo.dest"
  36.           }
  37.         },
  38.         "responses_total": {
  39.           "value_count": {
  40.             "field": "timestamp"
  41.           }
  42.         },
  43.         "success": {
  44.           "filter": {
  45.             "term": {
  46.               "response": "200"
  47.             }
  48.           }
  49.         },
  50.         "error404": {
  51.           "filter": {
  52.             "term": {
  53.               "response": "404"
  54.             }
  55.           }
  56.         },
  57.         "error503": {
  58.           "filter": {
  59.             "term": {
  60.               "response": "503"
  61.             }
  62.           }
  63.         },
  64.         "malicious_client_ip": { 
  65.           "inference": {
  66.             "model_id": "malicious_clients_model",
  67.             "buckets_path": {
  68.               "response_count": "responses_total",
  69.               "url_dc": "url_dc",
  70.               "bytes_sum": "bytes_sum",
  71.               "geo_src_dc": "geo_src_dc",
  72.               "geo_dest_dc": "geo_dest_dc",
  73.               "success": "success._count",
  74.               "error404": "error404._count",
  75.               "error503": "error503._count"
  76.             }
  77.           }
  78.         }
  79.       }
  80.     }
  81.   }
  82. }

A composite bucket aggregation that aggregates the data by client_ip.

A series of metrics and bucket sub-aggregations.

Inference bucket aggregation that specifies the trained model and maps the aggregation names to the model’s input fields.