Transform examples

Transform examples

These examples demonstrate how to use transforms to derive useful insights from your data. All the examples use one of the Kibana sample datasets. For a more detailed, step-by-step example, see Tutorial: Transforming the eCommerce sample data.

Finding your best customers

This example uses the eCommerce orders sample data set to find the customers who spent the most in a hypothetical webshop. Let’s use the pivot type of transform such that the destination index contains the number of orders, the total price of the orders, the amount of unique products and the average price per order, and the total amount of ordered products for each customer.

Finding your best customers with transforms in Kibana

Alternatively, you can use the preview transform and the create transform API.

API example

  1. resp = client.transform.preview_transform(
  2.     source={
  3.         "index": "kibana_sample_data_ecommerce"
  4.     },
  5.     dest={
  6.         "index": "sample_ecommerce_orders_by_customer"
  7.     },
  8.     pivot={
  9.         "group_by": {
  10.             "user": {
  11.                 "terms": {
  12.                     "field": "user"
  13.                 }
  14.             },
  15.             "customer_id": {
  16.                 "terms": {
  17.                     "field": "customer_id"
  18.                 }
  19.             }
  20.         },
  21.         "aggregations": {
  22.             "order_count": {
  23.                 "value_count": {
  24.                     "field": "order_id"
  25.                 }
  26.             },
  27.             "total_order_amt": {
  28.                 "sum": {
  29.                     "field": "taxful_total_price"
  30.                 }
  31.             },
  32.             "avg_amt_per_order": {
  33.                 "avg": {
  34.                     "field": "taxful_total_price"
  35.                 }
  36.             },
  37.             "avg_unique_products_per_order": {
  38.                 "avg": {
  39.                     "field": "total_unique_products"
  40.                 }
  41.             },
  42.             "total_unique_products": {
  43.                 "cardinality": {
  44.                     "field": "products.product_id"
  45.                 }
  46.             }
  47.         }
  48.     },
  49. )
  50. print(resp)
  1. const response = await client.transform.previewTransform({
  2.   source: {
  3.     index: "kibana_sample_data_ecommerce",
  4.   },
  5.   dest: {
  6.     index: "sample_ecommerce_orders_by_customer",
  7.   },
  8.   pivot: {
  9.     group_by: {
  10.       user: {
  11.         terms: {
  12.           field: "user",
  13.         },
  14.       },
  15.       customer_id: {
  16.         terms: {
  17.           field: "customer_id",
  18.         },
  19.       },
  20.     },
  21.     aggregations: {
  22.       order_count: {
  23.         value_count: {
  24.           field: "order_id",
  25.         },
  26.       },
  27.       total_order_amt: {
  28.         sum: {
  29.           field: "taxful_total_price",
  30.         },
  31.       },
  32.       avg_amt_per_order: {
  33.         avg: {
  34.           field: "taxful_total_price",
  35.         },
  36.       },
  37.       avg_unique_products_per_order: {
  38.         avg: {
  39.           field: "total_unique_products",
  40.         },
  41.       },
  42.       total_unique_products: {
  43.         cardinality: {
  44.           field: "products.product_id",
  45.         },
  46.       },
  47.     },
  48.   },
  49. });
  50. console.log(response);
  1. POST _transform/_preview
  2. {
  3.   "source": {
  4.     "index": "kibana_sample_data_ecommerce"
  5.   },
  6.   "dest" : { 
  7.     "index" : "sample_ecommerce_orders_by_customer"
  8.   },
  9.   "pivot": {
  10.     "group_by": { 
  11.       "user": { "terms": { "field": "user" }},
  12.       "customer_id": { "terms": { "field": "customer_id" }}
  13.     },
  14.     "aggregations": {
  15.       "order_count": { "value_count": { "field": "order_id" }},
  16.       "total_order_amt": { "sum": { "field": "taxful_total_price" }},
  17.       "avg_amt_per_order": { "avg": { "field": "taxful_total_price" }},
  18.       "avg_unique_products_per_order": { "avg": { "field": "total_unique_products" }},
  19.       "total_unique_products": { "cardinality": { "field": "products.product_id" }}
  20.     }
  21.   }
  22. }

The destination index for the transform. It is ignored by _preview.

Two group_by fields is selected. This means the transform contains a unique row per user and customer_id combination. Within this data set, both these fields are unique. By including both in the transform, it gives more context to the final results.

In the example above, condensed JSON formatting is used for easier readability of the pivot object.

The preview transforms API enables you to see the layout of the transform in advance, populated with some sample values. For example:

  1. {
  2.   "preview" : [
  3.     {
  4.       "total_order_amt" : 3946.9765625,
  5.       "order_count" : 59.0,
  6.       "total_unique_products" : 116.0,
  7.       "avg_unique_products_per_order" : 2.0,
  8.       "customer_id" : "10",
  9.       "user" : "recip",
  10.       "avg_amt_per_order" : 66.89790783898304
  11.     },
  12.     ...
  13.     ]
  14.   }

This transform makes it easier to answer questions such as:

  • Which customers spend the most?
  • Which customers spend the most per order?
  • Which customers order most often?
  • Which customers ordered the least number of different products?

It’s possible to answer these questions using aggregations alone, however transforms allow us to persist this data as a customer centric index. This enables us to analyze data at scale and gives more flexibility to explore and navigate data from a customer centric perspective. In some cases, it can even make creating visualizations much simpler.

Finding air carriers with the most delays

This example uses the Flights sample data set to find out which air carrier had the most delays. First, filter the source data such that it excludes all the cancelled flights by using a query filter. Then transform the data to contain the distinct number of flights, the sum of delayed minutes, and the sum of the flight minutes by air carrier. Finally, use a bucket_script to determine what percentage of the flight time was actually delay.

  1. resp = client.transform.preview_transform(
  2.     source={
  3.         "index": "kibana_sample_data_flights",
  4.         "query": {
  5.             "bool": {
  6.                 "filter": [
  7.                     {
  8.                         "term": {
  9.                             "Cancelled": False
  10.                         }
  11.                     }
  12.                 ]
  13.             }
  14.         }
  15.     },
  16.     dest={
  17.         "index": "sample_flight_delays_by_carrier"
  18.     },
  19.     pivot={
  20.         "group_by": {
  21.             "carrier": {
  22.                 "terms": {
  23.                     "field": "Carrier"
  24.                 }
  25.             }
  26.         },
  27.         "aggregations": {
  28.             "flights_count": {
  29.                 "value_count": {
  30.                     "field": "FlightNum"
  31.                 }
  32.             },
  33.             "delay_mins_total": {
  34.                 "sum": {
  35.                     "field": "FlightDelayMin"
  36.                 }
  37.             },
  38.             "flight_mins_total": {
  39.                 "sum": {
  40.                     "field": "FlightTimeMin"
  41.                 }
  42.             },
  43.             "delay_time_percentage": {
  44.                 "bucket_script": {
  45.                     "buckets_path": {
  46.                         "delay_time": "delay_mins_total.value",
  47.                         "flight_time": "flight_mins_total.value"
  48.                     },
  49.                     "script": "(params.delay_time / params.flight_time) * 100"
  50.                 }
  51.             }
  52.         }
  53.     },
  54. )
  55. print(resp)
  1. const response = await client.transform.previewTransform({
  2.   source: {
  3.     index: "kibana_sample_data_flights",
  4.     query: {
  5.       bool: {
  6.         filter: [
  7.           {
  8.             term: {
  9.               Cancelled: false,
  10.             },
  11.           },
  12.         ],
  13.       },
  14.     },
  15.   },
  16.   dest: {
  17.     index: "sample_flight_delays_by_carrier",
  18.   },
  19.   pivot: {
  20.     group_by: {
  21.       carrier: {
  22.         terms: {
  23.           field: "Carrier",
  24.         },
  25.       },
  26.     },
  27.     aggregations: {
  28.       flights_count: {
  29.         value_count: {
  30.           field: "FlightNum",
  31.         },
  32.       },
  33.       delay_mins_total: {
  34.         sum: {
  35.           field: "FlightDelayMin",
  36.         },
  37.       },
  38.       flight_mins_total: {
  39.         sum: {
  40.           field: "FlightTimeMin",
  41.         },
  42.       },
  43.       delay_time_percentage: {
  44.         bucket_script: {
  45.           buckets_path: {
  46.             delay_time: "delay_mins_total.value",
  47.             flight_time: "flight_mins_total.value",
  48.           },
  49.           script: "(params.delay_time / params.flight_time) * 100",
  50.         },
  51.       },
  52.     },
  53.   },
  54. });
  55. console.log(response);
  1. POST _transform/_preview
  2. {
  3.   "source": {
  4.     "index": "kibana_sample_data_flights",
  5.     "query": { 
  6.       "bool": {
  7.         "filter": [
  8.           { "term":  { "Cancelled": false } }
  9.         ]
  10.       }
  11.     }
  12.   },
  13.   "dest" : { 
  14.     "index" : "sample_flight_delays_by_carrier"
  15.   },
  16.   "pivot": {
  17.     "group_by": { 
  18.       "carrier": { "terms": { "field": "Carrier" }}
  19.     },
  20.     "aggregations": {
  21.       "flights_count": { "value_count": { "field": "FlightNum" }},
  22.       "delay_mins_total": { "sum": { "field": "FlightDelayMin" }},
  23.       "flight_mins_total": { "sum": { "field": "FlightTimeMin" }},
  24.       "delay_time_percentage": { 
  25.         "bucket_script": {
  26.           "buckets_path": {
  27.             "delay_time": "delay_mins_total.value",
  28.             "flight_time": "flight_mins_total.value"
  29.           },
  30.           "script": "(params.delay_time / params.flight_time) * 100"
  31.         }
  32.       }
  33.     }
  34.   }
  35. }

Filter the source data to select only flights that are not cancelled.

The destination index for the transform. It is ignored by _preview.

The data is grouped by the Carrier field which contains the airline name.

This bucket_script performs calculations on the results that are returned by the aggregation. In this particular example, it calculates what percentage of travel time was taken up by delays.

The preview shows you that the new index would contain data like this for each carrier:

  1. {
  2.   "preview" : [
  3.     {
  4.       "carrier" : "ES-Air",
  5.       "flights_count" : 2802.0,
  6.       "flight_mins_total" : 1436927.5130677223,
  7.       "delay_time_percentage" : 9.335543983955839,
  8.       "delay_mins_total" : 134145.0
  9.     },
  10.     ...
  11.   ]
  12. }

This transform makes it easier to answer questions such as:

  • Which air carrier has the most delays as a percentage of flight time?

This data is fictional and does not reflect actual delays or flight stats for any of the featured destination or origin airports.

Finding suspicious client IPs

This example uses the web log sample data set to identify suspicious client IPs. It transforms the data such that the new index contains the sum of bytes and the number of distinct URLs, agents, incoming requests by location, and geographic destinations for each client IP. It also uses filter aggregations to count the specific types of HTTP responses that each client IP receives. Ultimately, the example below transforms web log data into an entity centric index where the entity is clientip.

  1. resp = client.transform.put_transform(
  2.     transform_id="suspicious_client_ips",
  3.     source={
  4.         "index": "kibana_sample_data_logs"
  5.     },
  6.     dest={
  7.         "index": "sample_weblogs_by_clientip"
  8.     },
  9.     sync={
  10.         "time": {
  11.             "field": "timestamp",
  12.             "delay": "60s"
  13.         }
  14.     },
  15.     pivot={
  16.         "group_by": {
  17.             "clientip": {
  18.                 "terms": {
  19.                     "field": "clientip"
  20.                 }
  21.             }
  22.         },
  23.         "aggregations": {
  24.             "url_dc": {
  25.                 "cardinality": {
  26.                     "field": "url.keyword"
  27.                 }
  28.             },
  29.             "bytes_sum": {
  30.                 "sum": {
  31.                     "field": "bytes"
  32.                 }
  33.             },
  34.             "geo.src_dc": {
  35.                 "cardinality": {
  36.                     "field": "geo.src"
  37.                 }
  38.             },
  39.             "agent_dc": {
  40.                 "cardinality": {
  41.                     "field": "agent.keyword"
  42.                 }
  43.             },
  44.             "geo.dest_dc": {
  45.                 "cardinality": {
  46.                     "field": "geo.dest"
  47.                 }
  48.             },
  49.             "responses.total": {
  50.                 "value_count": {
  51.                     "field": "timestamp"
  52.                 }
  53.             },
  54.             "success": {
  55.                 "filter": {
  56.                     "term": {
  57.                         "response": "200"
  58.                     }
  59.                 }
  60.             },
  61.             "error404": {
  62.                 "filter": {
  63.                     "term": {
  64.                         "response": "404"
  65.                     }
  66.                 }
  67.             },
  68.             "error5xx": {
  69.                 "filter": {
  70.                     "range": {
  71.                         "response": {
  72.                             "gte": 500,
  73.                             "lt": 600
  74.                         }
  75.                     }
  76.                 }
  77.             },
  78.             "timestamp.min": {
  79.                 "min": {
  80.                     "field": "timestamp"
  81.                 }
  82.             },
  83.             "timestamp.max": {
  84.                 "max": {
  85.                     "field": "timestamp"
  86.                 }
  87.             },
  88.             "timestamp.duration_ms": {
  89.                 "bucket_script": {
  90.                     "buckets_path": {
  91.                         "min_time": "timestamp.min.value",
  92.                         "max_time": "timestamp.max.value"
  93.                     },
  94.                     "script": "(params.max_time - params.min_time)"
  95.                 }
  96.             }
  97.         }
  98.     },
  99. )
  100. print(resp)
  1. const response = await client.transform.putTransform({
  2.   transform_id: "suspicious_client_ips",
  3.   source: {
  4.     index: "kibana_sample_data_logs",
  5.   },
  6.   dest: {
  7.     index: "sample_weblogs_by_clientip",
  8.   },
  9.   sync: {
  10.     time: {
  11.       field: "timestamp",
  12.       delay: "60s",
  13.     },
  14.   },
  15.   pivot: {
  16.     group_by: {
  17.       clientip: {
  18.         terms: {
  19.           field: "clientip",
  20.         },
  21.       },
  22.     },
  23.     aggregations: {
  24.       url_dc: {
  25.         cardinality: {
  26.           field: "url.keyword",
  27.         },
  28.       },
  29.       bytes_sum: {
  30.         sum: {
  31.           field: "bytes",
  32.         },
  33.       },
  34.       "geo.src_dc": {
  35.         cardinality: {
  36.           field: "geo.src",
  37.         },
  38.       },
  39.       agent_dc: {
  40.         cardinality: {
  41.           field: "agent.keyword",
  42.         },
  43.       },
  44.       "geo.dest_dc": {
  45.         cardinality: {
  46.           field: "geo.dest",
  47.         },
  48.       },
  49.       "responses.total": {
  50.         value_count: {
  51.           field: "timestamp",
  52.         },
  53.       },
  54.       success: {
  55.         filter: {
  56.           term: {
  57.             response: "200",
  58.           },
  59.         },
  60.       },
  61.       error404: {
  62.         filter: {
  63.           term: {
  64.             response: "404",
  65.           },
  66.         },
  67.       },
  68.       error5xx: {
  69.         filter: {
  70.           range: {
  71.             response: {
  72.               gte: 500,
  73.               lt: 600,
  74.             },
  75.           },
  76.         },
  77.       },
  78.       "timestamp.min": {
  79.         min: {
  80.           field: "timestamp",
  81.         },
  82.       },
  83.       "timestamp.max": {
  84.         max: {
  85.           field: "timestamp",
  86.         },
  87.       },
  88.       "timestamp.duration_ms": {
  89.         bucket_script: {
  90.           buckets_path: {
  91.             min_time: "timestamp.min.value",
  92.             max_time: "timestamp.max.value",
  93.           },
  94.           script: "(params.max_time - params.min_time)",
  95.         },
  96.       },
  97.     },
  98.   },
  99. });
  100. console.log(response);
  1. PUT _transform/suspicious_client_ips
  2. {
  3.   "source": {
  4.     "index": "kibana_sample_data_logs"
  5.   },
  6.   "dest" : { 
  7.     "index" : "sample_weblogs_by_clientip"
  8.   },
  9.   "sync" : { 
  10.     "time": {
  11.       "field": "timestamp",
  12.       "delay": "60s"
  13.     }
  14.   },
  15.   "pivot": {
  16.     "group_by": {  
  17.       "clientip": { "terms": { "field": "clientip" } }
  18.       },
  19.     "aggregations": {
  20.       "url_dc": { "cardinality": { "field": "url.keyword" }},
  21.       "bytes_sum": { "sum": { "field": "bytes" }},
  22.       "geo.src_dc": { "cardinality": { "field": "geo.src" }},
  23.       "agent_dc": { "cardinality": { "field": "agent.keyword" }},
  24.       "geo.dest_dc": { "cardinality": { "field": "geo.dest" }},
  25.       "responses.total": { "value_count": { "field": "timestamp" }},
  26.       "success" : { 
  27.          "filter": {
  28.             "term": { "response" : "200"}}
  29.         },
  30.       "error404" : {
  31.          "filter": {
  32.             "term": { "response" : "404"}}
  33.         },
  34.       "error5xx" : {
  35.          "filter": {
  36.             "range": { "response" : { "gte": 500, "lt": 600}}}
  37.         },
  38.       "timestamp.min": { "min": { "field": "timestamp" }},
  39.       "timestamp.max": { "max": { "field": "timestamp" }},
  40.       "timestamp.duration_ms": { 
  41.         "bucket_script": {
  42.           "buckets_path": {
  43.             "min_time": "timestamp.min.value",
  44.             "max_time": "timestamp.max.value"
  45.           },
  46.           "script": "(params.max_time - params.min_time)"
  47.         }
  48.       }
  49.     }
  50.   }
  51. }

The destination index for the transform.

Configures the transform to run continuously. It uses the timestamp field to synchronize the source and destination indices. The worst case ingestion delay is 60 seconds.

The data is grouped by the clientip field.

Filter aggregation that counts the occurrences of successful (200) responses in the response field. The following two aggregations (error404 and error5xx) count the error responses by error codes, matching an exact value or a range of response codes.

This bucket_script calculates the duration of the clientip access based on the results of the aggregation.

After you create the transform, you must start it:

  1. resp = client.transform.start_transform(
  2.     transform_id="suspicious_client_ips",
  3. )
  4. print(resp)
  1. response = client.transform.start_transform(
  2.   transform_id: 'suspicious_client_ips'
  3. )
  4. puts response
  1. const response = await client.transform.startTransform({
  2.   transform_id: "suspicious_client_ips",
  3. });
  4. console.log(response);
  1. POST _transform/suspicious_client_ips/_start

Shortly thereafter, the first results should be available in the destination index:

  1. resp = client.search(
  2.     index="sample_weblogs_by_clientip",
  3. )
  4. print(resp)
  1. response = client.search(
  2.   index: 'sample_weblogs_by_clientip'
  3. )
  4. puts response
  1. const response = await client.search({
  2.   index: "sample_weblogs_by_clientip",
  3. });
  4. console.log(response);
  1. GET sample_weblogs_by_clientip/_search

The search result shows you data like this for each client IP:

  1.     "hits" : [
  2.       {
  3.         "_index" : "sample_weblogs_by_clientip",
  4.         "_id" : "MOeHH_cUL5urmartKj-b5UQAAAAAAAAA",
  5.         "_score" : 1.0,
  6.         "_source" : {
  7.           "geo" : {
  8.             "src_dc" : 2.0,
  9.             "dest_dc" : 2.0
  10.           },
  11.           "success" : 2,
  12.           "error404" : 0,
  13.           "error503" : 0,
  14.           "clientip" : "0.72.176.46",
  15.           "agent_dc" : 2.0,
  16.           "bytes_sum" : 4422.0,
  17.           "responses" : {
  18.             "total" : 2.0
  19.           },
  20.           "url_dc" : 2.0,
  21.           "timestamp" : {
  22.             "duration_ms" : 5.2191698E8,
  23.             "min" : "2020-03-16T07:51:57.333Z",
  24.             "max" : "2020-03-22T08:50:34.313Z"
  25.           }
  26.         }
  27.       }
  28.     ]

Like other Kibana sample data sets, the web log sample dataset contains timestamps relative to when you installed it, including timestamps in the future. The continuous transform will pick up the data points once they are in the past. If you installed the web log sample dataset some time ago, you can uninstall and reinstall it and the timestamps will change.

This transform makes it easier to answer questions such as:

  • Which client IPs are transferring the most amounts of data?
  • Which client IPs are interacting with a high number of different URLs?
  • Which client IPs have high error rates?
  • Which client IPs are interacting with a high number of destination countries?

Finding the last log event for each IP address

This example uses the web log sample data set to find the last log from an IP address. Let’s use the latest type of transform in continuous mode. It copies the most recent document for each unique key from the source index to the destination index and updates the destination index as new data comes into the source index.

Pick the clientip field as the unique key; the data is grouped by this field. Select timestamp as the date field that sorts the data chronologically. For continuous mode, specify a date field that is used to identify new documents, and an interval between checks for changes in the source index.

Finding the last log event for each IP address with transforms in Kibana

Let’s assume that we’re interested in retaining documents only for IP addresses that appeared recently in the log. You can define a retention policy and specify a date field that is used to calculate the age of a document. This example uses the same date field that is used to sort the data. Then set the maximum age of a document; documents that are older than the value you set will be removed from the destination index.

Defining retention policy for transforms in Kibana

This transform creates the destination index that contains the latest login date for each client IP. As the transform runs in continuous mode, the destination index will be updated as new data that comes into the source index. Finally, every document that is older than 30 days will be removed from the destination index due to the applied retention policy.

API example

  1. resp = client.transform.put_transform(
  2.     transform_id="last-log-from-clientip",
  3.     source={
  4.         "index": [
  5.             "kibana_sample_data_logs"
  6.         ]
  7.     },
  8.     latest={
  9.         "unique_key": [
  10.             "clientip"
  11.         ],
  12.         "sort": "timestamp"
  13.     },
  14.     frequency="1m",
  15.     dest={
  16.         "index": "last-log-from-clientip"
  17.     },
  18.     sync={
  19.         "time": {
  20.             "field": "timestamp",
  21.             "delay": "60s"
  22.         }
  23.     },
  24.     retention_policy={
  25.         "time": {
  26.             "field": "timestamp",
  27.             "max_age": "30d"
  28.         }
  29.     },
  30.     settings={
  31.         "max_page_search_size": 500
  32.     },
  33. )
  34. print(resp)
  1. const response = await client.transform.putTransform({
  2.   transform_id: "last-log-from-clientip",
  3.   source: {
  4.     index: ["kibana_sample_data_logs"],
  5.   },
  6.   latest: {
  7.     unique_key: ["clientip"],
  8.     sort: "timestamp",
  9.   },
  10.   frequency: "1m",
  11.   dest: {
  12.     index: "last-log-from-clientip",
  13.   },
  14.   sync: {
  15.     time: {
  16.       field: "timestamp",
  17.       delay: "60s",
  18.     },
  19.   },
  20.   retention_policy: {
  21.     time: {
  22.       field: "timestamp",
  23.       max_age: "30d",
  24.     },
  25.   },
  26.   settings: {
  27.     max_page_search_size: 500,
  28.   },
  29. });
  30. console.log(response);
  1. PUT _transform/last-log-from-clientip
  2. {
  3.   "source": {
  4.     "index": [
  5.       "kibana_sample_data_logs"
  6.     ]
  7.   },
  8.   "latest": {
  9.     "unique_key": [ 
  10.       "clientip"
  11.     ],
  12.     "sort": "timestamp" 
  13.   },
  14.   "frequency": "1m", 
  15.   "dest": {
  16.     "index": "last-log-from-clientip"
  17.   },
  18.   "sync": { 
  19.     "time": {
  20.       "field": "timestamp",
  21.       "delay": "60s"
  22.     }
  23.   },
  24.   "retention_policy": { 
  25.     "time": {
  26.       "field": "timestamp",
  27.       "max_age": "30d"
  28.     }
  29.   },
  30.   "settings": {
  31.     "max_page_search_size": 500
  32.   }
  33. }

Specifies the field for grouping the data.

Specifies the date field that is used for sorting the data.

Sets the interval for the transform to check for changes in the source index.

Contains the time field and delay settings used to synchronize the source and destination indices.

Specifies the retention policy for the transform. Documents that are older than the configured value will be removed from the destination index.

After you create the transform, start it:

  1. resp = client.transform.start_transform(
  2.     transform_id="last-log-from-clientip",
  3. )
  4. print(resp)
  1. response = client.transform.start_transform(
  2.   transform_id: 'last-log-from-clientip'
  3. )
  4. puts response
  1. const response = await client.transform.startTransform({
  2.   transform_id: "last-log-from-clientip",
  3. });
  4. console.log(response);
  1. POST _transform/last-log-from-clientip/_start

After the transform processes the data, search the destination index:

  1. resp = client.search(
  2.     index="last-log-from-clientip",
  3. )
  4. print(resp)
  1. response = client.search(
  2.   index: 'last-log-from-clientip'
  3. )
  4. puts response
  1. const response = await client.search({
  2.   index: "last-log-from-clientip",
  3. });
  4. console.log(response);
  1. GET last-log-from-clientip/_search

The search result shows you data like this for each client IP:

  1. {
  2.   "_index" : "last-log-from-clientip",
  3.   "_id" : "MOeHH_cUL5urmartKj-b5UQAAAAAAAAA",
  4.   "_score" : 1.0,
  5.   "_source" : {
  6.     "referer" : "http://twitter.com/error/don-lind",
  7.     "request" : "/elasticsearch",
  8.     "agent" : "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
  9.     "extension" : "",
  10.     "memory" : null,
  11.     "ip" : "0.72.176.46",
  12.     "index" : "kibana_sample_data_logs",
  13.     "message" : "0.72.176.46 - - [2018-09-18T06:31:00.572Z] \"GET /elasticsearch HTTP/1.1\" 200 7065 \"-\" \"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)\"",
  14.     "url" : "https://www.elastic.co/downloads/elasticsearch",
  15.     "tags" : [
  16.       "success",
  17.       "info"
  18.     ],
  19.     "geo" : {
  20.       "srcdest" : "IN:PH",
  21.       "src" : "IN",
  22.       "coordinates" : {
  23.         "lon" : -124.1127917,
  24.         "lat" : 40.80338889
  25.       },
  26.       "dest" : "PH"
  27.     },
  28.     "utc_time" : "2021-05-04T06:31:00.572Z",
  29.     "bytes" : 7065,
  30.     "machine" : {
  31.       "os" : "ios",
  32.       "ram" : 12884901888
  33.     },
  34.     "response" : 200,
  35.     "clientip" : "0.72.176.46",
  36.     "host" : "www.elastic.co",
  37.     "event" : {
  38.       "dataset" : "sample_web_logs"
  39.     },
  40.     "phpmemory" : null,
  41.     "timestamp" : "2021-05-04T06:31:00.572Z"
  42.   }
  43. }

This transform makes it easier to answer questions such as:

  • What was the most recent log event associated with a specific IP address?

Finding client IPs that sent the most bytes to the server

This example uses the web log sample data set to find the client IP that sent the most bytes to the server in every hour. The example uses a pivot transform with a top_metrics aggregation.

Group the data by a date histogram on the time field with an interval of one hour. Use a max aggregation on the bytes field to get the maximum amount of data that is sent to the server. Without the max aggregation, the API call still returns the client IP that sent the most bytes, however, the amount of bytes that it sent is not returned. In the top_metrics property, specify clientip and geo.src, then sort them by the bytes field in descending order. The transform returns the client IP that sent the biggest amount of data and the 2-letter ISO code of the corresponding location.

  1. resp = client.transform.preview_transform(
  2.     source={
  3.         "index": "kibana_sample_data_logs"
  4.     },
  5.     pivot={
  6.         "group_by": {
  7.             "timestamp": {
  8.                 "date_histogram": {
  9.                     "field": "timestamp",
  10.                     "fixed_interval": "1h"
  11.                 }
  12.             }
  13.         },
  14.         "aggregations": {
  15.             "bytes.max": {
  16.                 "max": {
  17.                     "field": "bytes"
  18.                 }
  19.             },
  20.             "top": {
  21.                 "top_metrics": {
  22.                     "metrics": [
  23.                         {
  24.                             "field": "clientip"
  25.                         },
  26.                         {
  27.                             "field": "geo.src"
  28.                         }
  29.                     ],
  30.                     "sort": {
  31.                         "bytes": "desc"
  32.                     }
  33.                 }
  34.             }
  35.         }
  36.     },
  37. )
  38. print(resp)
  1. const response = await client.transform.previewTransform({
  2.   source: {
  3.     index: "kibana_sample_data_logs",
  4.   },
  5.   pivot: {
  6.     group_by: {
  7.       timestamp: {
  8.         date_histogram: {
  9.           field: "timestamp",
  10.           fixed_interval: "1h",
  11.         },
  12.       },
  13.     },
  14.     aggregations: {
  15.       "bytes.max": {
  16.         max: {
  17.           field: "bytes",
  18.         },
  19.       },
  20.       top: {
  21.         top_metrics: {
  22.           metrics: [
  23.             {
  24.               field: "clientip",
  25.             },
  26.             {
  27.               field: "geo.src",
  28.             },
  29.           ],
  30.           sort: {
  31.             bytes: "desc",
  32.           },
  33.         },
  34.       },
  35.     },
  36.   },
  37. });
  38. console.log(response);
  1. POST _transform/_preview
  2. {
  3.   "source": {
  4.     "index": "kibana_sample_data_logs"
  5.   },
  6.   "pivot": {
  7.     "group_by": { 
  8.       "timestamp": {
  9.         "date_histogram": {
  10.           "field": "timestamp",
  11.           "fixed_interval": "1h"
  12.         }
  13.       }
  14.     },
  15.     "aggregations": {
  16.       "bytes.max": { 
  17.         "max": {
  18.           "field": "bytes"
  19.         }
  20.       },
  21.       "top": {
  22.         "top_metrics": { 
  23.           "metrics": [
  24.             {
  25.               "field": "clientip"
  26.             },
  27.             {
  28.               "field": "geo.src"
  29.             }
  30.           ],
  31.           "sort": {
  32.             "bytes": "desc"
  33.           }
  34.         }
  35.       }
  36.     }
  37.   }
  38. }

The data is grouped by a date histogram of the time field with a one hour interval.

Calculates the maximum value of the bytes field.

Specifies the fields (clientip and geo.src) of the top document to return and the sorting method (document with the highest bytes value).

The API call above returns a response similar to this:

  1. {
  2.   "preview" : [
  3.     {
  4.       "top" : {
  5.         "clientip" : "223.87.60.27",
  6.         "geo.src" : "IN"
  7.       },
  8.       "bytes" : {
  9.         "max" : 6219
  10.       },
  11.       "timestamp" : "2021-04-25T00:00:00.000Z"
  12.     },
  13.     {
  14.       "top" : {
  15.         "clientip" : "99.74.118.237",
  16.         "geo.src" : "LK"
  17.       },
  18.       "bytes" : {
  19.         "max" : 14113
  20.       },
  21.       "timestamp" : "2021-04-25T03:00:00.000Z"
  22.     },
  23.     {
  24.       "top" : {
  25.         "clientip" : "218.148.135.12",
  26.         "geo.src" : "BR"
  27.       },
  28.       "bytes" : {
  29.         "max" : 4531
  30.       },
  31.       "timestamp" : "2021-04-25T04:00:00.000Z"
  32.     },
  33.     ...
  34.   ]
  35. }

Getting customer name and email address by customer ID

This example uses the ecommerce sample data set to create an entity-centric index based on customer ID, and to get the customer name and email address by using the top_metrics aggregation.

Group the data by customer_id, then add a top_metrics aggregation where the metrics are the email, the customer_first_name.keyword, and the customer_last_name.keyword fields. Sort the top_metrics by order_date in descending order. The API call looks like this:

  1. resp = client.transform.preview_transform(
  2.     source={
  3.         "index": "kibana_sample_data_ecommerce"
  4.     },
  5.     pivot={
  6.         "group_by": {
  7.             "customer_id": {
  8.                 "terms": {
  9.                     "field": "customer_id"
  10.                 }
  11.             }
  12.         },
  13.         "aggregations": {
  14.             "last": {
  15.                 "top_metrics": {
  16.                     "metrics": [
  17.                         {
  18.                             "field": "email"
  19.                         },
  20.                         {
  21.                             "field": "customer_first_name.keyword"
  22.                         },
  23.                         {
  24.                             "field": "customer_last_name.keyword"
  25.                         }
  26.                     ],
  27.                     "sort": {
  28.                         "order_date": "desc"
  29.                     }
  30.                 }
  31.             }
  32.         }
  33.     },
  34. )
  35. print(resp)
  1. const response = await client.transform.previewTransform({
  2.   source: {
  3.     index: "kibana_sample_data_ecommerce",
  4.   },
  5.   pivot: {
  6.     group_by: {
  7.       customer_id: {
  8.         terms: {
  9.           field: "customer_id",
  10.         },
  11.       },
  12.     },
  13.     aggregations: {
  14.       last: {
  15.         top_metrics: {
  16.           metrics: [
  17.             {
  18.               field: "email",
  19.             },
  20.             {
  21.               field: "customer_first_name.keyword",
  22.             },
  23.             {
  24.               field: "customer_last_name.keyword",
  25.             },
  26.           ],
  27.           sort: {
  28.             order_date: "desc",
  29.           },
  30.         },
  31.       },
  32.     },
  33.   },
  34. });
  35. console.log(response);
  1. POST _transform/_preview
  2. {
  3.   "source": {
  4.     "index": "kibana_sample_data_ecommerce"
  5.   },
  6.   "pivot": {
  7.     "group_by": { 
  8.       "customer_id": {
  9.         "terms": {
  10.           "field": "customer_id"
  11.         }
  12.       }
  13.     },
  14.     "aggregations": {
  15.       "last": {
  16.         "top_metrics": { 
  17.           "metrics": [
  18.             {
  19.               "field": "email"
  20.             },
  21.             {
  22.               "field": "customer_first_name.keyword"
  23.             },
  24.             {
  25.               "field": "customer_last_name.keyword"
  26.             }
  27.           ],
  28.           "sort": {
  29.             "order_date": "desc"
  30.           }
  31.         }
  32.       }
  33.     }
  34.   }
  35. }

The data is grouped by a terms aggregation on the customer_id field.

Specifies the fields to return (email and name fields) in a descending order by the order date.

The API returns a response that is similar to this:

  1.  {
  2.   "preview" : [
  3.     {
  4.       "last" : {
  5.         "customer_last_name.keyword" : "Long",
  6.         "customer_first_name.keyword" : "Recip",
  7.         "email" : "recip@long-family.zzz"
  8.       },
  9.       "customer_id" : "10"
  10.     },
  11.     {
  12.       "last" : {
  13.         "customer_last_name.keyword" : "Jackson",
  14.         "customer_first_name.keyword" : "Fitzgerald",
  15.         "email" : "fitzgerald@jackson-family.zzz"
  16.       },
  17.       "customer_id" : "11"
  18.     },
  19.     {
  20.       "last" : {
  21.         "customer_last_name.keyword" : "Cross",
  22.         "customer_first_name.keyword" : "Brigitte",
  23.         "email" : "brigitte@cross-family.zzz"
  24.       },
  25.       "customer_id" : "12"
  26.     },
  27.     ...
  28.   ]
  29. }