Run downsampling using data stream lifecycle

Run downsampling using data stream lifecycle

This is a simplified example that allows you to see quickly how downsampling works as part of a datastream lifecycle to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with data stream lifecycle, follow these steps:

  1. Check the prerequisites.
  2. Create an index template with data stream lifecycle.
  3. Ingest time series data.
  4. View current state of data stream.
  5. Roll over the data stream.
  6. View downsampling results.

Prerequisites

Refer to time series data stream prerequisites.

Create an index template with data stream lifecycle

This creates an index template for a basic data stream. The available parameters for an index template are described in detail in Set up a time series data stream.

For simplicity, in the time series mapping all time_series_metric parameters are set to type gauge, but the counter metric type may also be used. The time_series_metric values determine the kind of statistical representations that are used during downsampling.

The index template includes a set of static time series dimensions: host, namespace, node, and pod. The time series dimensions are not changed by the downsampling process.

To enable downsampling, this template includes a lifecycle section with downsampling object. fixed_interval parameter sets downsampling interval at which you want to aggregate the original time series data. after parameter specifies how much time after index was rolled over should pass before downsampling is performed.

  1. resp = client.indices.put_index_template(
  2.     name="datastream_template",
  3.     index_patterns=[
  4.         "datastream*"
  5.     ],
  6.     data_stream={},
  7.     template={
  8.         "lifecycle": {
  9.             "downsampling": [
  10.                 {
  11.                     "after": "1m",
  12.                     "fixed_interval": "1h"
  13.                 }
  14.             ]
  15.         },
  16.         "settings": {
  17.             "index": {
  18.                 "mode": "time_series"
  19.             }
  20.         },
  21.         "mappings": {
  22.             "properties": {
  23.                 "@timestamp": {
  24.                     "type": "date"
  25.                 },
  26.                 "kubernetes": {
  27.                     "properties": {
  28.                         "container": {
  29.                             "properties": {
  30.                                 "cpu": {
  31.                                     "properties": {
  32.                                         "usage": {
  33.                                             "properties": {
  34.                                                 "core": {
  35.                                                     "properties": {
  36.                                                         "ns": {
  37.                                                             "type": "long"
  38.                                                         }
  39.                                                     }
  40.                                                 },
  41.                                                 "limit": {
  42.                                                     "properties": {
  43.                                                         "pct": {
  44.                                                             "type": "float"
  45.                                                         }
  46.                                                     }
  47.                                                 },
  48.                                                 "nanocores": {
  49.                                                     "type": "long",
  50.                                                     "time_series_metric": "gauge"
  51.                                                 },
  52.                                                 "node": {
  53.                                                     "properties": {
  54.                                                         "pct": {
  55.                                                             "type": "float"
  56.                                                         }
  57.                                                     }
  58.                                                 }
  59.                                             }
  60.                                         }
  61.                                     }
  62.                                 },
  63.                                 "memory": {
  64.                                     "properties": {
  65.                                         "available": {
  66.                                             "properties": {
  67.                                                 "bytes": {
  68.                                                     "type": "long",
  69.                                                     "time_series_metric": "gauge"
  70.                                                 }
  71.                                             }
  72.                                         },
  73.                                         "majorpagefaults": {
  74.                                             "type": "long"
  75.                                         },
  76.                                         "pagefaults": {
  77.                                             "type": "long",
  78.                                             "time_series_metric": "gauge"
  79.                                         },
  80.                                         "rss": {
  81.                                             "properties": {
  82.                                                 "bytes": {
  83.                                                     "type": "long",
  84.                                                     "time_series_metric": "gauge"
  85.                                                 }
  86.                                             }
  87.                                         },
  88.                                         "usage": {
  89.                                             "properties": {
  90.                                                 "bytes": {
  91.                                                     "type": "long",
  92.                                                     "time_series_metric": "gauge"
  93.                                                 },
  94.                                                 "limit": {
  95.                                                     "properties": {
  96.                                                         "pct": {
  97.                                                             "type": "float"
  98.                                                         }
  99.                                                     }
  100.                                                 },
  101.                                                 "node": {
  102.                                                     "properties": {
  103.                                                         "pct": {
  104.                                                             "type": "float"
  105.                                                         }
  106.                                                     }
  107.                                                 }
  108.                                             }
  109.                                         },
  110.                                         "workingset": {
  111.                                             "properties": {
  112.                                                 "bytes": {
  113.                                                     "type": "long",
  114.                                                     "time_series_metric": "gauge"
  115.                                                 }
  116.                                             }
  117.                                         }
  118.                                     }
  119.                                 },
  120.                                 "name": {
  121.                                     "type": "keyword"
  122.                                 },
  123.                                 "start_time": {
  124.                                     "type": "date"
  125.                                 }
  126.                             }
  127.                         },
  128.                         "host": {
  129.                             "type": "keyword",
  130.                             "time_series_dimension": True
  131.                         },
  132.                         "namespace": {
  133.                             "type": "keyword",
  134.                             "time_series_dimension": True
  135.                         },
  136.                         "node": {
  137.                             "type": "keyword",
  138.                             "time_series_dimension": True
  139.                         },
  140.                         "pod": {
  141.                             "type": "keyword",
  142.                             "time_series_dimension": True
  143.                         }
  144.                     }
  145.                 }
  146.             }
  147.         }
  148.     },
  149. )
  150. print(resp)
  1. const response = await client.indices.putIndexTemplate({
  2.   name: "datastream_template",
  3.   index_patterns: ["datastream*"],
  4.   data_stream: {},
  5.   template: {
  6.     lifecycle: {
  7.       downsampling: [
  8.         {
  9.           after: "1m",
  10.           fixed_interval: "1h",
  11.         },
  12.       ],
  13.     },
  14.     settings: {
  15.       index: {
  16.         mode: "time_series",
  17.       },
  18.     },
  19.     mappings: {
  20.       properties: {
  21.         "@timestamp": {
  22.           type: "date",
  23.         },
  24.         kubernetes: {
  25.           properties: {
  26.             container: {
  27.               properties: {
  28.                 cpu: {
  29.                   properties: {
  30.                     usage: {
  31.                       properties: {
  32.                         core: {
  33.                           properties: {
  34.                             ns: {
  35.                               type: "long",
  36.                             },
  37.                           },
  38.                         },
  39.                         limit: {
  40.                           properties: {
  41.                             pct: {
  42.                               type: "float",
  43.                             },
  44.                           },
  45.                         },
  46.                         nanocores: {
  47.                           type: "long",
  48.                           time_series_metric: "gauge",
  49.                         },
  50.                         node: {
  51.                           properties: {
  52.                             pct: {
  53.                               type: "float",
  54.                             },
  55.                           },
  56.                         },
  57.                       },
  58.                     },
  59.                   },
  60.                 },
  61.                 memory: {
  62.                   properties: {
  63.                     available: {
  64.                       properties: {
  65.                         bytes: {
  66.                           type: "long",
  67.                           time_series_metric: "gauge",
  68.                         },
  69.                       },
  70.                     },
  71.                     majorpagefaults: {
  72.                       type: "long",
  73.                     },
  74.                     pagefaults: {
  75.                       type: "long",
  76.                       time_series_metric: "gauge",
  77.                     },
  78.                     rss: {
  79.                       properties: {
  80.                         bytes: {
  81.                           type: "long",
  82.                           time_series_metric: "gauge",
  83.                         },
  84.                       },
  85.                     },
  86.                     usage: {
  87.                       properties: {
  88.                         bytes: {
  89.                           type: "long",
  90.                           time_series_metric: "gauge",
  91.                         },
  92.                         limit: {
  93.                           properties: {
  94.                             pct: {
  95.                               type: "float",
  96.                             },
  97.                           },
  98.                         },
  99.                         node: {
  100.                           properties: {
  101.                             pct: {
  102.                               type: "float",
  103.                             },
  104.                           },
  105.                         },
  106.                       },
  107.                     },
  108.                     workingset: {
  109.                       properties: {
  110.                         bytes: {
  111.                           type: "long",
  112.                           time_series_metric: "gauge",
  113.                         },
  114.                       },
  115.                     },
  116.                   },
  117.                 },
  118.                 name: {
  119.                   type: "keyword",
  120.                 },
  121.                 start_time: {
  122.                   type: "date",
  123.                 },
  124.               },
  125.             },
  126.             host: {
  127.               type: "keyword",
  128.               time_series_dimension: true,
  129.             },
  130.             namespace: {
  131.               type: "keyword",
  132.               time_series_dimension: true,
  133.             },
  134.             node: {
  135.               type: "keyword",
  136.               time_series_dimension: true,
  137.             },
  138.             pod: {
  139.               type: "keyword",
  140.               time_series_dimension: true,
  141.             },
  142.           },
  143.         },
  144.       },
  145.     },
  146.   },
  147. });
  148. console.log(response);
  1. PUT _index_template/datastream_template
  2. {
  3.   "index_patterns": [
  4.     "datastream*"
  5.   ],
  6.   "data_stream": {},
  7.   "template": {
  8.     "lifecycle": {
  9.       "downsampling": [
  10.         {
  11.           "after": "1m",
  12.           "fixed_interval": "1h"
  13.         }
  14.       ]
  15.     },
  16.     "settings": {
  17.       "index": {
  18.         "mode": "time_series"
  19.       }
  20.     },
  21.     "mappings": {
  22.       "properties": {
  23.         "@timestamp": {
  24.           "type": "date"
  25.         },
  26.         "kubernetes": {
  27.           "properties": {
  28.             "container": {
  29.               "properties": {
  30.                 "cpu": {
  31.                   "properties": {
  32.                     "usage": {
  33.                       "properties": {
  34.                         "core": {
  35.                           "properties": {
  36.                             "ns": {
  37.                               "type": "long"
  38.                             }
  39.                           }
  40.                         },
  41.                         "limit": {
  42.                           "properties": {
  43.                             "pct": {
  44.                               "type": "float"
  45.                             }
  46.                           }
  47.                         },
  48.                         "nanocores": {
  49.                           "type": "long",
  50.                           "time_series_metric": "gauge"
  51.                         },
  52.                         "node": {
  53.                           "properties": {
  54.                             "pct": {
  55.                               "type": "float"
  56.                             }
  57.                           }
  58.                         }
  59.                       }
  60.                     }
  61.                   }
  62.                 },
  63.                 "memory": {
  64.                   "properties": {
  65.                     "available": {
  66.                       "properties": {
  67.                         "bytes": {
  68.                           "type": "long",
  69.                           "time_series_metric": "gauge"
  70.                         }
  71.                       }
  72.                     },
  73.                     "majorpagefaults": {
  74.                       "type": "long"
  75.                     },
  76.                     "pagefaults": {
  77.                       "type": "long",
  78.                       "time_series_metric": "gauge"
  79.                     },
  80.                     "rss": {
  81.                       "properties": {
  82.                         "bytes": {
  83.                           "type": "long",
  84.                           "time_series_metric": "gauge"
  85.                         }
  86.                       }
  87.                     },
  88.                     "usage": {
  89.                       "properties": {
  90.                         "bytes": {
  91.                           "type": "long",
  92.                           "time_series_metric": "gauge"
  93.                         },
  94.                         "limit": {
  95.                           "properties": {
  96.                             "pct": {
  97.                               "type": "float"
  98.                             }
  99.                           }
  100.                         },
  101.                         "node": {
  102.                           "properties": {
  103.                             "pct": {
  104.                               "type": "float"
  105.                             }
  106.                           }
  107.                         }
  108.                       }
  109.                     },
  110.                     "workingset": {
  111.                       "properties": {
  112.                         "bytes": {
  113.                           "type": "long",
  114.                           "time_series_metric": "gauge"
  115.                         }
  116.                       }
  117.                     }
  118.                   }
  119.                 },
  120.                 "name": {
  121.                   "type": "keyword"
  122.                 },
  123.                 "start_time": {
  124.                   "type": "date"
  125.                 }
  126.               }
  127.             },
  128.             "host": {
  129.               "type": "keyword",
  130.               "time_series_dimension": true
  131.             },
  132.             "namespace": {
  133.               "type": "keyword",
  134.               "time_series_dimension": true
  135.             },
  136.             "node": {
  137.               "type": "keyword",
  138.               "time_series_dimension": true
  139.             },
  140.             "pod": {
  141.               "type": "keyword",
  142.               "time_series_dimension": true
  143.             }
  144.           }
  145.         }
  146.       }
  147.     }
  148.   }
  149. }

Ingest time series data

Use a bulk API request to automatically create your TSDS and index a set of ten documents.

Important: Before running this bulk request you need to update the timestamps to within three to five hours after your current time. That is, search 2022-06-21T15 and replace with your present date, and adjust the hour to your current time plus three hours.

  1. resp = client.bulk(
  2.     index="datastream",
  3.     refresh=True,
  4.     operations=[
  5.         {
  6.             "create": {}
  7.         },
  8.         {
  9.             "@timestamp": "2022-06-21T15:49:00Z",
  10.             "kubernetes": {
  11.                 "host": "gke-apps-0",
  12.                 "node": "gke-apps-0-0",
  13.                 "pod": "gke-apps-0-0-0",
  14.                 "container": {
  15.                     "cpu": {
  16.                         "usage": {
  17.                             "nanocores": 91153,
  18.                             "core": {
  19.                                 "ns": 12828317850
  20.                             },
  21.                             "node": {
  22.                                 "pct": 0.0000277905
  23.                             },
  24.                             "limit": {
  25.                                 "pct": 0.0000277905
  26.                             }
  27.                         }
  28.                     },
  29.                     "memory": {
  30.                         "available": {
  31.                             "bytes": 463314616
  32.                         },
  33.                         "usage": {
  34.                             "bytes": 307007078,
  35.                             "node": {
  36.                                 "pct": 0.01770037710617187
  37.                             },
  38.                             "limit": {
  39.                                 "pct": 0.00009923134671484496
  40.                             }
  41.                         },
  42.                         "workingset": {
  43.                             "bytes": 585236
  44.                         },
  45.                         "rss": {
  46.                             "bytes": 102728
  47.                         },
  48.                         "pagefaults": 120901,
  49.                         "majorpagefaults": 0
  50.                     },
  51.                     "start_time": "2021-03-30T07:59:06Z",
  52.                     "name": "container-name-44"
  53.                 },
  54.                 "namespace": "namespace26"
  55.             }
  56.         },
  57.         {
  58.             "create": {}
  59.         },
  60.         {
  61.             "@timestamp": "2022-06-21T15:45:50Z",
  62.             "kubernetes": {
  63.                 "host": "gke-apps-0",
  64.                 "node": "gke-apps-0-0",
  65.                 "pod": "gke-apps-0-0-0",
  66.                 "container": {
  67.                     "cpu": {
  68.                         "usage": {
  69.                             "nanocores": 124501,
  70.                             "core": {
  71.                                 "ns": 12828317850
  72.                             },
  73.                             "node": {
  74.                                 "pct": 0.0000277905
  75.                             },
  76.                             "limit": {
  77.                                 "pct": 0.0000277905
  78.                             }
  79.                         }
  80.                     },
  81.                     "memory": {
  82.                         "available": {
  83.                             "bytes": 982546514
  84.                         },
  85.                         "usage": {
  86.                             "bytes": 360035574,
  87.                             "node": {
  88.                                 "pct": 0.01770037710617187
  89.                             },
  90.                             "limit": {
  91.                                 "pct": 0.00009923134671484496
  92.                             }
  93.                         },
  94.                         "workingset": {
  95.                             "bytes": 1339884
  96.                         },
  97.                         "rss": {
  98.                             "bytes": 381174
  99.                         },
  100.                         "pagefaults": 178473,
  101.                         "majorpagefaults": 0
  102.                     },
  103.                     "start_time": "2021-03-30T07:59:06Z",
  104.                     "name": "container-name-44"
  105.                 },
  106.                 "namespace": "namespace26"
  107.             }
  108.         },
  109.         {
  110.             "create": {}
  111.         },
  112.         {
  113.             "@timestamp": "2022-06-21T15:44:50Z",
  114.             "kubernetes": {
  115.                 "host": "gke-apps-0",
  116.                 "node": "gke-apps-0-0",
  117.                 "pod": "gke-apps-0-0-0",
  118.                 "container": {
  119.                     "cpu": {
  120.                         "usage": {
  121.                             "nanocores": 38907,
  122.                             "core": {
  123.                                 "ns": 12828317850
  124.                             },
  125.                             "node": {
  126.                                 "pct": 0.0000277905
  127.                             },
  128.                             "limit": {
  129.                                 "pct": 0.0000277905
  130.                             }
  131.                         }
  132.                     },
  133.                     "memory": {
  134.                         "available": {
  135.                             "bytes": 862723768
  136.                         },
  137.                         "usage": {
  138.                             "bytes": 379572388,
  139.                             "node": {
  140.                                 "pct": 0.01770037710617187
  141.                             },
  142.                             "limit": {
  143.                                 "pct": 0.00009923134671484496
  144.                             }
  145.                         },
  146.                         "workingset": {
  147.                             "bytes": 431227
  148.                         },
  149.                         "rss": {
  150.                             "bytes": 386580
  151.                         },
  152.                         "pagefaults": 233166,
  153.                         "majorpagefaults": 0
  154.                     },
  155.                     "start_time": "2021-03-30T07:59:06Z",
  156.                     "name": "container-name-44"
  157.                 },
  158.                 "namespace": "namespace26"
  159.             }
  160.         },
  161.         {
  162.             "create": {}
  163.         },
  164.         {
  165.             "@timestamp": "2022-06-21T15:44:40Z",
  166.             "kubernetes": {
  167.                 "host": "gke-apps-0",
  168.                 "node": "gke-apps-0-0",
  169.                 "pod": "gke-apps-0-0-0",
  170.                 "container": {
  171.                     "cpu": {
  172.                         "usage": {
  173.                             "nanocores": 86706,
  174.                             "core": {
  175.                                 "ns": 12828317850
  176.                             },
  177.                             "node": {
  178.                                 "pct": 0.0000277905
  179.                             },
  180.                             "limit": {
  181.                                 "pct": 0.0000277905
  182.                             }
  183.                         }
  184.                     },
  185.                     "memory": {
  186.                         "available": {
  187.                             "bytes": 567160996
  188.                         },
  189.                         "usage": {
  190.                             "bytes": 103266017,
  191.                             "node": {
  192.                                 "pct": 0.01770037710617187
  193.                             },
  194.                             "limit": {
  195.                                 "pct": 0.00009923134671484496
  196.                             }
  197.                         },
  198.                         "workingset": {
  199.                             "bytes": 1724908
  200.                         },
  201.                         "rss": {
  202.                             "bytes": 105431
  203.                         },
  204.                         "pagefaults": 233166,
  205.                         "majorpagefaults": 0
  206.                     },
  207.                     "start_time": "2021-03-30T07:59:06Z",
  208.                     "name": "container-name-44"
  209.                 },
  210.                 "namespace": "namespace26"
  211.             }
  212.         },
  213.         {
  214.             "create": {}
  215.         },
  216.         {
  217.             "@timestamp": "2022-06-21T15:44:00Z",
  218.             "kubernetes": {
  219.                 "host": "gke-apps-0",
  220.                 "node": "gke-apps-0-0",
  221.                 "pod": "gke-apps-0-0-0",
  222.                 "container": {
  223.                     "cpu": {
  224.                         "usage": {
  225.                             "nanocores": 150069,
  226.                             "core": {
  227.                                 "ns": 12828317850
  228.                             },
  229.                             "node": {
  230.                                 "pct": 0.0000277905
  231.                             },
  232.                             "limit": {
  233.                                 "pct": 0.0000277905
  234.                             }
  235.                         }
  236.                     },
  237.                     "memory": {
  238.                         "available": {
  239.                             "bytes": 639054643
  240.                         },
  241.                         "usage": {
  242.                             "bytes": 265142477,
  243.                             "node": {
  244.                                 "pct": 0.01770037710617187
  245.                             },
  246.                             "limit": {
  247.                                 "pct": 0.00009923134671484496
  248.                             }
  249.                         },
  250.                         "workingset": {
  251.                             "bytes": 1786511
  252.                         },
  253.                         "rss": {
  254.                             "bytes": 189235
  255.                         },
  256.                         "pagefaults": 138172,
  257.                         "majorpagefaults": 0
  258.                     },
  259.                     "start_time": "2021-03-30T07:59:06Z",
  260.                     "name": "container-name-44"
  261.                 },
  262.                 "namespace": "namespace26"
  263.             }
  264.         },
  265.         {
  266.             "create": {}
  267.         },
  268.         {
  269.             "@timestamp": "2022-06-21T15:42:40Z",
  270.             "kubernetes": {
  271.                 "host": "gke-apps-0",
  272.                 "node": "gke-apps-0-0",
  273.                 "pod": "gke-apps-0-0-0",
  274.                 "container": {
  275.                     "cpu": {
  276.                         "usage": {
  277.                             "nanocores": 82260,
  278.                             "core": {
  279.                                 "ns": 12828317850
  280.                             },
  281.                             "node": {
  282.                                 "pct": 0.0000277905
  283.                             },
  284.                             "limit": {
  285.                                 "pct": 0.0000277905
  286.                             }
  287.                         }
  288.                     },
  289.                     "memory": {
  290.                         "available": {
  291.                             "bytes": 854735585
  292.                         },
  293.                         "usage": {
  294.                             "bytes": 309798052,
  295.                             "node": {
  296.                                 "pct": 0.01770037710617187
  297.                             },
  298.                             "limit": {
  299.                                 "pct": 0.00009923134671484496
  300.                             }
  301.                         },
  302.                         "workingset": {
  303.                             "bytes": 924058
  304.                         },
  305.                         "rss": {
  306.                             "bytes": 110838
  307.                         },
  308.                         "pagefaults": 259073,
  309.                         "majorpagefaults": 0
  310.                     },
  311.                     "start_time": "2021-03-30T07:59:06Z",
  312.                     "name": "container-name-44"
  313.                 },
  314.                 "namespace": "namespace26"
  315.             }
  316.         },
  317.         {
  318.             "create": {}
  319.         },
  320.         {
  321.             "@timestamp": "2022-06-21T15:42:10Z",
  322.             "kubernetes": {
  323.                 "host": "gke-apps-0",
  324.                 "node": "gke-apps-0-0",
  325.                 "pod": "gke-apps-0-0-0",
  326.                 "container": {
  327.                     "cpu": {
  328.                         "usage": {
  329.                             "nanocores": 153404,
  330.                             "core": {
  331.                                 "ns": 12828317850
  332.                             },
  333.                             "node": {
  334.                                 "pct": 0.0000277905
  335.                             },
  336.                             "limit": {
  337.                                 "pct": 0.0000277905
  338.                             }
  339.                         }
  340.                     },
  341.                     "memory": {
  342.                         "available": {
  343.                             "bytes": 279586406
  344.                         },
  345.                         "usage": {
  346.                             "bytes": 214904955,
  347.                             "node": {
  348.                                 "pct": 0.01770037710617187
  349.                             },
  350.                             "limit": {
  351.                                 "pct": 0.00009923134671484496
  352.                             }
  353.                         },
  354.                         "workingset": {
  355.                             "bytes": 1047265
  356.                         },
  357.                         "rss": {
  358.                             "bytes": 91914
  359.                         },
  360.                         "pagefaults": 302252,
  361.                         "majorpagefaults": 0
  362.                     },
  363.                     "start_time": "2021-03-30T07:59:06Z",
  364.                     "name": "container-name-44"
  365.                 },
  366.                 "namespace": "namespace26"
  367.             }
  368.         },
  369.         {
  370.             "create": {}
  371.         },
  372.         {
  373.             "@timestamp": "2022-06-21T15:40:20Z",
  374.             "kubernetes": {
  375.                 "host": "gke-apps-0",
  376.                 "node": "gke-apps-0-0",
  377.                 "pod": "gke-apps-0-0-0",
  378.                 "container": {
  379.                     "cpu": {
  380.                         "usage": {
  381.                             "nanocores": 125613,
  382.                             "core": {
  383.                                 "ns": 12828317850
  384.                             },
  385.                             "node": {
  386.                                 "pct": 0.0000277905
  387.                             },
  388.                             "limit": {
  389.                                 "pct": 0.0000277905
  390.                             }
  391.                         }
  392.                     },
  393.                     "memory": {
  394.                         "available": {
  395.                             "bytes": 822782853
  396.                         },
  397.                         "usage": {
  398.                             "bytes": 100475044,
  399.                             "node": {
  400.                                 "pct": 0.01770037710617187
  401.                             },
  402.                             "limit": {
  403.                                 "pct": 0.00009923134671484496
  404.                             }
  405.                         },
  406.                         "workingset": {
  407.                             "bytes": 2109932
  408.                         },
  409.                         "rss": {
  410.                             "bytes": 278446
  411.                         },
  412.                         "pagefaults": 74843,
  413.                         "majorpagefaults": 0
  414.                     },
  415.                     "start_time": "2021-03-30T07:59:06Z",
  416.                     "name": "container-name-44"
  417.                 },
  418.                 "namespace": "namespace26"
  419.             }
  420.         },
  421.         {
  422.             "create": {}
  423.         },
  424.         {
  425.             "@timestamp": "2022-06-21T15:40:10Z",
  426.             "kubernetes": {
  427.                 "host": "gke-apps-0",
  428.                 "node": "gke-apps-0-0",
  429.                 "pod": "gke-apps-0-0-0",
  430.                 "container": {
  431.                     "cpu": {
  432.                         "usage": {
  433.                             "nanocores": 100046,
  434.                             "core": {
  435.                                 "ns": 12828317850
  436.                             },
  437.                             "node": {
  438.                                 "pct": 0.0000277905
  439.                             },
  440.                             "limit": {
  441.                                 "pct": 0.0000277905
  442.                             }
  443.                         }
  444.                     },
  445.                     "memory": {
  446.                         "available": {
  447.                             "bytes": 567160996
  448.                         },
  449.                         "usage": {
  450.                             "bytes": 362826547,
  451.                             "node": {
  452.                                 "pct": 0.01770037710617187
  453.                             },
  454.                             "limit": {
  455.                                 "pct": 0.00009923134671484496
  456.                             }
  457.                         },
  458.                         "workingset": {
  459.                             "bytes": 1986724
  460.                         },
  461.                         "rss": {
  462.                             "bytes": 402801
  463.                         },
  464.                         "pagefaults": 296495,
  465.                         "majorpagefaults": 0
  466.                     },
  467.                     "start_time": "2021-03-30T07:59:06Z",
  468.                     "name": "container-name-44"
  469.                 },
  470.                 "namespace": "namespace26"
  471.             }
  472.         },
  473.         {
  474.             "create": {}
  475.         },
  476.         {
  477.             "@timestamp": "2022-06-21T15:38:30Z",
  478.             "kubernetes": {
  479.                 "host": "gke-apps-0",
  480.                 "node": "gke-apps-0-0",
  481.                 "pod": "gke-apps-0-0-0",
  482.                 "container": {
  483.                     "cpu": {
  484.                         "usage": {
  485.                             "nanocores": 40018,
  486.                             "core": {
  487.                                 "ns": 12828317850
  488.                             },
  489.                             "node": {
  490.                                 "pct": 0.0000277905
  491.                             },
  492.                             "limit": {
  493.                                 "pct": 0.0000277905
  494.                             }
  495.                         }
  496.                     },
  497.                     "memory": {
  498.                         "available": {
  499.                             "bytes": 1062428344
  500.                         },
  501.                         "usage": {
  502.                             "bytes": 265142477,
  503.                             "node": {
  504.                                 "pct": 0.01770037710617187
  505.                             },
  506.                             "limit": {
  507.                                 "pct": 0.00009923134671484496
  508.                             }
  509.                         },
  510.                         "workingset": {
  511.                             "bytes": 2294743
  512.                         },
  513.                         "rss": {
  514.                             "bytes": 340623
  515.                         },
  516.                         "pagefaults": 224530,
  517.                         "majorpagefaults": 0
  518.                     },
  519.                     "start_time": "2021-03-30T07:59:06Z",
  520.                     "name": "container-name-44"
  521.                 },
  522.                 "namespace": "namespace26"
  523.             }
  524.         }
  525.     ],
  526. )
  527. print(resp)
  1. response = client.bulk(
  2.   index: 'datastream',
  3.   refresh: true,
  4.   body: [
  5.     {
  6.       create: {}
  7.     },
  8.     {
  9.       "@timestamp": '2022-06-21T15:49:00Z',
  10.       kubernetes: {
  11.         host: 'gke-apps-0',
  12.         node: 'gke-apps-0-0',
  13.         pod: 'gke-apps-0-0-0',
  14.         container: {
  15.           cpu: {
  16.             usage: {
  17.               nanocores: 91_153,
  18.               core: {
  19.                 ns: 12_828_317_850
  20.               },
  21.               node: {
  22.                 pct: 2.77905e-05
  23.               },
  24.               limit: {
  25.                 pct: 2.77905e-05
  26.               }
  27.             }
  28.           },
  29.           memory: {
  30.             available: {
  31.               bytes: 463_314_616
  32.             },
  33.             usage: {
  34.               bytes: 307_007_078,
  35.               node: {
  36.                 pct: 0.01770037710617187
  37.               },
  38.               limit: {
  39.                 pct: 9.923134671484496e-05
  40.               }
  41.             },
  42.             workingset: {
  43.               bytes: 585_236
  44.             },
  45.             rss: {
  46.               bytes: 102_728
  47.             },
  48.             pagefaults: 120_901,
  49.             majorpagefaults: 0
  50.           },
  51.           start_time: '2021-03-30T07:59:06Z',
  52.           name: 'container-name-44'
  53.         },
  54.         namespace: 'namespace26'
  55.       }
  56.     },
  57.     {
  58.       create: {}
  59.     },
  60.     {
  61.       "@timestamp": '2022-06-21T15:45:50Z',
  62.       kubernetes: {
  63.         host: 'gke-apps-0',
  64.         node: 'gke-apps-0-0',
  65.         pod: 'gke-apps-0-0-0',
  66.         container: {
  67.           cpu: {
  68.             usage: {
  69.               nanocores: 124_501,
  70.               core: {
  71.                 ns: 12_828_317_850
  72.               },
  73.               node: {
  74.                 pct: 2.77905e-05
  75.               },
  76.               limit: {
  77.                 pct: 2.77905e-05
  78.               }
  79.             }
  80.           },
  81.           memory: {
  82.             available: {
  83.               bytes: 982_546_514
  84.             },
  85.             usage: {
  86.               bytes: 360_035_574,
  87.               node: {
  88.                 pct: 0.01770037710617187
  89.               },
  90.               limit: {
  91.                 pct: 9.923134671484496e-05
  92.               }
  93.             },
  94.             workingset: {
  95.               bytes: 1_339_884
  96.             },
  97.             rss: {
  98.               bytes: 381_174
  99.             },
  100.             pagefaults: 178_473,
  101.             majorpagefaults: 0
  102.           },
  103.           start_time: '2021-03-30T07:59:06Z',
  104.           name: 'container-name-44'
  105.         },
  106.         namespace: 'namespace26'
  107.       }
  108.     },
  109.     {
  110.       create: {}
  111.     },
  112.     {
  113.       "@timestamp": '2022-06-21T15:44:50Z',
  114.       kubernetes: {
  115.         host: 'gke-apps-0',
  116.         node: 'gke-apps-0-0',
  117.         pod: 'gke-apps-0-0-0',
  118.         container: {
  119.           cpu: {
  120.             usage: {
  121.               nanocores: 38_907,
  122.               core: {
  123.                 ns: 12_828_317_850
  124.               },
  125.               node: {
  126.                 pct: 2.77905e-05
  127.               },
  128.               limit: {
  129.                 pct: 2.77905e-05
  130.               }
  131.             }
  132.           },
  133.           memory: {
  134.             available: {
  135.               bytes: 862_723_768
  136.             },
  137.             usage: {
  138.               bytes: 379_572_388,
  139.               node: {
  140.                 pct: 0.01770037710617187
  141.               },
  142.               limit: {
  143.                 pct: 9.923134671484496e-05
  144.               }
  145.             },
  146.             workingset: {
  147.               bytes: 431_227
  148.             },
  149.             rss: {
  150.               bytes: 386_580
  151.             },
  152.             pagefaults: 233_166,
  153.             majorpagefaults: 0
  154.           },
  155.           start_time: '2021-03-30T07:59:06Z',
  156.           name: 'container-name-44'
  157.         },
  158.         namespace: 'namespace26'
  159.       }
  160.     },
  161.     {
  162.       create: {}
  163.     },
  164.     {
  165.       "@timestamp": '2022-06-21T15:44:40Z',
  166.       kubernetes: {
  167.         host: 'gke-apps-0',
  168.         node: 'gke-apps-0-0',
  169.         pod: 'gke-apps-0-0-0',
  170.         container: {
  171.           cpu: {
  172.             usage: {
  173.               nanocores: 86_706,
  174.               core: {
  175.                 ns: 12_828_317_850
  176.               },
  177.               node: {
  178.                 pct: 2.77905e-05
  179.               },
  180.               limit: {
  181.                 pct: 2.77905e-05
  182.               }
  183.             }
  184.           },
  185.           memory: {
  186.             available: {
  187.               bytes: 567_160_996
  188.             },
  189.             usage: {
  190.               bytes: 103_266_017,
  191.               node: {
  192.                 pct: 0.01770037710617187
  193.               },
  194.               limit: {
  195.                 pct: 9.923134671484496e-05
  196.               }
  197.             },
  198.             workingset: {
  199.               bytes: 1_724_908
  200.             },
  201.             rss: {
  202.               bytes: 105_431
  203.             },
  204.             pagefaults: 233_166,
  205.             majorpagefaults: 0
  206.           },
  207.           start_time: '2021-03-30T07:59:06Z',
  208.           name: 'container-name-44'
  209.         },
  210.         namespace: 'namespace26'
  211.       }
  212.     },
  213.     {
  214.       create: {}
  215.     },
  216.     {
  217.       "@timestamp": '2022-06-21T15:44:00Z',
  218.       kubernetes: {
  219.         host: 'gke-apps-0',
  220.         node: 'gke-apps-0-0',
  221.         pod: 'gke-apps-0-0-0',
  222.         container: {
  223.           cpu: {
  224.             usage: {
  225.               nanocores: 150_069,
  226.               core: {
  227.                 ns: 12_828_317_850
  228.               },
  229.               node: {
  230.                 pct: 2.77905e-05
  231.               },
  232.               limit: {
  233.                 pct: 2.77905e-05
  234.               }
  235.             }
  236.           },
  237.           memory: {
  238.             available: {
  239.               bytes: 639_054_643
  240.             },
  241.             usage: {
  242.               bytes: 265_142_477,
  243.               node: {
  244.                 pct: 0.01770037710617187
  245.               },
  246.               limit: {
  247.                 pct: 9.923134671484496e-05
  248.               }
  249.             },
  250.             workingset: {
  251.               bytes: 1_786_511
  252.             },
  253.             rss: {
  254.               bytes: 189_235
  255.             },
  256.             pagefaults: 138_172,
  257.             majorpagefaults: 0
  258.           },
  259.           start_time: '2021-03-30T07:59:06Z',
  260.           name: 'container-name-44'
  261.         },
  262.         namespace: 'namespace26'
  263.       }
  264.     },
  265.     {
  266.       create: {}
  267.     },
  268.     {
  269.       "@timestamp": '2022-06-21T15:42:40Z',
  270.       kubernetes: {
  271.         host: 'gke-apps-0',
  272.         node: 'gke-apps-0-0',
  273.         pod: 'gke-apps-0-0-0',
  274.         container: {
  275.           cpu: {
  276.             usage: {
  277.               nanocores: 82_260,
  278.               core: {
  279.                 ns: 12_828_317_850
  280.               },
  281.               node: {
  282.                 pct: 2.77905e-05
  283.               },
  284.               limit: {
  285.                 pct: 2.77905e-05
  286.               }
  287.             }
  288.           },
  289.           memory: {
  290.             available: {
  291.               bytes: 854_735_585
  292.             },
  293.             usage: {
  294.               bytes: 309_798_052,
  295.               node: {
  296.                 pct: 0.01770037710617187
  297.               },
  298.               limit: {
  299.                 pct: 9.923134671484496e-05
  300.               }
  301.             },
  302.             workingset: {
  303.               bytes: 924_058
  304.             },
  305.             rss: {
  306.               bytes: 110_838
  307.             },
  308.             pagefaults: 259_073,
  309.             majorpagefaults: 0
  310.           },
  311.           start_time: '2021-03-30T07:59:06Z',
  312.           name: 'container-name-44'
  313.         },
  314.         namespace: 'namespace26'
  315.       }
  316.     },
  317.     {
  318.       create: {}
  319.     },
  320.     {
  321.       "@timestamp": '2022-06-21T15:42:10Z',
  322.       kubernetes: {
  323.         host: 'gke-apps-0',
  324.         node: 'gke-apps-0-0',
  325.         pod: 'gke-apps-0-0-0',
  326.         container: {
  327.           cpu: {
  328.             usage: {
  329.               nanocores: 153_404,
  330.               core: {
  331.                 ns: 12_828_317_850
  332.               },
  333.               node: {
  334.                 pct: 2.77905e-05
  335.               },
  336.               limit: {
  337.                 pct: 2.77905e-05
  338.               }
  339.             }
  340.           },
  341.           memory: {
  342.             available: {
  343.               bytes: 279_586_406
  344.             },
  345.             usage: {
  346.               bytes: 214_904_955,
  347.               node: {
  348.                 pct: 0.01770037710617187
  349.               },
  350.               limit: {
  351.                 pct: 9.923134671484496e-05
  352.               }
  353.             },
  354.             workingset: {
  355.               bytes: 1_047_265
  356.             },
  357.             rss: {
  358.               bytes: 91_914
  359.             },
  360.             pagefaults: 302_252,
  361.             majorpagefaults: 0
  362.           },
  363.           start_time: '2021-03-30T07:59:06Z',
  364.           name: 'container-name-44'
  365.         },
  366.         namespace: 'namespace26'
  367.       }
  368.     },
  369.     {
  370.       create: {}
  371.     },
  372.     {
  373.       "@timestamp": '2022-06-21T15:40:20Z',
  374.       kubernetes: {
  375.         host: 'gke-apps-0',
  376.         node: 'gke-apps-0-0',
  377.         pod: 'gke-apps-0-0-0',
  378.         container: {
  379.           cpu: {
  380.             usage: {
  381.               nanocores: 125_613,
  382.               core: {
  383.                 ns: 12_828_317_850
  384.               },
  385.               node: {
  386.                 pct: 2.77905e-05
  387.               },
  388.               limit: {
  389.                 pct: 2.77905e-05
  390.               }
  391.             }
  392.           },
  393.           memory: {
  394.             available: {
  395.               bytes: 822_782_853
  396.             },
  397.             usage: {
  398.               bytes: 100_475_044,
  399.               node: {
  400.                 pct: 0.01770037710617187
  401.               },
  402.               limit: {
  403.                 pct: 9.923134671484496e-05
  404.               }
  405.             },
  406.             workingset: {
  407.               bytes: 2_109_932
  408.             },
  409.             rss: {
  410.               bytes: 278_446
  411.             },
  412.             pagefaults: 74_843,
  413.             majorpagefaults: 0
  414.           },
  415.           start_time: '2021-03-30T07:59:06Z',
  416.           name: 'container-name-44'
  417.         },
  418.         namespace: 'namespace26'
  419.       }
  420.     },
  421.     {
  422.       create: {}
  423.     },
  424.     {
  425.       "@timestamp": '2022-06-21T15:40:10Z',
  426.       kubernetes: {
  427.         host: 'gke-apps-0',
  428.         node: 'gke-apps-0-0',
  429.         pod: 'gke-apps-0-0-0',
  430.         container: {
  431.           cpu: {
  432.             usage: {
  433.               nanocores: 100_046,
  434.               core: {
  435.                 ns: 12_828_317_850
  436.               },
  437.               node: {
  438.                 pct: 2.77905e-05
  439.               },
  440.               limit: {
  441.                 pct: 2.77905e-05
  442.               }
  443.             }
  444.           },
  445.           memory: {
  446.             available: {
  447.               bytes: 567_160_996
  448.             },
  449.             usage: {
  450.               bytes: 362_826_547,
  451.               node: {
  452.                 pct: 0.01770037710617187
  453.               },
  454.               limit: {
  455.                 pct: 9.923134671484496e-05
  456.               }
  457.             },
  458.             workingset: {
  459.               bytes: 1_986_724
  460.             },
  461.             rss: {
  462.               bytes: 402_801
  463.             },
  464.             pagefaults: 296_495,
  465.             majorpagefaults: 0
  466.           },
  467.           start_time: '2021-03-30T07:59:06Z',
  468.           name: 'container-name-44'
  469.         },
  470.         namespace: 'namespace26'
  471.       }
  472.     },
  473.     {
  474.       create: {}
  475.     },
  476.     {
  477.       "@timestamp": '2022-06-21T15:38:30Z',
  478.       kubernetes: {
  479.         host: 'gke-apps-0',
  480.         node: 'gke-apps-0-0',
  481.         pod: 'gke-apps-0-0-0',
  482.         container: {
  483.           cpu: {
  484.             usage: {
  485.               nanocores: 40_018,
  486.               core: {
  487.                 ns: 12_828_317_850
  488.               },
  489.               node: {
  490.                 pct: 2.77905e-05
  491.               },
  492.               limit: {
  493.                 pct: 2.77905e-05
  494.               }
  495.             }
  496.           },
  497.           memory: {
  498.             available: {
  499.               bytes: 1_062_428_344
  500.             },
  501.             usage: {
  502.               bytes: 265_142_477,
  503.               node: {
  504.                 pct: 0.01770037710617187
  505.               },
  506.               limit: {
  507.                 pct: 9.923134671484496e-05
  508.               }
  509.             },
  510.             workingset: {
  511.               bytes: 2_294_743
  512.             },
  513.             rss: {
  514.               bytes: 340_623
  515.             },
  516.             pagefaults: 224_530,
  517.             majorpagefaults: 0
  518.           },
  519.           start_time: '2021-03-30T07:59:06Z',
  520.           name: 'container-name-44'
  521.         },
  522.         namespace: 'namespace26'
  523.       }
  524.     }
  525.   ]
  526. )
  527. puts response
  1. const response = await client.bulk({
  2.   index: "datastream",
  3.   refresh: "true",
  4.   operations: [
  5.     {
  6.       create: {},
  7.     },
  8.     {
  9.       "@timestamp": "2022-06-21T15:49:00Z",
  10.       kubernetes: {
  11.         host: "gke-apps-0",
  12.         node: "gke-apps-0-0",
  13.         pod: "gke-apps-0-0-0",
  14.         container: {
  15.           cpu: {
  16.             usage: {
  17.               nanocores: 91153,
  18.               core: {
  19.                 ns: 12828317850,
  20.               },
  21.               node: {
  22.                 pct: 0.0000277905,
  23.               },
  24.               limit: {
  25.                 pct: 0.0000277905,
  26.               },
  27.             },
  28.           },
  29.           memory: {
  30.             available: {
  31.               bytes: 463314616,
  32.             },
  33.             usage: {
  34.               bytes: 307007078,
  35.               node: {
  36.                 pct: 0.01770037710617187,
  37.               },
  38.               limit: {
  39.                 pct: 0.00009923134671484496,
  40.               },
  41.             },
  42.             workingset: {
  43.               bytes: 585236,
  44.             },
  45.             rss: {
  46.               bytes: 102728,
  47.             },
  48.             pagefaults: 120901,
  49.             majorpagefaults: 0,
  50.           },
  51.           start_time: "2021-03-30T07:59:06Z",
  52.           name: "container-name-44",
  53.         },
  54.         namespace: "namespace26",
  55.       },
  56.     },
  57.     {
  58.       create: {},
  59.     },
  60.     {
  61.       "@timestamp": "2022-06-21T15:45:50Z",
  62.       kubernetes: {
  63.         host: "gke-apps-0",
  64.         node: "gke-apps-0-0",
  65.         pod: "gke-apps-0-0-0",
  66.         container: {
  67.           cpu: {
  68.             usage: {
  69.               nanocores: 124501,
  70.               core: {
  71.                 ns: 12828317850,
  72.               },
  73.               node: {
  74.                 pct: 0.0000277905,
  75.               },
  76.               limit: {
  77.                 pct: 0.0000277905,
  78.               },
  79.             },
  80.           },
  81.           memory: {
  82.             available: {
  83.               bytes: 982546514,
  84.             },
  85.             usage: {
  86.               bytes: 360035574,
  87.               node: {
  88.                 pct: 0.01770037710617187,
  89.               },
  90.               limit: {
  91.                 pct: 0.00009923134671484496,
  92.               },
  93.             },
  94.             workingset: {
  95.               bytes: 1339884,
  96.             },
  97.             rss: {
  98.               bytes: 381174,
  99.             },
  100.             pagefaults: 178473,
  101.             majorpagefaults: 0,
  102.           },
  103.           start_time: "2021-03-30T07:59:06Z",
  104.           name: "container-name-44",
  105.         },
  106.         namespace: "namespace26",
  107.       },
  108.     },
  109.     {
  110.       create: {},
  111.     },
  112.     {
  113.       "@timestamp": "2022-06-21T15:44:50Z",
  114.       kubernetes: {
  115.         host: "gke-apps-0",
  116.         node: "gke-apps-0-0",
  117.         pod: "gke-apps-0-0-0",
  118.         container: {
  119.           cpu: {
  120.             usage: {
  121.               nanocores: 38907,
  122.               core: {
  123.                 ns: 12828317850,
  124.               },
  125.               node: {
  126.                 pct: 0.0000277905,
  127.               },
  128.               limit: {
  129.                 pct: 0.0000277905,
  130.               },
  131.             },
  132.           },
  133.           memory: {
  134.             available: {
  135.               bytes: 862723768,
  136.             },
  137.             usage: {
  138.               bytes: 379572388,
  139.               node: {
  140.                 pct: 0.01770037710617187,
  141.               },
  142.               limit: {
  143.                 pct: 0.00009923134671484496,
  144.               },
  145.             },
  146.             workingset: {
  147.               bytes: 431227,
  148.             },
  149.             rss: {
  150.               bytes: 386580,
  151.             },
  152.             pagefaults: 233166,
  153.             majorpagefaults: 0,
  154.           },
  155.           start_time: "2021-03-30T07:59:06Z",
  156.           name: "container-name-44",
  157.         },
  158.         namespace: "namespace26",
  159.       },
  160.     },
  161.     {
  162.       create: {},
  163.     },
  164.     {
  165.       "@timestamp": "2022-06-21T15:44:40Z",
  166.       kubernetes: {
  167.         host: "gke-apps-0",
  168.         node: "gke-apps-0-0",
  169.         pod: "gke-apps-0-0-0",
  170.         container: {
  171.           cpu: {
  172.             usage: {
  173.               nanocores: 86706,
  174.               core: {
  175.                 ns: 12828317850,
  176.               },
  177.               node: {
  178.                 pct: 0.0000277905,
  179.               },
  180.               limit: {
  181.                 pct: 0.0000277905,
  182.               },
  183.             },
  184.           },
  185.           memory: {
  186.             available: {
  187.               bytes: 567160996,
  188.             },
  189.             usage: {
  190.               bytes: 103266017,
  191.               node: {
  192.                 pct: 0.01770037710617187,
  193.               },
  194.               limit: {
  195.                 pct: 0.00009923134671484496,
  196.               },
  197.             },
  198.             workingset: {
  199.               bytes: 1724908,
  200.             },
  201.             rss: {
  202.               bytes: 105431,
  203.             },
  204.             pagefaults: 233166,
  205.             majorpagefaults: 0,
  206.           },
  207.           start_time: "2021-03-30T07:59:06Z",
  208.           name: "container-name-44",
  209.         },
  210.         namespace: "namespace26",
  211.       },
  212.     },
  213.     {
  214.       create: {},
  215.     },
  216.     {
  217.       "@timestamp": "2022-06-21T15:44:00Z",
  218.       kubernetes: {
  219.         host: "gke-apps-0",
  220.         node: "gke-apps-0-0",
  221.         pod: "gke-apps-0-0-0",
  222.         container: {
  223.           cpu: {
  224.             usage: {
  225.               nanocores: 150069,
  226.               core: {
  227.                 ns: 12828317850,
  228.               },
  229.               node: {
  230.                 pct: 0.0000277905,
  231.               },
  232.               limit: {
  233.                 pct: 0.0000277905,
  234.               },
  235.             },
  236.           },
  237.           memory: {
  238.             available: {
  239.               bytes: 639054643,
  240.             },
  241.             usage: {
  242.               bytes: 265142477,
  243.               node: {
  244.                 pct: 0.01770037710617187,
  245.               },
  246.               limit: {
  247.                 pct: 0.00009923134671484496,
  248.               },
  249.             },
  250.             workingset: {
  251.               bytes: 1786511,
  252.             },
  253.             rss: {
  254.               bytes: 189235,
  255.             },
  256.             pagefaults: 138172,
  257.             majorpagefaults: 0,
  258.           },
  259.           start_time: "2021-03-30T07:59:06Z",
  260.           name: "container-name-44",
  261.         },
  262.         namespace: "namespace26",
  263.       },
  264.     },
  265.     {
  266.       create: {},
  267.     },
  268.     {
  269.       "@timestamp": "2022-06-21T15:42:40Z",
  270.       kubernetes: {
  271.         host: "gke-apps-0",
  272.         node: "gke-apps-0-0",
  273.         pod: "gke-apps-0-0-0",
  274.         container: {
  275.           cpu: {
  276.             usage: {
  277.               nanocores: 82260,
  278.               core: {
  279.                 ns: 12828317850,
  280.               },
  281.               node: {
  282.                 pct: 0.0000277905,
  283.               },
  284.               limit: {
  285.                 pct: 0.0000277905,
  286.               },
  287.             },
  288.           },
  289.           memory: {
  290.             available: {
  291.               bytes: 854735585,
  292.             },
  293.             usage: {
  294.               bytes: 309798052,
  295.               node: {
  296.                 pct: 0.01770037710617187,
  297.               },
  298.               limit: {
  299.                 pct: 0.00009923134671484496,
  300.               },
  301.             },
  302.             workingset: {
  303.               bytes: 924058,
  304.             },
  305.             rss: {
  306.               bytes: 110838,
  307.             },
  308.             pagefaults: 259073,
  309.             majorpagefaults: 0,
  310.           },
  311.           start_time: "2021-03-30T07:59:06Z",
  312.           name: "container-name-44",
  313.         },
  314.         namespace: "namespace26",
  315.       },
  316.     },
  317.     {
  318.       create: {},
  319.     },
  320.     {
  321.       "@timestamp": "2022-06-21T15:42:10Z",
  322.       kubernetes: {
  323.         host: "gke-apps-0",
  324.         node: "gke-apps-0-0",
  325.         pod: "gke-apps-0-0-0",
  326.         container: {
  327.           cpu: {
  328.             usage: {
  329.               nanocores: 153404,
  330.               core: {
  331.                 ns: 12828317850,
  332.               },
  333.               node: {
  334.                 pct: 0.0000277905,
  335.               },
  336.               limit: {
  337.                 pct: 0.0000277905,
  338.               },
  339.             },
  340.           },
  341.           memory: {
  342.             available: {
  343.               bytes: 279586406,
  344.             },
  345.             usage: {
  346.               bytes: 214904955,
  347.               node: {
  348.                 pct: 0.01770037710617187,
  349.               },
  350.               limit: {
  351.                 pct: 0.00009923134671484496,
  352.               },
  353.             },
  354.             workingset: {
  355.               bytes: 1047265,
  356.             },
  357.             rss: {
  358.               bytes: 91914,
  359.             },
  360.             pagefaults: 302252,
  361.             majorpagefaults: 0,
  362.           },
  363.           start_time: "2021-03-30T07:59:06Z",
  364.           name: "container-name-44",
  365.         },
  366.         namespace: "namespace26",
  367.       },
  368.     },
  369.     {
  370.       create: {},
  371.     },
  372.     {
  373.       "@timestamp": "2022-06-21T15:40:20Z",
  374.       kubernetes: {
  375.         host: "gke-apps-0",
  376.         node: "gke-apps-0-0",
  377.         pod: "gke-apps-0-0-0",
  378.         container: {
  379.           cpu: {
  380.             usage: {
  381.               nanocores: 125613,
  382.               core: {
  383.                 ns: 12828317850,
  384.               },
  385.               node: {
  386.                 pct: 0.0000277905,
  387.               },
  388.               limit: {
  389.                 pct: 0.0000277905,
  390.               },
  391.             },
  392.           },
  393.           memory: {
  394.             available: {
  395.               bytes: 822782853,
  396.             },
  397.             usage: {
  398.               bytes: 100475044,
  399.               node: {
  400.                 pct: 0.01770037710617187,
  401.               },
  402.               limit: {
  403.                 pct: 0.00009923134671484496,
  404.               },
  405.             },
  406.             workingset: {
  407.               bytes: 2109932,
  408.             },
  409.             rss: {
  410.               bytes: 278446,
  411.             },
  412.             pagefaults: 74843,
  413.             majorpagefaults: 0,
  414.           },
  415.           start_time: "2021-03-30T07:59:06Z",
  416.           name: "container-name-44",
  417.         },
  418.         namespace: "namespace26",
  419.       },
  420.     },
  421.     {
  422.       create: {},
  423.     },
  424.     {
  425.       "@timestamp": "2022-06-21T15:40:10Z",
  426.       kubernetes: {
  427.         host: "gke-apps-0",
  428.         node: "gke-apps-0-0",
  429.         pod: "gke-apps-0-0-0",
  430.         container: {
  431.           cpu: {
  432.             usage: {
  433.               nanocores: 100046,
  434.               core: {
  435.                 ns: 12828317850,
  436.               },
  437.               node: {
  438.                 pct: 0.0000277905,
  439.               },
  440.               limit: {
  441.                 pct: 0.0000277905,
  442.               },
  443.             },
  444.           },
  445.           memory: {
  446.             available: {
  447.               bytes: 567160996,
  448.             },
  449.             usage: {
  450.               bytes: 362826547,
  451.               node: {
  452.                 pct: 0.01770037710617187,
  453.               },
  454.               limit: {
  455.                 pct: 0.00009923134671484496,
  456.               },
  457.             },
  458.             workingset: {
  459.               bytes: 1986724,
  460.             },
  461.             rss: {
  462.               bytes: 402801,
  463.             },
  464.             pagefaults: 296495,
  465.             majorpagefaults: 0,
  466.           },
  467.           start_time: "2021-03-30T07:59:06Z",
  468.           name: "container-name-44",
  469.         },
  470.         namespace: "namespace26",
  471.       },
  472.     },
  473.     {
  474.       create: {},
  475.     },
  476.     {
  477.       "@timestamp": "2022-06-21T15:38:30Z",
  478.       kubernetes: {
  479.         host: "gke-apps-0",
  480.         node: "gke-apps-0-0",
  481.         pod: "gke-apps-0-0-0",
  482.         container: {
  483.           cpu: {
  484.             usage: {
  485.               nanocores: 40018,
  486.               core: {
  487.                 ns: 12828317850,
  488.               },
  489.               node: {
  490.                 pct: 0.0000277905,
  491.               },
  492.               limit: {
  493.                 pct: 0.0000277905,
  494.               },
  495.             },
  496.           },
  497.           memory: {
  498.             available: {
  499.               bytes: 1062428344,
  500.             },
  501.             usage: {
  502.               bytes: 265142477,
  503.               node: {
  504.                 pct: 0.01770037710617187,
  505.               },
  506.               limit: {
  507.                 pct: 0.00009923134671484496,
  508.               },
  509.             },
  510.             workingset: {
  511.               bytes: 2294743,
  512.             },
  513.             rss: {
  514.               bytes: 340623,
  515.             },
  516.             pagefaults: 224530,
  517.             majorpagefaults: 0,
  518.           },
  519.           start_time: "2021-03-30T07:59:06Z",
  520.           name: "container-name-44",
  521.         },
  522.         namespace: "namespace26",
  523.       },
  524.     },
  525.   ],
  526. });
  527. console.log(response);
  1. PUT /datastream/_bulk?refresh
  2. {"create": {}}
  3. {"@timestamp":"2022-06-21T15:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  4. {"create": {}}
  5. {"@timestamp":"2022-06-21T15:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  6. {"create": {}}
  7. {"@timestamp":"2022-06-21T15:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  8. {"create": {}}
  9. {"@timestamp":"2022-06-21T15:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  10. {"create": {}}
  11. {"@timestamp":"2022-06-21T15:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  12. {"create": {}}
  13. {"@timestamp":"2022-06-21T15:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  14. {"create": {}}
  15. {"@timestamp":"2022-06-21T15:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  16. {"create": {}}
  17. {"@timestamp":"2022-06-21T15:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  18. {"create": {}}
  19. {"@timestamp":"2022-06-21T15:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
  20. {"create": {}}
  21. {"@timestamp":"2022-06-21T15:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}

View current state of data stream

Now that you’ve created and added documents to the data stream, check to confirm the current state of the new index.

  1. resp = client.indices.get_data_stream()
  2. print(resp)
  1. response = client.indices.get_data_stream
  2. puts response
  1. const response = await client.indices.getDataStream();
  2. console.log(response);
  1. GET _data_stream

If the data stream lifecycle policy has not yet been applied, your results will be like the following. Note the original index_name: .ds-datastream-2024.04.29-000001.

  1. {
  2.   "data_streams": [
  3.     {
  4.       "name": "datastream",
  5.       "timestamp_field": {
  6.         "name": "@timestamp"
  7.       },
  8.       "indices": [
  9.         {
  10.           "index_name": ".ds-datastream-2024.04.29-000001",
  11.           "index_uuid": "vUMNtCyXQhGdlo1BD-cGRw",
  12.           "managed_by": "Data stream lifecycle"
  13.         }
  14.       ],
  15.       "generation": 1,
  16.       "status": "GREEN",
  17.       "template": "datastream_template",
  18.       "lifecycle": {
  19.         "enabled": true,
  20.         "downsampling": [
  21.           {
  22.             "after": "1m",
  23.             "fixed_interval": "1h"
  24.           }
  25.         ]
  26.       },
  27.       "next_generation_managed_by": "Data stream lifecycle",
  28.       "hidden": false,
  29.       "system": false,
  30.       "allow_custom_routing": false,
  31.       "replicated": false,
  32.       "rollover_on_write": false,
  33.       "time_series": {
  34.         "temporal_ranges": [
  35.           {
  36.             "start": "2024-04-29T15:55:46.000Z",
  37.             "end": "2024-04-29T18:25:46.000Z"
  38.           }
  39.         ]
  40.       }
  41.     }
  42.   ]
  43. }

Next, run a search query:

  1. resp = client.search(
  2.     index="datastream",
  3. )
  4. print(resp)
  1. response = client.search(
  2.   index: 'datastream'
  3. )
  4. puts response
  1. const response = await client.search({
  2.   index: "datastream",
  3. });
  4. console.log(response);
  1. GET datastream/_search

The query returns your ten newly added documents.

  1. {
  2.   "took": 23,
  3.   "timed_out": false,
  4.   "_shards": {
  5.     "total": 1,
  6.     "successful": 1,
  7.     "skipped": 0,
  8.     "failed": 0
  9.   },
  10.   "hits": {
  11.     "total": {
  12.       "value": 10,
  13.       "relation": "eq"
  14.     },
  15. ...

Roll over the data stream

Data stream lifecycle will automatically roll over data stream and perform downsampling. This step is only needed in order to see downsampling results in scope of this tutorial.

Roll over the data stream using the rollover API:

  1. resp = client.indices.rollover(
  2.     alias="datastream",
  3. )
  4. print(resp)
  1. const response = await client.indices.rollover({
  2.   alias: "datastream",
  3. });
  4. console.log(response);
  1. POST /datastream/_rollover/

View downsampling results

By default, data stream lifecycle actions are executed every five minutes. Downsampling takes place after the index is rolled over and the index time series end time has lapsed as the source index is still expected to receive major writes until then. Index is now rolled over after previous step but its time series range end is likely still in the future. Once index time series range is in the past, re-run the GET _data_stream request.

  1. resp = client.indices.get_data_stream()
  2. print(resp)
  1. response = client.indices.get_data_stream
  2. puts response
  1. const response = await client.indices.getDataStream();
  2. console.log(response);
  1. GET _data_stream

After the data stream lifecycle action was executed, original .ds-datastream-2024.04.29-000001 index is replaced with a new, downsampled index, in this case downsample-1h-.ds-datastream-2024.04.29-000001.

  1. {
  2.   "data_streams": [
  3.     {
  4.       "name": "datastream",
  5.       "timestamp_field": {
  6.         "name": "@timestamp"
  7.       },
  8.       "indices": [
  9.         {
  10.           "index_name": "downsample-1h-.ds-datastream-2024.04.29-000001",
  11.           "index_uuid": "VqXuShP4T8ODAOnWFcqitg",
  12.           "managed_by": "Data stream lifecycle"
  13.         },
  14.         {
  15.           "index_name": ".ds-datastream-2024.04.29-000002",
  16.           "index_uuid": "8gCeSdjUSWG-o-PeEAJ0jA",
  17.           "managed_by": "Data stream lifecycle"
  18.         }
  19.       ],
  20. ...

Run a search query on the datastream (note that when querying downsampled indices there are a few nuances to be aware of).

  1. resp = client.search(
  2.     index="datastream",
  3. )
  4. print(resp)
  1. response = client.search(
  2.   index: 'datastream'
  3. )
  4. puts response
  1. const response = await client.search({
  2.   index: "datastream",
  3. });
  4. console.log(response);
  1. GET datastream/_search

The new downsampled index contains just one document that includes the min, max, sum, and value_count statistics based off of the original sampled metrics.

  1. {
  2.   "took": 26,
  3.   "timed_out": false,
  4.   "_shards": {
  5.     "total": 2,
  6.     "successful": 2,
  7.     "skipped": 0,
  8.     "failed": 0
  9.   },
  10.   "hits": {
  11.     "total": {
  12.       "value": 1,
  13.       "relation": "eq"
  14.     },
  15.     "max_score": 1,
  16.     "hits": [
  17.       {
  18.         "_index": "downsample-1h-.ds-datastream-2024.04.29-000001",
  19.         "_id": "0eL0wMf38sl_s5JnAAABjyrMjoA",
  20.         "_score": 1,
  21.         "_source": {
  22.           "@timestamp": "2024-04-29T17:00:00.000Z",
  23.           "_doc_count": 10,
  24.           "kubernetes": {
  25.             "container": {
  26.               "cpu": {
  27.                 "usage": {
  28.                   "core": {
  29.                     "ns": 12828317850
  30.                   },
  31.                   "limit": {
  32.                     "pct": 0.0000277905
  33.                   },
  34.                   "nanocores": {
  35.                     "min": 38907,
  36.                     "max": 153404,
  37.                     "sum": 992677,
  38.                     "value_count": 10
  39.                   },
  40.                   "node": {
  41.                     "pct": 0.0000277905
  42.                   }
  43.                 }
  44.               },
  45.               "memory": {
  46.                 "available": {
  47.                   "bytes": {
  48.                     "min": 279586406,
  49.                     "max": 1062428344,
  50.                     "sum": 7101494721,
  51.                     "value_count": 10
  52.                   }
  53.                 },
  54.                 "majorpagefaults": 0,
  55.                 "pagefaults": {
  56.                   "min": 74843,
  57.                   "max": 302252,
  58.                   "sum": 2061071,
  59.                   "value_count": 10
  60.                 },
  61.                 "rss": {
  62.                   "bytes": {
  63.                     "min": 91914,
  64.                     "max": 402801,
  65.                     "sum": 2389770,
  66.                     "value_count": 10
  67.                   }
  68.                 },
  69.                 "usage": {
  70.                   "bytes": {
  71.                     "min": 100475044,
  72.                     "max": 379572388,
  73.                     "sum": 2668170609,
  74.                     "value_count": 10
  75.                   },
  76.                   "limit": {
  77.                     "pct": 0.00009923134
  78.                   },
  79.                   "node": {
  80.                     "pct": 0.017700378
  81.                   }
  82.                 },
  83.                 "workingset": {
  84.                   "bytes": {
  85.                     "min": 431227,
  86.                     "max": 2294743,
  87.                     "sum": 14230488,
  88.                     "value_count": 10
  89.                   }
  90.                 }
  91.               },
  92.               "name": "container-name-44",
  93.               "start_time": "2021-03-30T07:59:06.000Z"
  94.             },
  95.             "host": "gke-apps-0",
  96.             "namespace": "namespace26",
  97.             "node": "gke-apps-0-0",
  98.             "pod": "gke-apps-0-0-0"
  99.           }
  100.         }
  101.       }
  102.     ]
  103.   }
  104. }

Use the data stream stats API to get statistics for the data stream, including the storage size.

  1. resp = client.indices.data_streams_stats(
  2.     name="datastream",
  3.     human=True,
  4. )
  5. print(resp)
  1. response = client.indices.data_streams_stats(
  2.   name: 'datastream',
  3.   human: true
  4. )
  5. puts response
  1. const response = await client.indices.dataStreamsStats({
  2.   name: "datastream",
  3.   human: "true",
  4. });
  5. console.log(response);
  1. GET /_data_stream/datastream/_stats?human=true
  1. {
  2.   "_shards": {
  3.     "total": 4,
  4.     "successful": 4,
  5.     "failed": 0
  6.   },
  7.   "data_stream_count": 1,
  8.   "backing_indices": 2,
  9.   "total_store_size": "37.3kb",
  10.   "total_store_size_bytes": 38230,
  11.   "data_streams": [
  12.     {
  13.       "data_stream": "datastream",
  14.       "backing_indices": 2,
  15.       "store_size": "37.3kb",
  16.       "store_size_bytes": 38230,
  17.       "maximum_timestamp": 1714410000000
  18.     }
  19.   ]
  20. }

This example demonstrates how downsampling works as part of a data stream lifecycle to reduce the storage size of metrics data as it becomes less current and less frequently queried.