Collect Kubernetes Events

In addition to using Logconfig to collect logs, Loggie can also configure any source/sink/interceptor through CRD. In essence, Loggie is a data stream that supports multiple Pipelines, integrating common functions such as queue retry, data processing, configuration delivery, monitoring alarms and other functions, which reduces the development cost for similar requirements. Collecting Kubernetes Events is a good example of this.

Kubernetes Events are events generated by Kubernetes’ own components and some controllers. We can use kubectl describe to view the event information of associated resources. Collecting and recording these events can help us trace back, troubleshoot, audit, and summarize problems, and better understand internal state of Kubernetes clusters.

Preparation

Similar to Loggie Aggregator, we can Deploy Aggregator cluster separately or reuse the existing aggregator cluster.

Configuration Example

Configure the kubeEvents source and use type: cluster to distribute configuration to the Aggregator cluster.

Config

  1. apiVersion: loggie.io/v1beta1
  2. kind: ClusterLogConfig
  3. metadata:
  4.   name: kubeevent
  5. spec:
  6.   selector:
  7.     type: cluster
  8.     cluster: aggregator
  9.   pipeline:
  10.     sources: |
  11.       - type: kubeEvent
  12.         name: event
  13.     sinkRef: dev

By default, whether it is sent to Elasticsearch or other sinks, the output is in a format similar to the following:

event

  1. {
  2. "body": "{\"metadata\":{\"name\":\"loggie-aggregator.16c277f8fc4ff0d0\",\"namespace\":\"loggie-aggregator\",\"uid\":\"084cea27-cd4a-4ce4-97ef-12e70f37880e\",\"resourceVersion\":\"2975193\",\"creationTimestamp\":\"2021-12-20T12:58:45Z\",\"managedFields\":[{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2021-12-20T12:58:45Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:count\":{},\"f:firstTimestamp\":{},\"f:involvedObject\":{\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{},\"f:resourceVersion\":{},\"f:uid\":{}},\"f:lastTimestamp\":{},\"f:message\":{},\"f:reason\":{},\"f:source\":{\"f:component\":{}},\"f:type\":{}}}]},\"involvedObject\":{\"kind\":\"DaemonSet\",\"namespace\":\"loggie-aggregator\",\"name\":\"loggie-aggregator\",\"uid\":\"7cdf4792-815d-4eba-8a81-d60131ad1fc4\",\"apiVersion\":\"apps/v1\",\"resourceVersion\":\"2975170\"},\"reason\":\"SuccessfulCreate\",\"message\":\"Created pod: loggie-aggregator-pbkjk\",\"source\":{\"component\":\"daemonset-controller\"},\"firstTimestamp\":\"2021-12-20T12:58:45Z\",\"lastTimestamp\":\"2021-12-20T12:58:45Z\",\"count\":1,\"type\":\"Normal\",\"eventTime\":null,\"reportingComponent\":\"\",\"reportingInstance\":\"\"}",
  3. "systemPipelineName": "default/kubeevent/",
  4. "systemSourceName": "event"
  5. }

To facilitate analysis and display, we can add some interceptors to json-decode the collected events data.

The configuration example is as follows. For details, please refer to Log Segmentation。

Config

interceptor

  1. apiVersion: loggie.io/v1beta1
  2. kind: Interceptor
  3. metadata:
  4.   name: jsondecode
  5. spec:
  6.   interceptors: |
  7.     - type: normalize
  8.       name: json
  9.       processors:
  10.       - jsonDecode: ~
  11.       - drop:
  12.           targets: ["body"]

clusterLogConfig

  1. apiVersion: loggie.io/v1beta1
  2. kind: ClusterLogConfig
  3. metadata:
  4.   name: kubeevent
  5. spec:
  6.   selector:
  7.     type: cluster
  8.     cluster: aggregator
  9.   pipeline:
  10.     sources: |
  11.       - type: kubeEvent
  12.         name: event
  13.     interceptorRef: jsondecode
  14.     sinkRef: dev

The data after jsonDecode in normalize interceptor is as follows:

event

  1. {
  2. "metadata": {
  3.     "name": "loggie-aggregator.16c277f8fc4ff0d0",
  4.     "namespace": "loggie-aggregator",
  5.     "uid": "084cea27-cd4a-4ce4-97ef-12e70f37880e",
  6.     "resourceVersion": "2975193",
  7.     "creationTimestamp": "2021-12-20T12:58:45Z",
  8.     "managedFields": [
  9.         {
  10.             "fieldsType": "FieldsV1",
  11.             "fieldsV1": {
  12.                 "f:type": {
  14.                 },
  15.                 "f:count": {
  17.                 },
  18.                 "f:firstTimestamp": {
  20.                 },
  21.                 "f:involvedObject": {
  22.                     "f:apiVersion": {
  24.                     },
  25.                     "f:kind": {
  27.                     },
  28.                     "f:name": {
  30.                     },
  31.                     "f:namespace": {
  33.                     },
  34.                     "f:resourceVersion": {
  36.                     },
  37.                     "f:uid": {
  39.                     }
  40.                 },
  41.                 "f:lastTimestamp": {
  43.                 },
  44.                 "f:message": {
  46.                 },
  47.                 "f:reason": {
  49.                 },
  50.                 "f:source": {
  51.                     "f:component": {
  53.                     }
  54.                 }
  55.             },
  56.             "manager": "kube-controller-manager",
  57.             "operation": "Update",
  58.             "apiVersion": "v1",
  59.             "time": "2021-12-20T12:58:45Z"
  60.         }
  61.     ]
  62. },
  63. "reportingComponent": "",
  64. "type": "Normal",
  65. "message": "Created pod: loggie-aggregator-pbkjk",
  66. "reason": "SuccessfulCreate",
  67. "reportingInstance": "",
  68. "source": {
  69.     "component": "daemonset-controller"
  70. },
  71. "count": 1,
  72. "lastTimestamp": "2021-12-20T12:58:45Z",
  73. "firstTimestamp": "2021-12-20T12:58:45Z",
  74. "eventTime": null,
  75. "involvedObject": {
  76.     "kind": "DaemonSet",
  77.     "namespace": "loggie-aggregator",
  78.     "name": "loggie-aggregator",
  79.     "uid": "7cdf4792-815d-4eba-8a81-d60131ad1fc4",
  80.     "apiVersion": "apps/v1",
  81.     "resourceVersion": "2975170"
  82. },
  83. }

If you feel that there are too many data fields or the format does not meet the requirements, you can also configure the normalize interceptor to modify it.