Collecting and storing Kubernetes events

The OKD Event Router is a pod that watches Kubernetes events and logs them for collection by cluster logging. You must manually deploy the Event Router.

The Event Router collects events from all projects and writes them to STDOUT. Fluentd collects those events and forwards them into the OKD Elasticsearch instance. Elasticsearch indexes the events to the infra index.

The Event Router adds additional load to Fluentd and can impact the number of other log messages that can be processed.

Deploying and configuring the Event Router

Use the following steps to deploy the Event Router into your cluster. You should always deploy the Event Router to the openshift-logging project to ensure it collects events from across the cluster.

The following Template object creates the service account, cluster role, and cluster role binding required for the Event Router. The template also configures and deploys the Event Router pod. You can use this template without making changes, or change the deployment object CPU and memory requests.

Prerequisites

  • You need proper permissions to create service accounts and update cluster role bindings. For example, you can run the following template with a user that has the cluster-admin role.

  • Cluster logging must be installed.

Procedure

  1. Create a template for the Event Router:

    1. kind: Template
    2. apiVersion: v1
    3. metadata:
    4.   name: eventrouter-template
    5.   annotations:
    6.     description: "A pod forwarding kubernetes events to cluster logging stack."
    7.     tags: "events,EFK,logging,cluster-logging"
    8. objects:
    9.   - kind: ServiceAccount (1)
    10.     apiVersion: v1
    11.     metadata:
    12.       name: eventrouter
    13.       namespace: ${NAMESPACE}
    14.   - kind: ClusterRole (2)
    15.     apiVersion: v1
    16.     metadata:
    17.       name: event-reader
    18.     rules:
    19.     - apiGroups: [""]
    20.       resources: ["events"]
    21.       verbs: ["get", "watch", "list"]
    22.   - kind: ClusterRoleBinding  (3)
    23.     apiVersion: v1
    24.     metadata:
    25.       name: event-reader-binding
    26.     subjects:
    27.     - kind: ServiceAccount
    28.       name: eventrouter
    29.       namespace: ${NAMESPACE}
    30.     roleRef:
    31.       kind: ClusterRole
    32.       name: event-reader
    33.   - kind: ConfigMap (4)
    34.     apiVersion: v1
    35.     metadata:
    36.       name: eventrouter
    37.       namespace: ${NAMESPACE}
    38.     data:
    39.       config.json: |-
    40.         {
    41.           "sink": "stdout"
    42.         }
    43.   - kind: Deployment (5)
    44.     apiVersion: apps/v1
    45.     metadata:
    46.       name: eventrouter
    47.       namespace: ${NAMESPACE}
    48.       labels:
    49.         component: "eventrouter"
    50.         logging-infra: "eventrouter"
    51.         provider: "openshift"
    52.     spec:
    53.       selector:
    54.         matchLabels:
    55.           component: "eventrouter"
    56.           logging-infra: "eventrouter"
    57.           provider: "openshift"
    58.       replicas: 1
    59.       template:
    60.         metadata:
    61.           labels:
    62.             component: "eventrouter"
    63.             logging-infra: "eventrouter"
    64.             provider: "openshift"
    65.           name: eventrouter
    66.         spec:
    67.           serviceAccount: eventrouter
    68.           containers:
    69.             - name: kube-eventrouter
    70.               image: ${IMAGE}
    71.               imagePullPolicy: IfNotPresent
    72.               resources:
    73.                 requests:
    74.                   cpu: ${CPU}
    75.                   memory: ${MEMORY}
    76.               volumeMounts:
    77.               - name: config-volume
    78.                 mountPath: /etc/eventrouter
    79.           volumes:
    80.             - name: config-volume
    81.               configMap:
    82.                 name: eventrouter
    83. parameters:
    84.   - name: IMAGE
    85.     displayName: Image
    86.     value: "registry.redhat.io/openshift4/ose-logging-eventrouter:latest"
    87.   - name: CPU  (6)
    88.     displayName: CPU
    89.     value: "100m"
    90.   - name: MEMORY (7)
    91.     displayName: Memory
    92.     value: "128Mi"
    93.   - name: NAMESPACE
    94.     displayName: Namespace
    95.     value: "openshift-logging" (8)
    1Creates a Service Account in the openshift-logging project for the Event Router.
    2Creates a ClusterRole to monitor for events in the cluster.
    3Creates a ClusterRoleBinding to bind the ClusterRole to the service account.
    4Creates a config map in the openshift-logging project to generate the required config.json file.
    5Creates a deployment in the openshift-logging project to generate and configure the Event Router pod.
    6Specifies the minimum amount of memory to allocate to the Event Router pod. Defaults to 128Mi.
    7Specifies the minimum amount of CPU to allocate to the Event Router pod. Defaults to 100m.
    8Specifies the openshift-logging project to install objects in.

  2. Use the following command to process and apply the template:

    1. $ oc process -f <templatefile> | oc apply -n openshift-logging -f -

    For example:

    1. $ oc process -f eventrouter.yaml | oc apply -n openshift-logging -f -

    Example output

    1. serviceaccount/logging-eventrouter created
    2. clusterrole.authorization.openshift.io/event-reader created
    3. clusterrolebinding.authorization.openshift.io/event-reader-binding created
    4. configmap/logging-eventrouter created
    5. deployment.apps/logging-eventrouter created

  3. Validate that the Event Router installed in the openshift-logging project:

    1. View the new Event Router pod:

      1. $ oc get pods --selector  component=eventrouter -o name -n openshift-logging

      Example output

      1. pod/cluster-logging-eventrouter-d649f97c8-qvv8r

    2. View the events collected by the Event Router:

      1. $ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging

      For example:

      1. $ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging

      Example output

      1. {"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}

      You can also use Kibana to view events by creating an index pattern using the Elasticsearch infra index.