Collecting and storing Kubernetes events

The OKD Event Router is a pod that watches Kubernetes events and logs them for collection by the logging subsystem. You must manually deploy the Event Router.

The Event Router collects events from all projects and writes them to STDOUT. The collector then forwards those events to the store defined in the ClusterLogForwarder custom resource (CR).

The Event Router adds additional load to Fluentd and can impact the number of other log messages that can be processed.

Deploying and configuring the Event Router

Use the following steps to deploy the Event Router into your cluster. You should always deploy the Event Router to the openshift-logging project to ensure it collects events from across the cluster.

The Event Router image is not a part of the Red Hat OpenShift Logging Operator and must be downloaded separately.

The following Template object creates the service account, cluster role, and cluster role binding required for the Event Router. The template also configures and deploys the Event Router pod. You can either use this template without making changes or edit the template to change the deployment object CPU and memory requests.

Prerequisites

  • You need proper permissions to create service accounts and update cluster role bindings. For example, you can run the following template with a user that has the cluster-admin role.

  • The logging subsystem for Red Hat OpenShift must be installed.

Procedure

  1. Create a template for the Event Router:

    1. apiVersion: template.openshift.io/v1
    2. kind: Template
    3. metadata:
    4.   name: eventrouter-template
    5.   annotations:
    6.     description: "A pod forwarding kubernetes events to OpenShift Logging stack."
    7.     tags: "events,EFK,logging,cluster-logging"
    8. objects:
    9.   - kind: ServiceAccount (1)
    10.     apiVersion: v1
    11.     metadata:
    12.       name: eventrouter
    13.       namespace: ${NAMESPACE}
    14.   - kind: ClusterRole (2)
    15.     apiVersion: rbac.authorization.k8s.io/v1
    16.     metadata:
    17.       name: event-reader
    18.     rules:
    19.     - apiGroups: [""]
    20.       resources: ["events"]
    21.       verbs: ["get", "watch", "list"]
    22.   - kind: ClusterRoleBinding (3)
    23.     apiVersion: rbac.authorization.k8s.io/v1
    24.     metadata:
    25.       name: event-reader-binding
    26.     subjects:
    27.     - kind: ServiceAccount
    28.       name: eventrouter
    29.       namespace: ${NAMESPACE}
    30.     roleRef:
    31.       kind: ClusterRole
    32.       name: event-reader
    33.   - kind: ConfigMap (4)
    34.     apiVersion: v1
    35.     metadata:
    36.       name: eventrouter
    37.       namespace: ${NAMESPACE}
    38.     data:
    39.       config.json: |-
    40.         {
    41.           "sink": "stdout"
    42.         }
    43.   - kind: Deployment (5)
    44.     apiVersion: apps/v1
    45.     metadata:
    46.       name: eventrouter
    47.       namespace: ${NAMESPACE}
    48.       labels:
    49.         component: "eventrouter"
    50.         logging-infra: "eventrouter"
    51.         provider: "openshift"
    52.     spec:
    53.       selector:
    54.         matchLabels:
    55.           component: "eventrouter"
    56.           logging-infra: "eventrouter"
    57.           provider: "openshift"
    58.       replicas: 1
    59.       template:
    60.         metadata:
    61.           labels:
    62.             component: "eventrouter"
    63.             logging-infra: "eventrouter"
    64.             provider: "openshift"
    65.           name: eventrouter
    66.         spec:
    67.           serviceAccount: eventrouter
    68.           containers:
    69.             - name: kube-eventrouter
    70.               image: ${IMAGE}
    71.               imagePullPolicy: IfNotPresent
    72.               resources:
    73.                 requests:
    74.                   cpu: ${CPU}
    75.                   memory: ${MEMORY}
    76.               volumeMounts:
    77.               - name: config-volume
    78.                 mountPath: /etc/eventrouter
    79.               securityContext:
    80.                 allowPrivilegeEscalation: false
    81.                 capabilities:
    82.                   drop: ["ALL"]
    83.           securityContext:
    84.             runAsNonRoot: true
    85.             seccompProfile:
    86.               type: RuntimeDefault
    87.           volumes:
    88.           - name: config-volume
    89.             configMap:
    90.               name: eventrouter
    91. parameters:
    92.   - name: IMAGE (6)
    93.     displayName: Image
    94.     value: "registry.redhat.io/openshift-logging/eventrouter-rhel9:v0.4"
    95.   - name: CPU (7)
    96.     displayName: CPU
    97.     value: "100m"
    98.   - name: MEMORY (8)
    99.     displayName: Memory
    100.     value: "128Mi"
    101.   - name: NAMESPACE
    102.     displayName: Namespace
    103.     value: "openshift-logging" (9)
    1Creates a Service Account in the openshift-logging project for the Event Router.
    2Creates a ClusterRole to monitor for events in the cluster.
    3Creates a ClusterRoleBinding to bind the ClusterRole to the service account.
    4Creates a config map in the openshift-logging project to generate the required config.json file.
    5Creates a deployment in the openshift-logging project to generate and configure the Event Router pod.
    6Specifies the image, identified by a tag such as v0.4.
    7Specifies the minimum amount of CPU to allocate to the Event Router pod. Defaults to 100m.
    8Specifies the minimum amount of memory to allocate to the Event Router pod. Defaults to 128Mi.
    9Specifies the openshift-logging project to install objects in.

  2. Use the following command to process and apply the template:

    1. $ oc process -f <templatefile> | oc apply -n openshift-logging -f -

    For example:

    1. $ oc process -f eventrouter.yaml | oc apply -n openshift-logging -f -

    Example output

    1. serviceaccount/eventrouter created
    2. clusterrole.rbac.authorization.k8s.io/event-reader created
    3. clusterrolebinding.rbac.authorization.k8s.io/event-reader-binding created
    4. configmap/eventrouter created
    5. deployment.apps/eventrouter created

  3. Validate that the Event Router installed in the openshift-logging project:

    1. View the new Event Router pod:

      1. $ oc get pods --selector  component=eventrouter -o name -n openshift-logging

      Example output

      1. pod/cluster-logging-eventrouter-d649f97c8-qvv8r

    2. View the events collected by the Event Router:

      1. $ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging

      For example:

      1. $ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging

      Example output

      1. {"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}

      You can also use Kibana to view events by creating an index pattern using the Elasticsearch infra index.