Configuring log forwarding

By default, the logging sends container and infrastructure logs to the default internal log store defined in the ClusterLogging custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, you do not need to configure the Cluster Log Forwarder.

To send audit logs to the internal Elasticsearch log store, use the Cluster Log Forwarder as described in Forwarding audit logs to the log store.

About forwarding logs to third-party systems

To send logs to specific endpoints inside and outside your OKD cluster, you specify a combination of outputs and pipelines in a ClusterLogForwarder custom resource (CR). You can also use inputs to forward the application logs associated with a specific project to an endpoint. Authentication is provided by a Kubernetes Secret object.

pipeline

Defines simple routing from one log type to one or more outputs, or which logs you want to send. The log types are one of the following:

  • application. Container logs generated by user applications running in the cluster, except infrastructure container applications.

  • infrastructure. Container logs from pods that run in the openshift*, kube*, or default projects and journal logs sourced from node file system.

  • audit. Audit logs generated by the node audit system, auditd, Kubernetes API server, OpenShift API server, and OVN network.

You can add labels to outbound log messages by using key:value pairs in the pipeline. For example, you might add a label to messages that are forwarded to other data centers or label the logs by type. Labels that are added to objects are also forwarded with the log message.

input

Forwards the application logs associated with a specific project to a pipeline.

In the pipeline, you define which log types to forward using an inputRef parameter and where to forward the logs to using an outputRef parameter.

Secret

A key:value map that contains confidential data such as user credentials.

Note the following:

  • If a ClusterLogForwarder CR object exists, logs are not forwarded to the default Elasticsearch instance, unless there is a pipeline with the default output.

  • By default, the logging sends container and infrastructure logs to the default internal Elasticsearch log store defined in the ClusterLogging custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, do not configure the Log Forwarding API.

  • If you do not define a pipeline for a log type, the logs of the undefined types are dropped. For example, if you specify a pipeline for the application and audit types, but do not specify a pipeline for the infrastructure type, infrastructure logs are dropped.

  • You can use multiple types of outputs in the ClusterLogForwarder custom resource (CR) to send logs to servers that support different protocols.

  • The internal OKD Elasticsearch instance does not provide secure storage for audit logs. We recommend you ensure that the system to which you forward audit logs is compliant with your organizational and governmental regulations and is properly secured. The logging does not comply with those regulations.

The following example forwards the audit logs to a secure external Elasticsearch instance, the infrastructure logs to an insecure external Elasticsearch instance, the application logs to a Kafka broker, and the application logs from the my-apps-logs project to the internal Elasticsearch instance.

Sample log forwarding outputs and pipelines

  1. apiVersion: "logging.openshift.io/v1"
  2. kind: ClusterLogForwarder
  3. metadata:
  4.   name: <log_forwarder_name> (1)
  5.   namespace: <log_forwarder_namespace> (2)
  6. spec:
  7.   serviceAccountName: <service_account_name> (3)
  8.   outputs:
  9.    - name: elasticsearch-secure (4)
  10.      type: "elasticsearch"
  11.      url: https://elasticsearch.secure.com:9200
  12.      secret:
  13.         name: elasticsearch
  14.    - name: elasticsearch-insecure (5)
  15.      type: "elasticsearch"
  16.      url: http://elasticsearch.insecure.com:9200
  17.    - name: kafka-app (6)
  18.      type: "kafka"
  19.      url: tls://kafka.secure.com:9093/app-topic
  20.   inputs: (7)
  21.    - name: my-app-logs
  22.      application:
  23.         namespaces:
  24.         - my-project
  25.   pipelines:
  26.    - name: audit-logs (8)
  27.      inputRefs:
  28.       - audit
  29.      outputRefs:
  30.       - elasticsearch-secure
  31.       - default
  32.      labels:
  33.        secure: "true" (9)
  34.        datacenter: "east"
  35.    - name: infrastructure-logs (10)
  36.      inputRefs:
  37.       - infrastructure
  38.      outputRefs:
  39.       - elasticsearch-insecure
  40.      labels:
  41.        datacenter: "west"
  42.    - name: my-app (11)
  43.      inputRefs:
  44.       - my-app-logs
  45.      outputRefs:
  46.       - default
  47.    - inputRefs: (12)
  48.       - application
  49.      outputRefs:
  50.       - kafka-app
  51.      labels:
  52.        datacenter: "south"
1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4Configuration for an secure Elasticsearch output using a secret with a secure URL.

  • A name to describe the output.

  • The type of output: elasticsearch.

  • The secure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.

  • The secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project.

5Configuration for an insecure Elasticsearch output:

  • A name to describe the output.

  • The type of output: elasticsearch.

  • The insecure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.

6Configuration for a Kafka output using a client-authenticated TLS communication over a secure URL:

  • A name to describe the output.

  • The type of output: kafka.

  • Specify the URL and port of the Kafka broker as a valid absolute URL, including the prefix.

7Configuration for an input to filter application logs from the my-project namespace.
8Configuration for a pipeline to send audit logs to the secure external Elasticsearch instance:

  • A name to describe the pipeline.

  • The inputRefs is the log type, in this example audit.

  • The outputRefs is the name of the output to use, in this example elasticsearch-secure to forward to the secure Elasticsearch instance and default to forward to the internal Elasticsearch instance.

  • Optional: Labels to add to the logs.

9Optional: String. One or more labels to add to the logs. Quote values like “true” so they are recognized as string values, not as a boolean.
10Configuration for a pipeline to send infrastructure logs to the insecure external Elasticsearch instance.
11Configuration for a pipeline to send logs from the my-project project to the internal Elasticsearch instance.

  • A name to describe the pipeline.

  • The inputRefs is a specific input: my-app-logs.

  • The outputRefs is default.

  • Optional: String. One or more labels to add to the logs.

12Configuration for a pipeline to send logs to the Kafka broker, with no pipeline name:

  • The inputRefs is the log type, in this example application.

  • The outputRefs is the name of the output to use.

  • Optional: String. One or more labels to add to the logs.

Fluentd log handling when the external log aggregator is unavailable

If your external logging aggregator becomes unavailable and cannot receive logs, Fluentd continues to collect logs and stores them in a buffer. When the log aggregator becomes available, log forwarding resumes, including the buffered logs. If the buffer fills completely, Fluentd stops collecting logs. OKD rotates the logs and deletes them. You cannot adjust the buffer size or add a persistent volume claim (PVC) to the Fluentd daemon set or pods.

Supported Authorization Keys

Common key types are provided here. Some output types support additional specialized keys, documented with the output-specific configuration field. All secret keys are optional. Enable the security features you want by setting the relevant keys. You are responsible for creating and maintaining any additional configurations that external destinations might require, such as keys and secrets, service accounts, port openings, or global proxy configuration. Open Shift Logging will not attempt to verify a mismatch between authorization combinations.

Transport Layer Security (TLS)

Using a TLS URL (http://... or ssl://...) without a secret enables basic TLS server-side authentication. Additional TLS features are enabled by including a secret and setting the following optional fields:

  • passphrase: (string) Passphrase to decode an encoded TLS private key. Requires tls.key.

  • ca-bundle.crt: (string) File name of a customer CA for server authentication.

Username and Password

  • username: (string) Authentication user name. Requires password.

  • password: (string) Authentication password. Requires username.

Simple Authentication Security Layer (SASL)

  • sasl.enable (boolean) Explicitly enable or disable SASL. If missing, SASL is automatically enabled when any of the other sasl. keys are set.

  • sasl.mechanisms: (array) List of allowed SASL mechanism names. If missing or empty, the system defaults are used.

  • sasl.allow-insecure: (boolean) Allow mechanisms that send clear-text passwords. Defaults to false.

Creating a Secret

You can create a secret in the directory that contains your certificate and key files by using the following command:

  1. $ oc create secret generic -n <namespace> <secret_name> \
  2.   --from-file=ca-bundle.crt=<your_bundle_file> \
  3.   --from-literal=username=<your_username> \
  4.   --from-literal=password=<your_password>

Generic or opaque secrets are recommended for best results.

Creating a log forwarder

To create a log forwarder, you must create a ClusterLogForwarder CR that specifies the log input types that the service account can collect. You can also specify which outputs the logs can be forwarded to. If you are using the multi log forwarder feature, you must also reference the service account in the ClusterLogForwarder CR.

If you are using the multi log forwarder feature on your cluster, you can create ClusterLogForwarder custom resources (CRs) in any namespace, using any name. If you are using a legacy implementation, the ClusterLogForwarder CR must be named instance, and must be created in the openshift-logging namespace.

You need administrator permissions for the namespace where you create the ClusterLogForwarder CR.

ClusterLogForwarder resource example

  1. apiVersion: logging.openshift.io/v1
  2. kind: ClusterLogForwarder
  3. metadata:
  4.   name: <log_forwarder_name> (1)
  5.   namespace: <log_forwarder_namespace> (2)
  6. spec:
  7.   serviceAccountName: <service_account_name> (3)
  8.   pipelines:
  9.    - inputRefs:
  10.      - <log_type> (4)
  11.      outputRefs:
  12.      - <output_name> (5)
  13.   outputs:
  14.   - name: <output_name> (6)
  15.     type: <output_type> (5)
  16.     url: <log_output_url> (7)
  17. # ...
1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4The log types that are collected. The value for this field can be audit for audit logs, application for application logs, infrastructure for infrastructure logs, or a named input that has been defined for your application.
5The type of output that you want to forward logs to. The value of this field can be default, loki, kafka, elasticsearch, fluentdForward, syslog, or cloudwatch.

The default output type is not supported in mutli log forwarder implementations.

6A name for the output that you want to forward logs to.
7The URL of the output that you want to forward logs to.

Enabling multi-line exception detection

Enables multi-line error detection of container logs.

Enabling this feature could have performance implications and may require additional computing resources or alternate logging solutions.

Log parsers often incorrectly identify separate lines of the same exception as separate exceptions. This leads to extra log entries and an incomplete or inaccurate view of the traced information.

Example java exception

  1. java.lang.NullPointerException: Cannot invoke "String.toString()" because "<param1>" is null
  2.     at testjava.Main.handle(Main.java:47)
  3.     at testjava.Main.printMe(Main.java:19)
  4.     at testjava.Main.main(Main.java:10)
  • To enable logging to detect multi-line exceptions and reassemble them into a single log entry, ensure that the ClusterLogForwarder Custom Resource (CR) contains a detectMultilineErrors field, with a value of true.

Example ClusterLogForwarder CR

  1. apiVersion: logging.openshift.io/v1
  2. kind: ClusterLogForwarder
  3. metadata:
  4.   name: instance
  5.   namespace: openshift-logging
  6. spec:
  7.   pipelines:
  8.     - name: my-app-logs
  9.       inputRefs:
  10.         - application
  11.       outputRefs:
  12.         - default
  13.       detectMultilineErrors: true

Details

When log messages appear as a consecutive sequence forming an exception stack trace, they are combined into a single, unified log record. The first log message’s content is replaced with the concatenated content of all the message fields in the sequence.

Table 1. Supported languages per collector:
LanguageFluentdVector

Java

✓

✓

JS

✓

✓

Ruby

✓

✓

Python

✓

✓

Golang

✓

✓

PHP

✓

✓

Dart

✓

✓

Troubleshooting

When enabled, the collector configuration will include a new section with type: detect_exceptions

Example vector configuration section

  1. [transforms.detect_exceptions_app-logs]
  2.  type = "detect_exceptions"
  3.  inputs = ["application"]
  4.  languages = ["All"]
  5.  group_by = ["kubernetes.namespace_name","kubernetes.pod_name","kubernetes.container_name"]
  6.  expire_after_ms = 2000
  7.  multiline_flush_interval_ms = 1000

Example fluentd config section

  1. <label @MULTILINE_APP_LOGS>
  2.   <match kubernetes.**>
  3.     @type detect_exceptions
  4.     remove_tag_prefix 'kubernetes'
  5.     message message
  6.     force_line_breaks true
  7.     multiline_flush_interval .2
  8.   </match>
  9. </label>

Forwarding logs to Google Cloud Platform (GCP)

You can forward logs to Google Cloud Logging in addition to, or instead of, the internal default OKD log store.

Using this feature with Fluentd is not supported.

Prerequisites

  • Red Hat OpenShift Logging Operator 5.5.1 and later

Procedure

  1. Create a secret using your Google service account key.

    1. $ oc -n openshift-logging create secret generic gcp-secret --from-file google-application-credentials.json=<your_service_account_key_file.json>

  2. Create a ClusterLogForwarder Custom Resource YAML using the template below:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.     - name: gcp-1
    10.       type: googleCloudLogging
    11.       secret:
    12.         name: gcp-secret
    13.       googleCloudLogging:
    14.         projectId : "openshift-gce-devel" (4)
    15.         logId : "app-gcp" (5)
    16.   pipelines:
    17.     - name: test-app
    18.       inputRefs: (6)
    19.         - application
    20.       outputRefs:
    21.         - gcp-1
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Set a projectId, folderId, organizationId, or billingAccountId field and its corresponding value, depending on where you want to store your logs in the GCP resource hierarchy.
    5Set the value to add to the logName field of the Log Entry.
    6Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

Additional resources

Forwarding logs to Splunk

You can forward logs to the Splunk HTTP Event Collector (HEC) in addition to, or instead of, the internal default OKD log store.

Using this feature with Fluentd is not supported.

Prerequisites

  • Red Hat OpenShift Logging Operator 5.6 or later

  • A ClusterLogging instance with vector specified as the collector

  • Base64 encoded Splunk HEC token

Procedure

  1. Create a secret using your Base64 encoded Splunk HEC token.

    1. $ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

  2. Create or edit the ClusterLogForwarder Custom Resource (CR) using the template below:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.     - name: splunk-receiver (4)
    10.       secret:
    11.         name: vector-splunk-secret (5)
    12.       type: splunk (6)
    13.       url: <http://your.splunk.hec.url:8088> (7)
    14.   pipelines: (8)
    15.     - inputRefs:
    16.         - application
    17.         - infrastructure
    18.       name: (9)
    19.       outputRefs:
    20.         - splunk-receiver (10)
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Specify a name for the output.
    5Specify the name of the secret that contains your HEC token.
    6Specify the output type as splunk.
    7Specify the URL (including port) of your Splunk HEC.
    8Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    9Optional: Specify a name for the pipeline.
    10Specify the name of the output to use when forwarding logs with this pipeline.

Forwarding logs over HTTP

Forwarding logs over HTTP is supported for both the Fluentd and Vector log collectors. To enable, specify http as the output type in the ClusterLogForwarder custom resource (CR).

Procedure

  • Create or edit the ClusterLogForwarder CR using the template below:

    Example ClusterLogForwarder CR

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.     - name: httpout-app
    10.       type: http
    11.       url: (4)
    12.       http:
    13.         headers: (5)
    14.           h1: v1
    15.           h2: v2
    16.         method: POST
    17.       secret:
    18.         name: (6)
    19.       tls:
    20.         insecureSkipVerify: (7)
    21.   pipelines:
    22.     - name:
    23.       inputRefs:
    24.         - application
    25.       outputRefs:
    26.         - (8)
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Destination address for logs.
    5Additional headers to send with the log record.
    6Secret name for destination credentials.
    7Values are either true or false.
    8This value should be the same as the output name.

Forwarding application logs from specific projects

You can forward a copy of the application logs from specific projects to an external log aggregator, in addition to, or instead of, using the internal log store. You must also configure the external log aggregator to receive log data from OKD.

To configure forwarding application logs from a project, you must create a ClusterLogForwarder custom resource (CR) with at least one input from a project, optional outputs for other log aggregators, and pipelines that use those inputs and outputs.

Prerequisites

  • You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR:

    Example ClusterLogForwarder CR

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: instance (1)
    5.   namespace: openshift-logging (2)
    6. spec:
    7.   outputs:
    8.    - name: fluentd-server-secure (3)
    9.      type: fluentdForward (4)
    10.      url: 'tls://fluentdserver.security.example.com:24224' (5)
    11.      secret: (6)
    12.         name: fluentd-secret
    13.    - name: fluentd-server-insecure
    14.      type: fluentdForward
    15.      url: 'tcp://fluentdserver.home.example.com:24224'
    16.   inputs: (7)
    17.    - name: my-app-logs
    18.      application:
    19.         namespaces:
    20.         - my-project (8)
    21.   pipelines:
    22.    - name: forward-to-fluentd-insecure (9)
    23.      inputRefs: (10)
    24.      - my-app-logs
    25.      outputRefs: (11)
    26.      - fluentd-server-insecure
    27.      labels:
    28.        project: "my-project" (12)
    29.    - name: forward-to-fluentd-secure (13)
    30.      inputRefs:
    31.      - application (14)
    32.      - audit
    33.      - infrastructure
    34.      outputRefs:
    35.      - fluentd-server-secure
    36.      - default
    37.      labels:
    38.        clusterId: "C1234"
    1The name of the ClusterLogForwarder CR must be instance.
    2The namespace for the ClusterLogForwarder CR must be openshift-logging.
    3The name of the output.
    4The output type: elasticsearch, fluentdForward, syslog, or kafka.
    5The URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.
    6If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent.
    7The configuration for an input to filter application logs from the specified projects.
    8If no namespace is specified, logs are collected from all namespaces.
    9The pipeline configuration directs logs from a named input to a named output. In this example, a pipeline named forward-to-fluentd-insecure forwards logs from an input named my-app-logs to an output named fluentd-server-insecure.
    10A list of inputs.
    11The name of the output to use.
    12Optional: String. One or more labels to add to the logs.
    13Configuration for a pipeline to send logs to other log aggregators.

    • Optional: Specify a name for the pipeline.

    • Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

    • Specify the name of the output to use when forwarding logs with this pipeline.

    • Optional: Specify the default output to forward logs to the default log store.

    • Optional: String. One or more labels to add to the logs.

    14Note that application logs from all namespaces are collected when using this configuration.

  2. Apply the ClusterLogForwarder CR by running the following command:

    1. $ oc apply -f <filename>.yaml

Forwarding application logs from specific pods

As a cluster administrator, you can use Kubernetes pod labels to gather log data from specific pods and forward it to a log collector.

Suppose that you have an application composed of pods running alongside other pods in various namespaces. If those pods have labels that identify the application, you can gather and output their log data to a specific log collector.

To specify the pod labels, you use one or more matchLabels key-value pairs. If you specify multiple key-value pairs, the pods must match all of them to be selected.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the pod labels using simple equality-based selectors under inputs[].name.application.selector.matchLabels, as shown in the following example.

    Example ClusterLogForwarder CR YAML file

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   pipelines:
    8.     - inputRefs: [ myAppLogData ] (3)
    9.       outputRefs: [ default ] (4)
    10.   inputs: (5)
    11.     - name: myAppLogData
    12.       application:
    13.         selector:
    14.           matchLabels: (6)
    15.             environment: production
    16.             app: nginx
    17.         namespaces: (7)
    18.         - app1
    19.         - app2
    20.   outputs: (8)
    21.     - <output_name>
    22.     ...
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3Specify one or more comma-separated values from inputs[].name.
    4Specify one or more comma-separated values from outputs[].
    5Define a unique inputs[].name for each application that has a unique set of pod labels.
    6Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.
    7Optional: Specify one or more namespaces.
    8Specify one or more outputs to forward your log data to.

  2. Optional: To restrict the gathering of log data to specific namespaces, use inputs[].name.application.namespaces, as shown in the preceding example.

  3. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.

    1. For each unique combination of pod labels, create an additional inputs[].name section similar to the one shown.

    2. Update the selectors to match the pod labels of this application.

    3. Add the new inputs[].name value to inputRefs. For example:

      1. - inputRefs: [ myAppLogData, myOtherAppLogData ]

  4. Create the CR object:

    1. $ oc create -f <file-name>.yaml

Additional resources

Additional resources

Forwarding logs to an external Loki logging system

You can forward logs to an external Loki logging system in addition to, or instead of, the default log store.

To configure log forwarding to Loki, you must create a ClusterLogForwarder custom resource (CR) with an output to Loki, and a pipeline that uses the output. The output to Loki can use the HTTP (insecure) or HTTPS (secure HTTP) connection.

Prerequisites

  • You must have a Loki logging system running at the URL you specify with the url field in the CR.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.   - name: loki-insecure (4)
    10.     type: "loki" (5)
    11.     url: http://loki.insecure.com:3100 (6)
    12.     loki:
    13.       tenantKey: kubernetes.namespace_name
    14.       labelKeys:
    15.       - kubernetes.labels.foo
    16.   - name: loki-secure (7)
    17.     type: "loki"
    18.     url: https://loki.secure.com:3100
    19.     secret:
    20.       name: loki-secret (8)
    21.     loki:
    22.       tenantKey: kubernetes.namespace_name (9)
    23.       labelKeys:
    24.       - kubernetes.labels.foo (10)
    25.   pipelines:
    26.   - name: application-logs (11)
    27.     inputRefs:  (12)
    28.     - application
    29.     - audit
    30.     outputRefs: (13)
    31.     - loki-secure
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Specify a name for the output.
    5Specify the type as “loki”.
    6Specify the URL and port of the Loki system as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki’s default port for HTTP(S) communication is 3100.
    7For a secure connection, you can specify an https or http URL that you authenticate by specifying a secret.
    8For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificates it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following “Example: Setting a secret that contains a username and password.”
    9Optional: Specify a metadata key field to generate values for the TenantID field in Loki. For example, setting tenantKey: kubernetes.namespacename uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the “Log Record Fields” link in the following “Additional resources” section.
    10Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression [a-zA-Z:][a-zA-Z0-9:]*. Illegal characters in metadata keys are replaced with to form the label name. For example, the kubernetes.labels.foo metadata key becomes Loki label kubernetes_labels_foo. If you do not set labelKeys, the default value is: [log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]. Keep the set of labels small because Loki limits the size and number of labels allowed. See Configuring Loki, limits_config. You can still query based on any log record field using query filters.
    11Optional: Specify a name for the pipeline.
    12Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    13Specify the name of the output to use when forwarding logs with this pipeline.

    Because Loki requires log streams to be correctly ordered by timestamp, labelKeys always includes the kubernetes_host label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.

  2. Apply the ClusterLogForwarder CR object by running the following command:

    1. $ oc apply -f <filename>.yaml

Additional resources

Forwarding logs to an external Elasticsearch instance

You can forward logs to an external Elasticsearch instance in addition to, or instead of, the internal log store. You are responsible for configuring the external log aggregator to receive log data from OKD.

To configure log forwarding to an external Elasticsearch instance, you must create a ClusterLogForwarder custom resource (CR) with an output to that instance, and a pipeline that uses the output. The external Elasticsearch output can use the HTTP (insecure) or HTTPS (secure HTTP) connection.

To forward logs to both an external and the internal Elasticsearch instance, create outputs and pipelines to the external instance and a pipeline that uses the default output to forward logs to the internal instance.

If you only want to forward logs to an internal Elasticsearch instance, you do not need to create a ClusterLogForwarder CR.

Prerequisites

  • You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR:

    Example ClusterLogForwarder CR

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.    - name: elasticsearch-example (4)
    10.      type: elasticsearch (5)
    11.      elasticsearch:
    12.        version: 8 (6)
    13.      url: http://elasticsearch.example.com:9200 (7)
    14.      secret:
    15.        name: es-secret (8)
    16.   pipelines:
    17.    - name: application-logs (9)
    18.      inputRefs: (10)
    19.      - application
    20.      - audit
    21.      outputRefs:
    22.      - elasticsearch-example (11)
    23.      - default (12)
    24.      labels:
    25.        myLabel: "myValue" (13)
    26. # ...
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Specify a name for the output.
    5Specify the elasticsearch type.
    6Specify the Elasticsearch version. This can be 6, 7, or 8.
    7Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.
    8For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following “Example: Setting a secret that contains a username and password.”
    9Optional: Specify a name for the pipeline.
    10Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    11Specify the name of the output to use when forwarding logs with this pipeline.
    12Optional: Specify the default output to send the logs to the internal Elasticsearch instance.
    13Optional: String. One or more labels to add to the logs.

  2. Apply the ClusterLogForwarder CR:

    1. $ oc apply -f <filename>.yaml

Example: Setting a secret that contains a username and password

You can use a secret that contains a username and password to authenticate a secure connection to an external Elasticsearch instance.

For example, if you cannot use mutual TLS (mTLS) keys because a third party operates the Elasticsearch instance, you can use HTTP or HTTPS and set a secret that contains the username and password.

  1. Create a Secret YAML file similar to the following example. Use base64-encoded values for the username and password fields. The secret type is opaque by default.

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: openshift-test-secret
    5. data:
    6.   username: <username>
    7.   password: <password>
    8. # ...

  2. Create the secret:

    1. $ oc create secret -n openshift-logging openshift-test-secret.yaml

  3. Specify the name of the secret in the ClusterLogForwarder CR:

    1. kind: ClusterLogForwarder
    2. metadata:
    3.   name: instance
    4.   namespace: openshift-logging
    5. spec:
    6.   outputs:
    7.    - name: elasticsearch
    8.      type: "elasticsearch"
    9.      url: https://elasticsearch.secure.com:9200
    10.      secret:
    11.         name: openshift-test-secret
    12. # ...

    In the value of the url field, the prefix can be http or https.

  4. Apply the CR object:

    1. $ oc apply -f <filename>.yaml

Forwarding logs using the Fluentd forward protocol

You can use the Fluentd forward protocol to send a copy of your logs to an external log aggregator that is configured to accept the protocol instead of, or in addition to, the default Elasticsearch log store. You are responsible for configuring the external log aggregator to receive the logs from OKD.

To configure log forwarding using the forward protocol, you must create a ClusterLogForwarder custom resource (CR) with one or more outputs to the Fluentd servers, and pipelines that use those outputs. The Fluentd output can use a TCP (insecure) or TLS (secure TCP) connection.

Prerequisites

  • You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: instance (1)
    5.   namespace: openshift-logging (2)
    6. spec:
    7.   outputs:
    8.    - name: fluentd-server-secure (3)
    9.      type: fluentdForward (4)
    10.      url: 'tls://fluentdserver.security.example.com:24224' (5)
    11.      secret: (6)
    12.         name: fluentd-secret
    13.    - name: fluentd-server-insecure
    14.      type: fluentdForward
    15.      url: 'tcp://fluentdserver.home.example.com:24224'
    16.   pipelines:
    17.    - name: forward-to-fluentd-secure (7)
    18.      inputRefs:  (8)
    19.      - application
    20.      - audit
    21.      outputRefs:
    22.      - fluentd-server-secure (9)
    23.      - default (10)
    24.      labels:
    25.        clusterId: "C1234" (11)
    26.    - name: forward-to-fluentd-insecure (12)
    27.      inputRefs:
    28.      - infrastructure
    29.      outputRefs:
    30.      - fluentd-server-insecure
    31.      labels:
    32.        clusterId: "C1234"
    1The name of the ClusterLogForwarder CR must be instance.
    2The namespace for the ClusterLogForwarder CR must be openshift-logging.
    3Specify a name for the output.
    4Specify the fluentdForward type.
    5Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.
    6If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.
    7Optional: Specify a name for the pipeline.
    8Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    9Specify the name of the output to use when forwarding logs with this pipeline.
    10Optional: Specify the default output to forward logs to the internal Elasticsearch instance.
    11Optional: String. One or more labels to add to the logs.
    12Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

    • A name to describe the pipeline.

    • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.

    • The outputRefs is the name of the output to use.

    • Optional: String. One or more labels to add to the logs.

  2. Create the CR object:

    1. $ oc create -f <file-name>.yaml

Enabling nanosecond precision for Logstash to ingest data from fluentd

For Logstash to ingest log data from fluentd, you must enable nanosecond precision in the Logstash configuration file.

Procedure

  • In the Logstash configuration file, set nanosecond_precision to true.

Example Logstash configuration file

  1. input { tcp { codec => fluent { nanosecond_precision => true } port => 24114 } }
  2. filter { }
  3. output { stdout { codec => rubydebug } }

Forwarding logs using the syslog protocol

You can use the syslog RFC3164 or RFC5424 protocol to send a copy of your logs to an external log aggregator that is configured to accept the protocol instead of, or in addition to, the default Elasticsearch log store. You are responsible for configuring the external log aggregator, such as a syslog server, to receive the logs from OKD.

To configure log forwarding using the syslog protocol, you must create a ClusterLogForwarder custom resource (CR) with one or more outputs to the syslog servers, and pipelines that use those outputs. The syslog output can use a UDP, TCP, or TLS connection.

Prerequisites

  • You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.    - name: rsyslog-east (4)
    10.      type: syslog (5)
    11.      syslog: (6)
    12.        facility: local0
    13.        rfc: RFC3164
    14.        payloadKey: message
    15.        severity: informational
    16.      url: 'tls://rsyslogserver.east.example.com:514' (7)
    17.      secret: (8)
    18.         name: syslog-secret
    19.    - name: rsyslog-west
    20.      type: syslog
    21.      syslog:
    22.       appName: myapp
    23.       facility: user
    24.       msgID: mymsg
    25.       procID: myproc
    26.       rfc: RFC5424
    27.       severity: debug
    28.      url: 'tcp://rsyslogserver.west.example.com:514'
    29.   pipelines:
    30.    - name: syslog-east (9)
    31.      inputRefs: (10)
    32.      - audit
    33.      - application
    34.      outputRefs: (11)
    35.      - rsyslog-east
    36.      - default (12)
    37.      labels:
    38.        secure: "true" (13)
    39.        syslog: "east"
    40.    - name: syslog-west (14)
    41.      inputRefs:
    42.      - infrastructure
    43.      outputRefs:
    44.      - rsyslog-west
    45.      - default
    46.      labels:
    47.        syslog: "west"
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Specify a name for the output.
    5Specify the syslog type.
    6Optional: Specify the syslog parameters, listed below.
    7Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.
    8If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.
    9Optional: Specify a name for the pipeline.
    10Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    11Specify the name of the output to use when forwarding logs with this pipeline.
    12Optional: Specify the default output to forward logs to the internal Elasticsearch instance.
    13Optional: String. One or more labels to add to the logs. Quote values like “true” so they are recognized as string values, not as a boolean.
    14Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

    • A name to describe the pipeline.

    • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.

    • The outputRefs is the name of the output to use.

    • Optional: String. One or more labels to add to the logs.

  2. Create the CR object:

    1. $ oc create -f <filename>.yaml

Adding log source information to message output

You can add namespace_name, pod_name, and container_name elements to the message field of the record by adding the AddLogSource field to your ClusterLogForwarder custom resource (CR).

  1.   spec:
  2.     outputs:
  3.     - name: syslogout
  4.       syslog:
  5.         addLogSource: true
  6.         facility: user
  7.         payloadKey: message
  8.         rfc: RFC3164
  9.         severity: debug
  10.         tag: mytag
  11.       type: syslog
  12.       url: tls://syslog-receiver.openshift-logging.svc:24224
  13.     pipelines:
  14.     - inputRefs:
  15.       - application
  16.       name: test-app
  17.       outputRefs:
  18.       - syslogout

This configuration is compatible with both RFC3164 and RFC5424.

Example syslog message output without AddLogSource

  1. <15>1 2020-11-15T17:06:14+00:00 fluentd-9hkb4 mytag - - -  {"msgcontent"=>"Message Contents", "timestamp"=>"2020-11-15 17:06:09", "tag_key"=>"rec_tag", "index"=>56}

Example syslog message output with AddLogSource

  1. <15>1 2020-11-16T10:49:37+00:00 crc-j55b9-master-0 mytag - - -  namespace_name=clo-test-6327,pod_name=log-generator-ff9746c49-qxm7l,container_name=log-generator,message={"msgcontent":"My life is my message", "timestamp":"2020-11-16 10:49:36", "tag_key":"rec_tag", "index":76}

Syslog parameters

You can configure the following for the syslog outputs. For more information, see the syslog RFC3164 or RFC5424 RFC.

  • facility: The syslog facility. The value can be a decimal integer or a case-insensitive keyword:

    • 0 or kern for kernel messages

    • 1 or user for user-level messages, the default.

    • 2 or mail for the mail system

    • 3 or daemon for system daemons

    • 4 or auth for security/authentication messages

    • 5 or syslog for messages generated internally by syslogd

    • 6 or lpr for the line printer subsystem

    • 7 or news for the network news subsystem

    • 8 or uucp for the UUCP subsystem

    • 9 or cron for the clock daemon

    • 10 or authpriv for security authentication messages

    • 11 or ftp for the FTP daemon

    • 12 or ntp for the NTP subsystem

    • 13 or security for the syslog audit log

    • 14 or console for the syslog alert log

    • 15 or solaris-cron for the scheduling daemon

    • 16–23 or local0 – local7 for locally used facilities

  • Optional: payloadKey: The record field to use as payload for the syslog message.

    Configuring the payloadKey parameter prevents other parameters from being forwarded to the syslog.

  • rfc: The RFC to be used for sending logs using syslog. The default is RFC5424.

  • severity: The syslog severity to set on outgoing syslog records. The value can be a decimal integer or a case-insensitive keyword:

    • 0 or Emergency for messages indicating the system is unusable

    • 1 or Alert for messages indicating action must be taken immediately

    • 2 or Critical for messages indicating critical conditions

    • 3 or Error for messages indicating error conditions

    • 4 or Warning for messages indicating warning conditions

    • 5 or Notice for messages indicating normal but significant conditions

    • 6 or Informational for messages indicating informational messages

    • 7 or Debug for messages indicating debug-level messages, the default

  • tag: Tag specifies a record field to use as a tag on the syslog message.

  • trimPrefix: Remove the specified prefix from the tag.

Additional RFC5424 syslog parameters

The following parameters apply to RFC5424:

  • appName: The APP-NAME is a free-text string that identifies the application that sent the log. Must be specified for RFC5424.

  • msgID: The MSGID is a free-text string that identifies the type of message. Must be specified for RFC5424.

  • procID: The PROCID is a free-text string. A change in the value indicates a discontinuity in syslog reporting. Must be specified for RFC5424.

Forwarding logs to a Kafka broker

You can forward logs to an external Kafka broker in addition to, or instead of, the default log store.

To configure log forwarding to an external Kafka instance, you must create a ClusterLogForwarder custom resource (CR) with an output to that instance, and a pipeline that uses the output. You can include a specific Kafka topic in the output or use the default. The Kafka output can use a TCP (insecure) or TLS (secure TCP) connection.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.    - name: app-logs (4)
    10.      type: kafka (5)
    11.      url: tls://kafka.example.devlab.com:9093/app-topic (6)
    12.      secret:
    13.        name: kafka-secret (7)
    14.    - name: infra-logs
    15.      type: kafka
    16.      url: tcp://kafka.devlab2.example.com:9093/infra-topic (8)
    17.    - name: audit-logs
    18.      type: kafka
    19.      url: tls://kafka.qelab.example.com:9093/audit-topic
    20.      secret:
    21.         name: kafka-secret-qe
    22.   pipelines:
    23.    - name: app-topic (9)
    24.      inputRefs: (10)
    25.      - application
    26.      outputRefs: (11)
    27.      - app-logs
    28.      labels:
    29.        logType: "application" (12)
    30.    - name: infra-topic (13)
    31.      inputRefs:
    32.      - infrastructure
    33.      outputRefs:
    34.      - infra-logs
    35.      labels:
    36.        logType: "infra"
    37.    - name: audit-topic
    38.      inputRefs:
    39.      - audit
    40.      outputRefs:
    41.      - audit-logs
    42.      labels:
    43.        logType: "audit"
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Specify a name for the output.
    5Specify the kafka type.
    6Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.
    7If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.
    8Optional: To send an insecure output, use a tcp prefix in front of the URL. Also omit the secret key and its name from this output.
    9Optional: Specify a name for the pipeline.
    10Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    11Specify the name of the output to use when forwarding logs with this pipeline.
    12Optional: String. One or more labels to add to the logs.
    13Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

    • A name to describe the pipeline.

    • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.

    • The outputRefs is the name of the output to use.

    • Optional: String. One or more labels to add to the logs.

  2. Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in the following example:

    1. # ...
    2. spec:
    3.   outputs:
    4.   - name: app-logs
    5.     type: kafka
    6.     secret:
    7.       name: kafka-secret-dev
    8.     kafka:  (1)
    9.       brokers: (2)
    10.         - tls://kafka-broker1.example.com:9093/
    11.         - tls://kafka-broker2.example.com:9093/
    12.       topic: app-topic (3)
    13. # ...
    1Specify a kafka key that has a brokers and topic key.
    2Use the brokers key to specify an array of one or more brokers.
    3Use the topic key to specify the target topic that receives the logs.

  3. Apply the ClusterLogForwarder CR by running the following command:

    1. $ oc apply -f <filename>.yaml

Forwarding logs to Amazon CloudWatch

You can forward logs to Amazon CloudWatch, a monitoring and log storage service hosted by Amazon Web Services (AWS). You can forward logs to CloudWatch in addition to, or instead of, the default log store.

To configure log forwarding to CloudWatch, you must create a ClusterLogForwarder custom resource (CR) with an output for CloudWatch, and a pipeline that uses the output.

Procedure

  1. Create a Secret YAML file that uses the aws_access_key_id and aws_secret_access_key fields to specify your base64-encoded AWS credentials. For example:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: cw-secret
    5.   namespace: openshift-logging
    6. data:
    7.   aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK
    8.   aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=

  2. Create the secret. For example:

    1. $ oc apply -f cw-secret.yaml

  3. Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the name of the secret. For example:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: <service_account_name> (3)
    8.   outputs:
    9.    - name: cw (4)
    10.      type: cloudwatch (5)
    11.      cloudwatch:
    12.        groupBy: logType (6)
    13.        groupPrefix: <group prefix> (7)
    14.        region: us-east-2 (8)
    15.      secret:
    16.         name: cw-secret (9)
    17.   pipelines:
    18.     - name: infra-logs (10)
    19.       inputRefs: (11)
    20.         - infrastructure
    21.         - audit
    22.         - application
    23.       outputRefs:
    24.         - cw (12)
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Specify a name for the output.
    5Specify the cloudwatch type.
    6Optional: Specify how to group the logs:

    • logType creates log groups for each log type.

    • namespaceName creates a log group for each application name space. It also creates separate log groups for infrastructure and audit logs.

    • namespaceUUID creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

    7Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.
    8Specify the AWS region.
    9Specify the name of the secret that contains your AWS credentials.
    10Optional: Specify a name for the pipeline.
    11Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    12Specify the name of the output to use when forwarding logs with this pipeline.

  4. Create the CR object:

    1. $ oc create -f <file-name>.yaml

Example: Using ClusterLogForwarder with Amazon CloudWatch

Here, you see an example ClusterLogForwarder custom resource (CR) and the log data that it outputs to Amazon CloudWatch.

Suppose that you are running an OKD cluster named mycluster. The following command returns the cluster’s infrastructureName, which you will use to compose aws commands later on:

  1. $ oc get Infrastructure/cluster -ojson | jq .status.infrastructureName
  2. "mycluster-7977k"

To generate log data for this example, you run a busybox pod in a namespace called app. The busybox pod writes a message to stdout every three seconds:

  1. $ oc run busybox --image=busybox -- sh -c 'while true; do echo "My life is my message"; sleep 3; done'
  2. $ oc logs -f busybox
  3. My life is my message
  4. My life is my message
  5. My life is my message
  6. ...

You can look up the UUID of the app namespace where the busybox pod runs:

  1. $ oc get ns/app -ojson | jq .metadata.uid
  2. "794e1e1a-b9f5-4958-a190-e76a9b53d7bf"

In your ClusterLogForwarder custom resource (CR), you configure the infrastructure, audit, and application log types as inputs to the all-logs pipeline. You also connect this pipeline to cw output, which forwards the logs to a CloudWatch instance in the us-east-2 region:

  1. apiVersion: "logging.openshift.io/v1"
  2. kind: ClusterLogForwarder
  3. metadata:
  4.   name: instance
  5.   namespace: openshift-logging
  6. spec:
  7.   outputs:
  8.    - name: cw
  9.      type: cloudwatch
  10.      cloudwatch:
  11.        groupBy: logType
  12.        region: us-east-2
  13.      secret:
  14.         name: cw-secret
  15.   pipelines:
  16.     - name: all-logs
  17.       inputRefs:
  18.         - infrastructure
  19.         - audit
  20.         - application
  21.       outputRefs:
  22.         - cw

Each region in CloudWatch contains three levels of objects:

  • log group

    • log stream

      • log event

With groupBy: logType in the ClusterLogForwarding CR, the three log types in the inputRefs produce three log groups in Amazon Cloudwatch:

  1. $ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName
  2. "mycluster-7977k.application"
  3. "mycluster-7977k.audit"
  4. "mycluster-7977k.infrastructure"

Each of the log groups contains log streams:

  1. $ aws --output json logs describe-log-streams --log-group-name mycluster-7977k.application | jq .logStreams[].logStreamName
  2. "kubernetes.var.log.containers.busybox_app_busybox-da085893053e20beddd6747acdbaf98e77c37718f85a7f6a4facf09ca195ad76.log"
  1. $ aws --output json logs describe-log-streams --log-group-name mycluster-7977k.audit | jq .logStreams[].logStreamName
  2. "ip-10-0-131-228.us-east-2.compute.internal.k8s-audit.log"
  3. "ip-10-0-131-228.us-east-2.compute.internal.linux-audit.log"
  4. "ip-10-0-131-228.us-east-2.compute.internal.openshift-audit.log"
  5. ...
  1. $ aws --output json logs describe-log-streams --log-group-name mycluster-7977k.infrastructure | jq .logStreams[].logStreamName
  2. "ip-10-0-131-228.us-east-2.compute.internal.kubernetes.var.log.containers.apiserver-69f9fd9b58-zqzw5_openshift-oauth-apiserver_oauth-apiserver-453c5c4ee026fe20a6139ba6b1cdd1bed25989c905bf5ac5ca211b7cbb5c3d7b.log"
  3. "ip-10-0-131-228.us-east-2.compute.internal.kubernetes.var.log.containers.apiserver-797774f7c5-lftrx_openshift-apiserver_openshift-apiserver-ce51532df7d4e4d5f21c4f4be05f6575b93196336be0027067fd7d93d70f66a4.log"
  4. "ip-10-0-131-228.us-east-2.compute.internal.kubernetes.var.log.containers.apiserver-797774f7c5-lftrx_openshift-apiserver_openshift-apiserver-check-endpoints-82a9096b5931b5c3b1d6dc4b66113252da4a6472c9fff48623baee761911a9ef.log"
  5. ...

Each log stream contains log events. To see a log event from the busybox Pod, you specify its log stream from the application log group:

  1. $ aws logs get-log-events --log-group-name mycluster-7977k.application --log-stream-name kubernetes.var.log.containers.busybox_app_busybox-da085893053e20beddd6747acdbaf98e77c37718f85a7f6a4facf09ca195ad76.log
  2. {
  3.     "events": [
  4.         {
  5.             "timestamp": 1629422704178,
  6.             "message": "{\"docker\":{\"container_id\":\"da085893053e20beddd6747acdbaf98e77c37718f85a7f6a4facf09ca195ad76\"},\"kubernetes\":{\"container_name\":\"busybox\",\"namespace_name\":\"app\",\"pod_name\":\"busybox\",\"container_image\":\"docker.io/library/busybox:latest\",\"container_image_id\":\"docker.io/library/busybox@sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60\",\"pod_id\":\"870be234-90a3-4258-b73f-4f4d6e2777c7\",\"host\":\"ip-10-0-216-3.us-east-2.compute.internal\",\"labels\":{\"run\":\"busybox\"},\"master_url\":\"https://kubernetes.default.svc\",\"namespace_id\":\"794e1e1a-b9f5-4958-a190-e76a9b53d7bf\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"app\"}},\"message\":\"My life is my message\",\"level\":\"unknown\",\"hostname\":\"ip-10-0-216-3.us-east-2.compute.internal\",\"pipeline_metadata\":{\"collector\":{\"ipaddr4\":\"10.0.216.3\",\"inputname\":\"fluent-plugin-systemd\",\"name\":\"fluentd\",\"received_at\":\"2021-08-20T01:25:08.085760+00:00\",\"version\":\"1.7.4 1.6.0\"}},\"@timestamp\":\"2021-08-20T01:25:04.178986+00:00\",\"viaq_index_name\":\"app-write\",\"viaq_msg_id\":\"NWRjZmUyMWQtZjgzNC00MjI4LTk3MjMtNTk3NmY3ZjU4NDk1\",\"log_type\":\"application\",\"time\":\"2021-08-20T01:25:04+00:00\"}",
  7.             "ingestionTime": 1629422744016
  8.         },
  9. ...

Example: Customizing the prefix in log group names

In the log group names, you can replace the default infrastructureName prefix, mycluster-7977k, with an arbitrary string like demo-group-prefix. To make this change, you update the groupPrefix field in the ClusterLogForwarding CR:

  1. cloudwatch:
  2.     groupBy: logType
  3.     groupPrefix: demo-group-prefix
  4.     region: us-east-2

The value of groupPrefix replaces the default infrastructureName prefix:

  1. $ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName
  2. "demo-group-prefix.application"
  3. "demo-group-prefix.audit"
  4. "demo-group-prefix.infrastructure"

Example: Naming log groups after application namespace names

For each application namespace in your cluster, you can create a log group in CloudWatch whose name is based on the name of the application namespace.

If you delete an application namespace object and create a new one that has the same name, CloudWatch continues using the same log group as before.

If you consider successive application namespace objects that have the same name as equivalent to each other, use the approach described in this example. Otherwise, if you need to distinguish the resulting log groups from each other, see the following “Naming log groups for application namespace UUIDs” section instead.

To create application log groups whose names are based on the names of the application namespaces, you set the value of the groupBy field to namespaceName in the ClusterLogForwarder CR:

  1. cloudwatch:
  2.     groupBy: namespaceName
  3.     region: us-east-2

Setting groupBy to namespaceName affects the application log group only. It does not affect the audit and infrastructure log groups.

In Amazon Cloudwatch, the namespace name appears at the end of each log group name. Because there is a single application namespace, “app”, the following output shows a new mycluster-7977k.app log group instead of mycluster-7977k.application:

  1. $ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName
  2. "mycluster-7977k.app"
  3. "mycluster-7977k.audit"
  4. "mycluster-7977k.infrastructure"

If the cluster in this example had contained multiple application namespaces, the output would show multiple log groups, one for each namespace.

The groupBy field affects the application log group only. It does not affect the audit and infrastructure log groups.

Example: Naming log groups after application namespace UUIDs

For each application namespace in your cluster, you can create a log group in CloudWatch whose name is based on the UUID of the application namespace.

If you delete an application namespace object and create a new one, CloudWatch creates a new log group.

If you consider successive application namespace objects with the same name as different from each other, use the approach described in this example. Otherwise, see the preceding “Example: Naming log groups for application namespace names” section instead.

To name log groups after application namespace UUIDs, you set the value of the groupBy field to namespaceUUID in the ClusterLogForwarder CR:

  1. cloudwatch:
  2.     groupBy: namespaceUUID
  3.     region: us-east-2

In Amazon Cloudwatch, the namespace UUID appears at the end of each log group name. Because there is a single application namespace, “app”, the following output shows a new mycluster-7977k.794e1e1a-b9f5-4958-a190-e76a9b53d7bf log group instead of mycluster-7977k.application:

  1. $ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName
  2. "mycluster-7977k.794e1e1a-b9f5-4958-a190-e76a9b53d7bf" // uid of the "app" namespace
  3. "mycluster-7977k.audit"
  4. "mycluster-7977k.infrastructure"

The groupBy field affects the application log group only. It does not affect the audit and infrastructure log groups.

Creating a secret for AWS CloudWatch with an existing AWS role

If you have an existing role for AWS, you can create a secret for AWS with STS using the oc create secret --from-literal command.

Procedure

  • In the CLI, enter the following to generate a secret for AWS:

    1. $ oc create secret generic cw-sts-secret -n openshift-logging --from-literal=role_arn=arn:aws:iam::123456789012:role/my-role_with-permissions

    Example Secret

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   namespace: openshift-logging
    5.   name: my-secret-name
    6. stringData:
    7.   role_arn: arn:aws:iam::123456789012:role/my-role_with-permissions

Forwarding logs to Amazon CloudWatch from STS enabled clusters

For clusters with AWS Security Token Service (STS) enabled, you can create an AWS service account manually or create a credentials request by using the Cloud Credential Operator(CCO) utility ccoctl.

Prerequisites

  • Logging for Red Hat OpenShift: 5.5 and later

Procedure

  1. Create a CredentialsRequest custom resource YAML by using the template below:

    CloudWatch credentials request template

    1. apiVersion: cloudcredential.openshift.io/v1
    2. kind: CredentialsRequest
    3. metadata:
    4.   name: <your_role_name>-credrequest
    5.   namespace: openshift-cloud-credential-operator
    6. spec:
    7.   providerSpec:
    8.     apiVersion: cloudcredential.openshift.io/v1
    9.     kind: AWSProviderSpec
    10.     statementEntries:
    11.       - action:
    12.           - logs:PutLogEvents
    13.           - logs:CreateLogGroup
    14.           - logs:PutRetentionPolicy
    15.           - logs:CreateLogStream
    16.           - logs:DescribeLogGroups
    17.           - logs:DescribeLogStreams
    18.         effect: Allow
    19.         resource: arn:aws:logs:*:*:*
    20.   secretRef:
    21.     name: <your_role_name>
    22.     namespace: openshift-logging
    23.   serviceAccountNames:
    24.     - logcollector

  2. Use the ccoctl command to create a role for AWS using your CredentialsRequest CR. With the CredentialsRequest object, this ccoctl command creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy that grants permissions to perform operations on CloudWatch resources. This command also creates a YAML configuration file in /<path_to_ccoctl_output_dir>/manifests/openshift-logging-<your_role_name>-credentials.yaml. This secret file contains the role_arn key/value used during authentication with the AWS IAM identity provider.

    1. $ ccoctl aws create-iam-roles \
    2. --name=<name> \
    3. --region=<aws_region> \
    4. --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests \
    5. --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com (1)
    1<name> is the name used to tag your cloud resources and should match the name used during your STS cluster install

  3. Apply the secret created:

    1. $ oc apply -f output/manifests/openshift-logging-<your_role_name>-credentials.yaml

  4. Create or edit a ClusterLogForwarder custom resource:

    1. apiVersion: logging.openshift.io/v1
    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: <log_forwarder_name> (1)
    5.   namespace: <log_forwarder_namespace> (2)
    6. spec:
    7.   serviceAccountName: clf-collector (3)
    8.   outputs:
    9.    - name: cw (4)
    10.      type: cloudwatch (5)
    11.      cloudwatch:
    12.        groupBy: logType (6)
    13.        groupPrefix: <group prefix> (7)
    14.        region: us-east-2 (8)
    15.      secret:
    16.         name: <your_secret_name> (9)
    17.   pipelines:
    18.     - name: to-cloudwatch (10)
    19.       inputRefs: (11)
    20.         - infrastructure
    21.         - audit
    22.         - application
    23.       outputRefs:
    24.         - cw (12)
    1In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
    2In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
    3Specify the clf-collector service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
    4Specify a name for the output.
    5Specify the cloudwatch type.
    6Optional: Specify how to group the logs:

    • logType creates log groups for each log type.

    • namespaceName creates a log group for each application name space. Infrastructure and audit logs are unaffected, remaining grouped by logType.

    • namespaceUUID creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

    7Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.
    8Specify the AWS region.
    9Specify the name of the secret that contains your AWS credentials.
    10Optional: Specify a name for the pipeline.
    11Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
    12Specify the name of the output to use when forwarding logs with this pipeline.

Additional resources