Aggregating Container Logs

Overview

As an OKD cluster administrator, you can deploy the EFK stack to aggregate logs for a range of OKD services. Application developers can view the logs of the projects for which they have view access. The EFK stack aggregates logs from hosts and applications, whether coming from multiple containers or even deleted pods.

The EFK stack is a modified version of the ELK stack and is comprised of:

• Elasticsearch (ES): An object store where all logs are stored.

• Fluentd: Gathers logs from nodes and feeds them to Elasticsearch.

• Kibana: A web UI for Elasticsearch.

• Curator: Removes old logs from Elasticsearch.

After deployment in a cluster, the stack aggregates logs from all nodes and projects into Elasticsearch, and provides a Kibana UI to view any logs. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. The stack components communicate securely.

Pre-deployment Configuration

1. An Ansible playbook is available to deploy and upgrade aggregated logging. You should familiarize yourself with the Installing Clusters guide. This provides information for preparing to use Ansible and includes information about configuration. Parameters are added to the Ansible inventory file to configure various areas of the EFK stack.

2. Review the sizing guidelines to determine how best to configure your deployment.

3. Ensure that you have deployed a router for the cluster.

4. Ensure that you have the necessary storage for Elasticsearch. Note that each Elasticsearch node requires its own storage volume. See Elasticsearch for more information.

5. Determine if you need highly-available Elasticsearch. A highly-available environment requires at least three Elasticsearch nodes, each on a different host. By default, OKD creates one shard for each index and zero replicas of those shards. High availability also requires multiple replicas of each shard. To create high availability, set the openshift_logging_es_number_of_replicas Ansible variable to a value higher than 1. See Elasticsearch for more information.

Specifying Logging Ansible Variables

You can override the default parameter values by specifying parameters for the EFK deployment in the inventory host file.

Read the Elasticsearch and the Fluentd sections before choosing parameters:

#oc auth can-i view pod/logs -n default
yes

Custom Certificates

You can specify custom certificates using the following inventory variables instead of relying on those generated during the deployment process. These certificates are used to encrypt and secure communication between a user’s browser and Kibana. The security-related files will be generated if they are not supplied.

If you need to redeploy these certificates, see Redeploy EFK Certificates.

Deploying the EFK Stack

The EFK stack is deployed using an Ansible playbook to the EFK components. Run the playbook from the default OpenShift Ansible location using the default inventory file.

$ ansible-playbook playbooks/openshift-logging/config.yml

Running the playbook deploys all resources needed to support the stack; such as Secrets, ServiceAccounts, and DeploymentConfigs, deployed to the project openshift-logging. The playbook waits to deploy the component pods until the stack is running. If the wait steps fail, the deployment could still be successful; it may be retrieving the component images from the registry which can take up to a few minutes. You can watch the process with:

$ oc get pods -w
logging-curator-1541129400-l5h77           0/1       Running   0          11h  (1)
logging-es-data-master-ecu30lr4-1-deploy   0/1       Running   0          11h  (2)
logging-fluentd-2lgwn                      1/1       Running   0          11h  (3)
logging-fluentd-lmvms                      1/1       Running   0          11h
logging-fluentd-p9nd7                      1/1       Running   0          11h
logging-kibana-1-zk94k                     2/2       Running   0          11h  (4)

You can use the `oc get pods -o wide command to see the nodes where the Fluentd pod are deployed:

$ oc get pods -o wide
NAME                                       READY     STATUS    RESTARTS   AGE       IP             NODE                         NOMINATED NODE
logging-es-data-master-5av030lk-1-2x494    2/2       Running   0          38m       154.128.0.80   ip-153-12-8-6.wef.internal   <none>
logging-fluentd-lqdxg                      1/1       Running   0          2m        154.128.0.85   ip-153-12-8-6.wef.internal   <none>
logging-kibana-1-gj5kc                     2/2       Running   0          39m       154.128.0.77   ip-153-12-8-6.wef.internal   <none>

They will eventually enter Running status. For additional details about the status of the pods during deployment by retrieving associated events:

$ oc describe pods/<pod_name>

Check the logs if the pods do not run successfully:

$ oc logs -f <pod_name>

Understanding and Adjusting the Deployment

This section describes adjustments that you can make to deployed components.

Ops Cluster

If you set openshift_logging_use_ops to true in your inventory file, Fluentd is configured to split logs between the main Elasticsearch cluster and another cluster reserved for operations logs, which are defined as node system logs and the projects default, openshift, and openshift-infra. Therefore, a separate Elasticsearch cluster, a separate Kibana, and a separate Curator are deployed to index, access, and manage operations logs. These deployments are set apart with names that include -ops. Keep these separate deployments in mind if you enable this option. Most of the following discussion also applies to the operations cluster if present, just with the names changed to include -ops.

Elasticsearch

Elasticsearch (ES) is an object store where all logs are stored.

Elasticsearch organizes the log data into datastores, each called an index. Elasticsearch subdivides each index into multiple pieces called shards, which it spreads across a set of Elasticsearch nodes in your cluster. You can configure Elasticsearch to make copies of the shards, called replicas. Elasticsearch also spreads replicas across the Elactisearch nodes. The combination of shards and replicas is intended to provide redundancy and resilience to failure. For example, if you configure three shards for the index with one replica, Elasticsearch generates a total of six shards for that index: three primary shards and three replicas as a backup.

The OKD logging installer ensures each Elasticsearch node is deployed using a unique deployment configuration that includes its own storage volume. You can create an additional deployment configuration for each Elasticsearch node you add to the logging system. During installation, you can use the openshift_logging_es_cluster_size Ansible variable to specify the number of Elasticsearch nodes.

Alternatively, you can scale up your existing cluster by modifying the openshift_logging_es_cluster_size in the inventory file and re-running the logging playbook. Additional clustering parameters can be modified and are described in Specifying Logging Ansible Variables.

Refer to Elastic’s documentation for considerations involved in choosing storage and network location as directed below.

Viewing all Elasticsearch Deployments

To view all current Elasticsearch deployments:

$ oc get dc --selector logging-infra=elasticsearch

Configuring Elasticsearch for High Availability

A highly-available Elasticsearch environment requires at least three Elasticsearch nodes, each on a different host, and setting the openshift_logging_es_number_of_replicas Ansible variable to a value of 1 or higher to create replicas.

Use the following scenarios as a guide for an OKD cluster with three Elasticsearch nodes:

• If you can tolerate one Elasticsearch node going down, set openshift_logging_es_number_of_replicas to 1. This ensures that two nodes have a copy of all of the Elasticsearch data in the cluster.

• If you must tolerate two Elasticsearch nodes going down, set openshift_logging_es_number_of_replicas to 2. This ensures that every node has a copy of all of the Elasticsearch data in the cluster.

Note that there is a trade-off between high availability and performance. For example, having openshift_logging_es_number_of_replicas=2 and openshift_logging_es_number_of_shards=3 requires Elasticsearch to spend significant resources replicating the shard data among the nodes in the cluster. Also, using a higher number of replicas requires doubling or tripling the data storage requirements on each node, so you must take that into account when planning persistent storage for Elasticsearch.

Considerations when Configuring the Number of Shards

For the openshift_logging_es_number_of_shards parameter, consider:

• For higher performance, increase the number of shards. For example, in a three node cluster, set openshift_logging_es_number_of_shards=3. This will cause each index to be split into three parts (shards), and the load for processing the index will be spread out over all 3 nodes.

• If you have a large number of projects, you might see performance degradation if you have more than a few thousand shards in the cluster. Either reduce the number of shards or reduce the curation time.

• If you have a small number of very large indices, you might want to configure openshift_logging_es_number_of_shards=3 or higher. Elasticsearch recommends using a maximum shard size of less than 50 GB.

Node Selector

Because Elasticsearch can use a lot of resources, all members of a cluster should have low latency network connections to each other and to any remote storage. Ensure this by directing the instances to dedicated nodes, or a dedicated region within your cluster, using a node selector.

To configure a node selector, specify the openshift_logging_es_nodeselector configuration option in the inventory file. This applies to all Elasticsearch deployments; if you need to individualize the node selectors, you must manually edit each deployment configuration after deployment. The node selector is specified as a python compatible dict. For example, {"node-type":"infra", "region":"east"}.

Persistent Elasticsearch Storage

By default, the openshift_logging Ansible role creates an ephemeral deployment in which all data in a pod is lost upon pod restart.

For production environments, each Elasticsearch deployment configuration requires a persistent storage volume. You can specify an existing persistent volume claim or allow OKD to create one.

• Use existing PVCs. If you create your own PVCs for the deployment, OKD uses those PVCs.

  Name the PVCs to match the openshift_logging_es_pvc_prefix setting, which defaults to logging-es. Assign each PVC a name with a sequence number added to it: logging-es-0, logging-es-1, logging-es-2, and so on.

• Allow OKD to create a PVC. If a PVC for Elsaticsearch does not exist, OKD creates the PVC based on parameters in the Ansible inventory file.

  Parameter	Description
  openshift_logging_es_pvc_size	Specify the size of the PVC request.
  openshift_logging_elasticsearch_storage_type	Specify the storage type as pvc. This is an optional parameter. If you set the openshift_logging_es_pvc_size parameter to a value greater than 0, the logging installer automatically sets this parameter to pvc by default.
  openshift_logging_es_pvc_prefix	Optionally, specify a custom prefix for the PVC.

  For example:

  openshift_logging_elasticsearch_storage_type=pvc
  openshift_logging_es_pvc_size=104802308Ki
  openshift_logging_es_pvc_prefix=es-logging

If using dynamically provisioned PVs, the OKD logging installer creates PVCs that use the default storage class or the PVC specified with the openshift_logging_elasticsearch_pvc_storage_class_name parameter.

If using NFS storage, the OKD installer creates the persistent volumes, based on the openshift_logging_storage_* parameters and the OKD logging installer creates PVCs, using the openshift_logging_es_pvc_* parameters. Make sure you specify the correct parameters in order to use persistent volumes with EFK. Also set the openshift_enable_unsupported_configurations=true parameter in the Ansible inventory file, as the logging installer blocks the installation of NFS with core infrastructure by default.

If your environment requires NFS storage, use one of the following methods:

• NFS as a persistent volume

• NFS storage as local storage

Using NFS as a persistent volume

You can deploy NFS as an automatically provisioned persistent volume or using a predefined NFS volume.

For more information, see Sharing an NFS mount across two persistent volume claims to leverage shared storage for use by two separate containers.

Using automatically provisioned NFS

To use NFS as a persistent volume where NFS is automatically provisioned:

1. Add the following lines to the Ansible inventory file to create an NFS auto-provisioned storage class and dynamically provision the backing storage:

   openshift_logging_es_pvc_storage_class_name=$nfsclass
   openshift_logging_es_pvc_dynamic=true

2. Use the following command to deploy the NFS volume using the logging playbook:

   ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.yml

3. Use the following steps to create a PVC:

   1. Edit the Ansible inventory file to set the PVC size:

      openshift_logging_es_pvc_size=50Gi

      The logging playbook selects a volume based on size and might use an unexpected volume if any other persistent volume has same size.

   2. Use the following command to rerun the Ansible deploy_cluster.yml playbook:

      ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

      The installer playbook creates the NFS volume based on the openshift_logging_storage variables.

Using a predefined NFS volume

To deploy logging alongside the OKD cluster using an existing NFS volume:

1. Edit the Ansible inventory file to configure the NFS volume and set the PVC size:

   openshift_logging_storage_kind=nfs
   openshift_logging_storage_access_modes=['ReadWriteOnce']
   openshift_logging_storage_nfs_directory=/share (1)
   openshift_logging_storage_nfs_options='*(rw,root_squash)' (1)
   openshift_logging_storage_labels={'storage': 'logging'}
   openshift_logging_elasticsearch_storage_type=pvc
   openshift_logging_es_pvc_size=10Gi
   openshift_logging_es_pvc_storage_class_name=''
   openshift_logging_es_pvc_dynamic=true
   openshift_logging_es_pvc_prefix=logging

   1	These parameters work only with the /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml installation playbook. The parameters will not work with the /usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.yml playbook.

2. Use the following command to redeploy the EFK stack:

   ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

Using NFS as local storage

You can allocate a large file on an NFS server and mount the file to the nodes. You can then use the file as a host path device.

$ mount -F nfs nfserver:/nfs/storage/elasticsearch-1 /usr/local/es-storage
$ chown 1000:1000 /usr/local/es-storage

Then, use /usr/local/es-storage as a host-mount as described below. Use a different backing file as storage for each Elasticsearch node.

This loopback must be maintained manually outside of OKD, on the node. You must not maintain it from inside a container.

It is possible to use a local disk volume (if available) on each node host as storage for an Elasticsearch replica. Doing so requires some preparation as follows.

1. The relevant service account must be given the privilege to mount and edit a local volume:

     $ oc adm policy add-scc-to-user privileged  \
            system:serviceaccount:openshift-logging:aggregated-logging-elasticsearch

   If you upgraded from an earlier version of OKD, cluster logging might have been installed in the logging project. You should adjust the service account accordingly.

2. Each Elasticsearch node definition must be patched to claim that privilege, for example:

   $ for dc in $(oc get deploymentconfig --selector component=es -o name); do
       oc scale $dc --replicas=0
       oc patch $dc \
          -p '{"spec":{"template":{"spec":{"containers":[{"name":"elasticsearch","securityContext":{"privileged": true}}]}}}}'
     done

3. The Elasticsearch replicas must be located on the correct nodes to use the local storage, and must not move around, even if those nodes are taken down for a period of time. This requires giving each Elasticsearch replica a node selector that is unique to a node where an administrator has allocated storage for it. To configure a node selector, edit each Elasticsearch deployment configuration, adding or editing the nodeSelector section to specify a unique label that you have applied for each desired node:

apiVersion: v1
kind: DeploymentConfig
spec:
  template:
    spec:
      nodeSelector:
        logging-es-node: "1" (1)

For example, if your deployment has three infrastructure nodes, you could add labels for those nodes as follows:

$ oc label node <nodename1> logging-es-node=0
$ oc label node <nodename2> logging-es-node=1
$ oc label node <nodename3> logging-es-node=2

For information about adding a label to a node, see Updating Labels on Nodes.

To automate applying the node selector, you can instead use the oc patch command:

$ oc patch dc/logging-es-<suffix> \
   -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-es-node":"0"}}}}}'

Once you have completed these steps, you can apply a local host mount to each replica. The following example assumes storage is mounted at the same path on each node.

$ for dc in $(oc get deploymentconfig --selector component=es -o name); do
    oc set volume $dc \
          --add --overwrite --name=elasticsearch-storage \
          --type=hostPath --path=/usr/local/es-storage
    oc rollout latest $dc
    oc scale $dc --replicas=1
  done

Configuring hostPath storage for Elasticsearch

You can provision OKD clusters using hostPath storage for Elasticsearch.

To use a local disk volume on each node host as storage for an Elasticsearch replica:

1. Create a local mount point on each infrastructure node for the local Elasticsearch storage:

   $ mkdir /usr/local/es-storage

2. Create a filesystem on the Elasticsearch volume:

   $ mkfs.ext4 /dev/xxx

3. Mount the elasticsearch volume:

   $ mount /dev/xxx /usr/local/es-storage

4. Add the following line to /etc/fstab:

   $ /dev/xxx /usr/local/es-storage ext4

5. Change ownership for the mount point:

   $ chown 1000:1000 /usr/local/es-storage

6. Give the privilege to mount and edit a local volume to the relevant service account:

     $ oc adm policy add-scc-to-user privileged  \
            system:serviceaccount:openshift-logging:aggregated-logging-elasticsearch

   If you upgraded from an earlier version of OKD, cluster logging might have been installed in the logging project. You should adjust the service account accordingly.

7. To claim that privilege, patch each Elasticsearch replica definition, as shown in the example, which specifies --selector component=es-ops for an Ops cluster:

     $ for dc in $(oc get deploymentconfig --selector component=es -o name);
   do
       oc scale $dc --replicas=0
       oc patch $dc \
          -p '{"spec":{"template":{"spec":{"containers":[{"name":"elasticsearch","securityContext":{"privileged":
   true}}]}}}}'
   done

8. Locate the Elasticsearch replicas on the correct nodes to use the local storage, and do not move them around, even if those nodes are taken down for a period of time. To specify the node location, give each Elasticsearch replica a node selector that is unique to a node where an administrator has allocated storage for it.

   To configure a node selector, edit each Elasticsearch deployment configuration, adding or editing the nodeSelector section to specify a unique label that you have applied for each node you desire:

   apiVersion: v1
   kind: DeploymentConfig
   spec:
     template:
       spec:
         nodeSelector:
           logging-es-node: "1"

   The label must uniquely identify a replica with a single node that bears that label, in this case logging-es-node=1.

9. Create a node selector for each required node. Use the oc label command to apply labels to as many nodes as needed.

   For example, if your deployment has three infrastructure nodes, you could add labels for those nodes as follows:

     $ oc label node <nodename1> logging-es-node=0
     $ oc label node <nodename2> logging-es-node=1
     $ oc label node <nodename3> logging-es-node=2

   To automate application of the node selector, use the oc patch command instead of the oc label command, as follows:

     $ oc patch dc/logging-es-<suffix> \
        -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-es-node":"1"}}}}}'

10. Once you have completed these steps, you can apply a local host mount to each replica. The following example assumes storage is mounted at the same path on each node, and specifies --selector component=es-ops for an Ops cluster.

    $ for dc in $(oc get deploymentconfig --selector component=es -o name);
    do
       oc set volume $dc \
             --add --overwrite --name=elasticsearch-storage \
             --type=hostPath --path=/usr/local/es-storage
       oc rollout latest $dc
       oc scale $dc --replicas=1
    done

Changing the Scale of Elasticsearch

If you need to scale up the number of Elasticsearch nodes in your cluster, you can create a deployment configuration for each Elasticsearch node you want to add.

Due to the nature of persistent volumes and how Elasticsearch is configured to store its data and recover the cluster, you cannot simply increase the nodes in an Elasticsearch deployment configuration.

The simplest way to change the scale of Elasticsearch is to modify the inventory host file and re-run the logging playbook as described previously. If you have supplied persistent storage for the deployment, this should not be disruptive.

Changing the Number of Elasticsearch Replicas

You can change the number of Elasticsearch replicas by editing the openshift_logging_es_number_of_replicas value in the inventory host file and re-running the logging playbook as described previously.

The changes apply only to the new indices. Existing indices continue to use the previous number of replicas. For example, if you change the number of indices from 3 to 2, your cluster will use 2 replicas for new indices and 3 replicas for existing indices.

You can modify the replica count for the existing indices by running the following command:

$ oc exec -c elasticsearch $pod -- es_util --query=project.* -d '{"index":{"number_of_replicas":"2"}}' (1)

Expose Elasticsearch as a Route

By default, Elasticsearch deployed with OpenShift aggregated logging is not accessible from outside the logging cluster. You can enable a route for external access to Elasticsearch for those tools that want to access its data.

You have access to Elasticsearch using your OpenShift token, and you can provide the external Elasticsearch and Elasticsearch Ops hostnames when creating the server certificate (similar to Kibana).

1. To access Elasticsearch as a reencrypt route, define the following variables:

   openshift_logging_es_allow_external=True
   openshift_logging_es_hostname=elasticsearch.example.com

2. Change to the playbook directory and run the following Ansible playbook:

   $ cd /usr/share/ansible/openshift-ansible
   $ ansible-playbook [-i </path/to/inventory>] \
       playbooks/openshift-logging/config.yml

3. To log in to Elasticsearch remotely, the request must contain three HTTP headers:

   Authorization: Bearer $token
   X-Proxy-Remote-User: $username
   X-Forwarded-For: $ip_address

4. You must have access to the project in order to be able to access to the logs. For example:

   $ oc login <user1>
   $ oc new-project <user1project>
   $ oc new-app <httpd-example>

5. You need to get the token of this ServiceAccount to be used in the request:

   $ token=$(oc whoami -t)

6. Using the token previously configured, you should be able access Elasticsearch through the exposed route:

   $ curl -k -H "Authorization: Bearer $token" -H "X-Proxy-Remote-User: $(oc whoami)" -H "X-Forwarded-For: 127.0.0.1" https://es.example.test/project.my-project.*/_search?q=level:err | python -mjson.tool

Fluentd

Fluentd is deployed as a DaemonSet that deploys nodes according to a node label selector, which you can specify with the inventory parameter openshift_logging_fluentd_nodeselector and the default is logging-infra-fluentd. As part of the OpenShift cluster installation, it is recommended that you add the Fluentd node selector to the list of persisted node labels.

Fluentd uses journald as the system log source. These are log messages from the operating system, the container runtime, and OpenShift.

The available container runtimes provide minimal information to identify the source of log messages. Log collection and normalization of logs can occur after a pod is deleted and additional metadata cannot be retrieved from the API server, such as labels or annotations.

If a pod with a given name and namespace is deleted before the log collector finishes processing logs, there might not be a way to distinguish the log messages from a similarly named pod and namespace. This can cause logs to be indexed and annotated to an index that is not owned by the user who deployed the pod.

Clean installations of OKD 3.9 or later use json-file as the default log driver, but environments upgraded from OKD 3.7 will maintain their existing journald log driver configuration. It is recommended to use the json-file log driver. See Changing the Aggregated Logging Driver for instructions to change your existing log driver configuration to json-file.

Viewing Fluentd Logs

How you view logs depends upon the LOGGING_FILE_PATH setting.

• If LOGGING_FILE_PATH points to a file, use the logs utility to print out the contents of Fluentd log files:

  oc exec <pod> -- logs (1)

  1	Specify the name of the Fluentd pod. Note the space before logs.

  For example:

  oc exec logging-fluentd-lmvms -- logs

  The contents of log files are printed out, starting with the oldest log. Use -f option to follow what is being written into the logs.

• If you are using LOGGING_FILE_PATH=console, Fluentd writes logs to its default location, /var/log/fluentd/fluentd.log. You can retrieve the logs with the oc logs -f <pod_name> command.

  For example

  oc logs -f fluentd.log

Configuring Fluentd Log Location

Fluentd writes logs to a specified file, by default /var/log/fluentd/fluentd.log, or to the console, based on the LOGGING_FILE_PATH environment variable.

To change the default output location for the Fluentd logs, use the LOGGING_FILE_PATH parameter in the default inventory file. You can specify a particular file or use the Fluentd default location:

LOGGING_FILE_PATH=console (1)
LOGGING_FILE_PATH=<path-to-log/fluentd.log> (2)

After changing these parameters, re-run the logging installer playbook:

$ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook [-i </path/to/inventory>] \
    playbooks/openshift-logging/config.yml

Configuring Fluentd Log Rotation

When the current Fluentd log file reaches a specified size, OKD automatically renames the fluentd.log log file so that new logging data can be collected. Log rotation is enabled by default.

The following example shows logs in a cluster where the maximum log size is 1Mb and four logs should be retained. When the fluentd.log reaches 1Mb, OKD deletes the current fluentd.log.4, renames each of the Fluentd logs in turn, and creates a new fluentd.log.

fluentd.log     0b
fluentd.log.1  1Mb
fluentd.log.2  1Mb
fluentd.log.3  1Mb
fluentd.log.4  1Mb

You can control the size of the Fluentd log files and how many of the renamed files that OKD retains using environment variables.

For example:

$ oc set env ds/logging-fluentd LOGGING_FILE_AGE=30 LOGGING_FILE_SIZE=1024000"

Turn off log rotation by setting LOGGING_FILE_PATH=console. This causes Fluentd to write logs to the Fluentd default location, /var/log/fluentd/fluentd.log, where you can retrieve them using the oc logs -f <pod_name> command.

$ oc set env ds/fluentd LOGGING_FILE_PATH=console

Disabling JSON parsing of logs with MERGE_JSON_LOG

By default, Fluentd determines if a log message is in JSON format and merges the message into the JSON payload document posted to Elasticsearch.

When using JSON parsing you might experience:

• log loss due to Elasticsearch rejecting documents due to inconsistent type mappings;

• buffer storage leaks caused by rejected message cycling;

• overwritten data for fields with same names.

For information on how to mitigate some of these problems, see Configuring how the log collector normalizes logs.

You can disable JSON parsing to avoid these problems or if you do not need to parse JSON from your logs.

To disable JSON parsing:

1. Run the following command:

   oc set env ds/logging-fluentd MERGE_JSON_LOG=false (1)

   1	Set this to false to disable this feature or true to enable this feature.

   To ensure this setting is applied each time you run Ansible, add openshift_logging_fluentd_merge_json_log="false" to your Ansible inventory.

Configuring how the log collector normalizes logs

Cluster Logging uses a specific data model, like a database schema, to store log records and their metadata in the logging store. There are some restrictions on the data:

• There must be a "message" field containing the actual log message.

• There must be a "@timestamp" field containing the log record timestamp in RFC 3339 format, preferably millisecond or better resolution.

• There must be a "level" field with the log level, such as err, info, unknown, and so forth.

Because of these requirements, conflicts and inconsistencies can arise with log data collected from different subsystems.

For example, if you use the MERGE_JSON_LOG feature (MERGE_JSON_LOG=true), it can be extremely useful to have your applications log their output in JSON, and have the log collector automatically parse and index the data in Elasticsearch. However, this leads to several problems, including:

• field names can be empty, or contain characters that are illegal in Elasticsearch;

• different applications in the same namespace might output the same field name with a different value data type;

• applications might emit too many fields;

• fields may conflict with the cluster logging built-in fields.

You can configure how cluster logging treats fields from disparate sources by editing the Fluentd log collector daemonset and setting environment variables in the table below.

• Undefined fields. Fields unknown to the ViaQ data model are called undefined. Log data from disparate systems can contain undefined fields. The data model requires all top-level fields to be defined and described.

  Use the parameters to configure how OKD moves any undefined fields under a top-level field called undefined to avoid conflicting with the well known top-level fields. You can add undefined fields to the top-level fields and move others to an undefined container.

  You can also replace special characters in undefined fields and convert undefined fields to their JSON string representation. Converting to JSON string preserves the structure of the value, so that you can retrieve the value later and convert it back to a map or an array.

  • Simple scalar values like numbers and booleans are changed to a quoted string. For example: 10 becomes "10", 3.1415 becomes "3.1415", false becomes "false".

  • Map/dict values and array values are converted to their JSON string representation: "mapfield":{"key":"value"} becomes "mapfield":"{\"key\":\"value\"}" and "arrayfield":[1,2,"three"] becomes "arrayfield":"[1,2,\"three\"]".

• Defined fields. Defined fields appear in the top levels of the logs. You can configure which fields are considered defined fields.

  The default top-level fields, defined through the CDM_DEFAULT_KEEP_FIELDS parameter, are CEE, time, @timestamp, aushape, ci_job, collectd, docker, fedora-ci, file, foreman, geoip, hostname, ipaddr4, ipaddr6, kubernetes, level, message, namespace_name, namespace_uuid, offset, openstack, ovirt, pid, pipeline_metadata, service, systemd, tags, testcase, tlog, viaq_msg_id.

  Any fields not included in ${CDM_DEFAULT_KEEP_FIELDS} or ${CDM_EXTRA_KEEP_FIELDS} are moved to ${CDM_UNDEFINED_NAME} if CDM_USE_UNDEFINED is true. See the table below for more information on these parameters.

  The CDM_DEFAULT_KEEP_FIELDS parameter is for only advanced users, or if you are instructed to do so by Red Hat support.

• Empty fields. Empty fields have no data. You can determine which empty fields to retain from logs.

To configure undefined and empty field processing, edit the logging-fluentd daemonset:

1. Configure how to process fields, as needed:

   1. Specify the fields to move using CDM_EXTRA_KEEP_FIELDS.

   2. Specify any empty fields to retain in the CDM_KEEP_EMPTY_FIELDS parameter in CSV format.

1. Configure how to process undefined fields, as needed:

   1. Set CDM_USE_UNDEFINED to true to move undefined fields to the top-level undefined field:

   2. Specify a name for the undefined fields using the CDM_UNDEFINED_NAME parameter.

   3. Set CDM_UNDEFINED_MAX_NUM_FIELDS to a value other than the default -1, to set an upper bound on the number of undefined fields in a single record.

1. Specify CDM_UNDEFINED_DOT_REPLACE_CHAR to change any dot . characters in an undefined field name to another character. For example, if CDM_UNDEFINED_DOT_REPLACE_CHAR=@@@ and there is a field named foo.bar.baz the field is transformed into foo@@@bar@@@baz.

2. Set UNDEFINED_TO_STRING to true to convert undefined fields to their JSON string representation.

Setting MERGE_JSON_LOG and CDM_UNDEFINED_TO_STRING

If you set the MERGE_JSON_LOG and CDM_UNDEFINED_TO_STRING enviroment variables to true, you might receive an Elasticsearch 400 error. When MERGE_JSON_LOG=true, the log collector adds fields with data types other than string. If you set CDM_UNDEFINED_TO_STRING=true, Fluentd attempts to add those fields as a string value resulting in the Elasticsearch 400 error. The error clears when the indices roll over for the next day.

When Fluentd rolls over the indices for the next day’s logs, it will create a brand new index. The field definitions are updated and you will not get the 400 error.

Records that have hard errors, such as schema violations, corrupted data, and so forth, cannot be retried. The log collector sends the records for error handling. If you add a <label @ERROR> section to your Fluentd config, as the last <label>, you can handle these records as needed.

For example:

data:
  fluent.conf:
....
    <label @ERROR>
      <match **>
        @type file
        path /var/log/fluent/dlq
        time_slice_format %Y%m%d
        time_slice_wait 10m
        time_format %Y%m%dT%H%M%S%z
        compress gzip
      </match>
    </label>

This section writes error records to the Elasticsearch dead letter queue (DLQ) file. See the fluentd documentation for more information about the file output.

Then you can edit the file to clean up the records manually, edit the file to use with the Elasticsearch /_bulk index API and use cURL to add those records. For more information on Elasticsearch Bulk API, see the Elasticsearch documentation.

Join Multi-line Docker Logs

You can configure Fluentd to reconstruct whole log records from Docker log partial fragments. With this feature active, Fluentd reads multi-line Docker logs, reconstructs them, and stores the logs as one record in Elasticsearch with no missing data.

However, because this feature can cause a performance regression, the feature is off by default and must be manually enabled.

The following Fluentd environment variables configure cluster logging to process multi-line Docker logs:

You can use the following command to determine which log driver is being used:

$ docker info | grep -i log

One of the following is output:

Logging Driver: json-file

Logging Driver: journald

To turn on multi-line Docker logs processing:

1. Use the following command to enable the multiline Docker logs:

   • For the json-file log driver:

     oc set env daemonset/logging-fluentd USE_MULTILINE_JSON=true

   • For the journald log driver:

     oc set env daemonset/logging-fluentd USE_MULTILINE_JOURNAL=true

The Fluentd pods in the cluster restart.

Configuring Fluentd to Send Logs to an External Log Aggregator

You can configure Fluentd to send a copy of its logs to an external log aggregator, in addition to the default Elasticsearch, using the secure-forward plug-in. From there, you can further process log records after the locally hosted Fluentd has processed them.

The secure-forward plug-in is provided with the Fluentd image as of v1.4.0.

The logging deployment provides a secure-forward.conf section in the Fluentd configmap for configuring the external aggregator:

<store>
@type secure_forward
self_hostname pod-${HOSTNAME}
shared_key thisisasharedkey
secure yes
enable_strict_verification yes
ca_cert_path /etc/fluent/keys/your_ca_cert
ca_private_key_path /etc/fluent/keys/your_private_key
ca_private_key_passphrase passphrase
<server>
  host ose1.example.com
  port 24284
</server>
<server>
  host ose2.example.com
  port 24284
  standby
</server>
<server>
  host ose3.example.com
  port 24284
  standby
</server>
</store>

This can be updated using the oc edit command:

$ oc edit configmap/logging-fluentd

Certificates to be used in secure-forward.conf can be added to the existing secret that is mounted on the Fluentd pods. The your_ca_cert and your_private_key values must match what is specified in secure-forward.conf in configmap/logging-fluentd:

$ oc patch secrets/logging-fluentd --type=json \
  --patch "[{'op':'add','path':'/data/your_ca_cert','value':'$(base64 -w 0 /path/to/your_ca_cert.pem)'}]"
$ oc patch secrets/logging-fluentd --type=json \
  --patch "[{'op':'add','path':'/data/your_private_key','value':'$(base64 -w 0 /path/to/your_private_key.pem)'}]"

When configuring the external aggregator, it must be able to accept messages securely from Fluentd.

If the external aggregator is another Fluentd server, it must have the fluent-plugin-secure-forward plug-in installed and make use of the input plug-in it provides:

<source>
  @type secure_forward
  self_hostname ${HOSTNAME}
  bind 0.0.0.0
  port 24284
  shared_key thisisasharedkey
  secure yes
  cert_path        /path/for/certificate/cert.pem
  private_key_path /path/for/certificate/key.pem
  private_key_passphrase secret_foo_bar_baz
</source>

You can find further explanation of how to set up the fluent-plugin-secure-forward plug-in in the fluent-plugin-secure-forward repository.

Reducing the Number of Connections from Fluentd to the API Server

mux is a Secure Forward listener service.

Throttling logs in Fluentd

For projects that are especially verbose, an administrator can throttle down the rate at which the logs are read in by Fluentd before being processed.

To tell Fluentd which projects it should be restricting, edit the throttle configuration in its ConfigMap after deployment:

$ oc edit configmap/logging-fluentd

The format of the throttle-config.yaml key is a YAML file that contains project names and the desired rate at which logs are read in on each node. The default is 1000 lines at a time per node. For example:

• Projects

project-name:
  read_lines_limit: 50
second-project-name:
  read_lines_limit: 100

• Logging

logging:
  read_lines_limit: 500
test-project:
  read_lines_limit: 10
.operations:
  read_lines_limit: 100

When you make changes to any part of the EFK stack, specifically Elasticsearch or Fluentd, you should first scale Elasticsearch down to zero and scale Fluentd so it does not match any other nodes. Then, make the changes and scale Elasticsearch and Fluentd back.

To scale Elasticsearch to zero:

$ oc scale --replicas=0 dc/<ELASTICSEARCH_DC>

Change nodeSelector in the daemonset configuration to match zero:

Get the Fluentd node selector:

$ oc get ds logging-fluentd -o yaml |grep -A 1 Selector
     nodeSelector:
       logging-infra-fluentd: "true"

Use the oc patch command to modify the daemonset nodeSelector:

$ oc patch ds logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"nonexistlabel":"true"}}}}}'

Get the Fluentd node selector:

$ oc get ds logging-fluentd -o yaml |grep -A 1 Selector
     nodeSelector:
       "nonexistlabel: "true"

Scale Elasticsearch back up from zero:

$ oc scale --replicas=# dc/<ELASTICSEARCH_DC>

Change nodeSelector in the daemonset configuration back to logging-infra-fluentd: “true”.

Use the oc patch command to modify the daemonset nodeSelector:

oc patch ds logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-fluentd":"true"}}}}}'

Tune Buffer Chunk Limit

If Fluentd logger is unable to keep up with a high number of logs, it will need to switch to file buffering to reduce memory usage and prevent data loss.

The Fluentd buffer_chunk_limit is determined by the environment variable BUFFER_SIZE_LIMIT, which has the default value 8m. The file buffer size per output is determined by the environment variable FILE_BUFFER_LIMIT, which has the default value 256Mi. The permanent volume size must be larger than FILE_BUFFER_LIMIT multiplied by the output.

On the Fluentd and Mux pods, permanent volume /var/lib/fluentd should be prepared by the PVC or hostmount, for example. That area is then used for the file buffers.

The buffer_type and buffer_path are configured in the Fluentd configuration files as follows:

$ egrep "buffer_type|buffer_path" *.conf
output-es-config.conf:
  buffer_type file
  buffer_path `/var/lib/fluentd/buffer-output-es-config`
output-es-ops-config.conf:
  buffer_type file
  buffer_path `/var/lib/fluentd/buffer-output-es-ops-config`
filter-pre-mux-client.conf:
  buffer_type file
  buffer_path `/var/lib/fluentd/buffer-mux-client`

The Fluentd buffer_queue_limit is the value of the variable BUFFER_QUEUE_LIMIT. This value is 32 by default.

The environment variable BUFFER_QUEUE_LIMIT is calculated as (FILE_BUFFER_LIMIT / (number_of_outputs * BUFFER_SIZE_LIMIT)).

If the BUFFER_QUEUE_LIMIT variable has the default set of values:

• FILE_BUFFER_LIMIT = 256Mi

• number_of_outputs = 1

• BUFFER_SIZE_LIMIT = 8Mi

The value of buffer_queue_limit will be 32. To change the buffer_queue_limit, you need to change the value of FILE_BUFFER_LIMIT.

In this formula, number_of_outputs is 1 if all the logs are sent to a single resource, and it is incremented by 1 for each additional resource. For example, the value of number_of_outputs is:

• 1 - if all logs are sent to a single ElasticSearch pod

• 2 - if application logs are sent to an ElasticSearch pod and ops logs are sent to another ElasticSearch pod

• 4 - if application logs are sent to an ElasticSearch pod, ops logs are sent to another ElasticSearch pod, and both of them are forwarded to other Fluentd instances

Kibana

To access the Kibana console from the OKD web console, add the loggingPublicURL parameter in the master webconsole-config configmap file, with the URL of the Kibana console (the kibana-hostname parameter). The value must be an HTTPS URL:

...
clusterInfo:
  ...
  loggingPublicURL: "https://kibana.example.com"
...

Setting the loggingPublicURL parameter creates a View Archive button on the OKD web console under the Browse → Pods → → Logs tab. This links to the Kibana console.

You can scale the Kibana deployment as usual for redundancy:

$ oc scale dc/logging-kibana --replicas=2

You can see the user interface by visiting the site specified by the openshift_logging_kibana_hostname variable.

See the Kibana documentation for more information on Kibana.

Kibana Visualize

Kibana Visualize enables you to create visualizations and dashboards for monitoring container and pod logs allows administrator users (cluster-admin or cluster-reader) to view logs by deployment, namespace, pod, and container.

Kibana Visualize exists inside the Elasticsearch and ES-OPS pod, and must be run inside those pods. To load dashboards and other Kibana UI objects, you must first log into Kibana as the user you want to add the dashboards to, then log out. This will create the necessary per-user configuration that the next step relies on. Then, run:

$ oc exec <$espod> -- es_load_kibana_ui_objects <user-name>

Where $espod is the name of any one of your Elasticsearch pods.

Curator

Curator allows administrators to configure scheduled Elasticsearch maintenance operations to be performed automatically on a per-project basis. It is scheduled to perform actions daily based on its configuration. Only one Curator pod is recommended per Elasticsearch cluster. Curator pods only run at the time stated in the cronjob and then the pod terminates upon completion. Curator is configured via a YAML configuration file with the following structure:

$PROJECT_NAME:
  $ACTION:
    $UNIT: $VALUE
$PROJECT_NAME:
  $ACTION:
    $UNIT: $VALUE
 ...

The available parameters are:

For example, to configure Curator to:

• Delete indices in the myapp-dev project older than 1 day

• Delete indices in the myapp-qe project older than 1 week

• Delete operations logs older than 8 weeks

• Delete all other projects indices after they are 31 days old

• Delete indices older than 1 day that are matched by the ‘^project\..+\-dev.*$’ regex

• Delete indices older than 2 days that are matched by the ‘^project\..+\-test.*$’ regex

Use:

config.yaml: |
  myapp-dev:
    delete:
      days: 1
  myapp-qe:
    delete:
      weeks: 1
  .operations:
    delete:
      weeks: 8
  .defaults:
    delete:
      days: 31
  .regex:
    - pattern: '^project\..+\-dev\..*$'
      delete:
        days: 1
    - pattern: '^project\..+\-test\..*$'
      delete:
        days: 2

Using the Curator Actions File

Setting the OKD custom configuration file format ensures internal indices are not mistakenly deleted.

To use the actions file, add an exclude rule to your Curator configuration to retain these indices. You must manually add all of the required patterns.

actions.yaml: |
actions:
    action: delete_indices
    description: be careful!
    filters:
    - exclude: false
      kind: regex
      filtertype: pattern
      value: '^project\.myapp\..*$'
    - direction: older
      filtertype: age
      source: name
      timestring: '%Y.%m.%d'
      unit_count: 7
      unit: days
    options:
      continue_if_exception: false
      timeout_override: '300'
      ignore_empty_list: true
    action: delete_indices
    description: be careful!
    filters:
    - exclude: false
      kind: regex
      filtertype: pattern
      value: '^\.operations\..*$'
    - direction: older
      filtertype: age
      source: name
      timestring: '%Y.%m.%d'
      unit_count: 56
      unit: days
    options:
      continue_if_exception: false
      timeout_override: '300'
      ignore_empty_list: true
    action: delete_indices
    description: be careful!
    filters:
    - exclude: true
      kind: regex
      filtertype: pattern
      value: '^project\.myapp\..*$|^\.operations\..*$|^\.searchguard\..*$|^\.kibana$'
    - direction: older
      filtertype: age
      source: name
      timestring: '%Y.%m.%d'
      unit_count: 30
      unit: days
    options:
      continue_if_exception: false
      timeout_override: '300'
      ignore_empty_list: true

Creating the Curator Configuration

The openshift_logging Ansible role provides a ConfigMap from which Curator reads its configuration. You may edit or replace this ConfigMap to reconfigure Curator. Currently the logging-curator ConfigMap is used to configure both your ops and non-ops Curator instances. Any .operations configurations are in the same location as your application logs configurations.

1. To create the Curator configuration, edit the configuration in the deployed ConfigMap:

   $ oc edit configmap/logging-curator

   Or, manually create the jobs from a cronjob:

   oc create job --from=cronjob/logging-curator <job_name>

   • For scripted deployments, copy the configuration file that was created by the installer and create your new OKD custom configuration:

     $ oc extract configmap/logging-curator --keys=curator5.yaml,config.yaml --to=/my/config
       edit /my/config/curator5.yaml
       edit /my/config/config.yaml
     $ oc delete configmap logging-curator ; sleep 1
     $ oc create configmap logging-curator \
         --from-file=curator5.yaml=/my/config/curator5.yaml \
         --from-file=config.yaml=/my/config/config.yaml \
         ; sleep 1

   • Alternatively, if you are using the actions file:

     $ oc extract configmap/logging-curator --keys=curator5.yaml,actions.yaml --to=/my/config
       edit /my/config/curator5.yaml
       edit /my/config/actions.yaml
     $ oc delete configmap logging-curator ; sleep 1
     $ oc create configmap logging-curator \
         --from-file=curator5.yaml=/my/config/curator5.yaml \
         --from-file=actions.yaml=/my/config/actions.yaml \
         ; sleep 1

The next scheduled job uses this configuration.

You can use the following commands to control the cronjob:

# suspend cronjob
oc patch cronjob logging-curator -p '{"spec":{"suspend":true}}'
# resume cronjob
oc patch cronjob logging-curator -p '{"spec":{"suspend":false}}
# change cronjob schedule
oc patch cronjob logging-curator -p '{"spec":{"schedule":"0 0 * * *"}}' (1)

Cleanup

Remove everything generated during the deployment.

$ ansible-playbook playbooks/openshift-logging/config.yml \
    -e openshift_logging_install_logging=False

Sending Logs to an External Elasticsearch Instance

Fluentd sends logs to the value of the ES_HOST, ES_PORT, OPS_HOST, and OPS_PORT environment variables of the Elasticsearch deployment configuration. The application logs are directed to the ES_HOST destination, and operations logs to OPS_HOST.

To direct logs to a specific Elasticsearch instance, edit the deployment configuration and replace the value of the above variables with the desired instance:

$ oc edit ds/<daemon_set>

For an external Elasticsearch instance to contain both application and operations logs, you can set ES_HOST and OPS_HOST to the same destination, while ensuring that ES_PORT and OPS_PORT also have the same value.

Only Mutual TLS configuration is supported, as the provided Elasticsearch instance does. Patch or recreate the logging-fluentd secret with your client key, client cert, and CA.

Sending Logs to an External Syslog Server

Use the fluent-plugin-remote-syslog plug-in on the host to send logs to an external syslog server.

Set environment variables in the logging-fluentd or logging-mux daemonsets:

- name: REMOTE_SYSLOG_HOST (1)
  value: host1
- name: REMOTE_SYSLOG_HOST_BACKUP
  value: host2
- name: REMOTE_SYSLOG_PORT_BACKUP
  value: 5555

This will build two destinations. The syslog server on host1 will be receiving messages on the default port of 514, while host2 will be receiving the same messages on port 5555.

Alternatively, you can configure your own custom fluent.conf in the logging-fluentd or logging-mux ConfigMaps.

Fluentd Environment Variables

Fluentd Logging Ansible Variables

Mux Logging Ansible Variables

Performing Administrative Elasticsearch Operations

As of logging version 1.2.0, an administrator certificate, key, and CA that can be used to communicate with and perform administrative operations on Elasticsearch are provided within the logging-elasticsearch secret.

$ oc describe secret logging-elasticsearch

1. Connect to an Elasticsearch pod that is in the cluster on which you are attempting to perform maintenance.

2. To find a pod in a cluster use either:

   $ oc get pods -l component=es -o name | head -1
   $ oc get pods -l component=es-ops -o name | head -1

3. Connect to a pod:

   $ oc rsh <your_Elasticsearch_pod>

4. Once connected to an Elasticsearch container, you can use the certificates mounted from the secret to communicate with Elasticsearch per its Indices APIs documentation.

   Fluentd sends its logs to Elasticsearch using the index format project.{project_name}.{project_uuid}.YYYY.MM.DD where YYYY.MM.DD is the date of the log record.

   For example, to delete all logs for the openshift-logging project with uuid 3b3594fa-2ccd-11e6-acb7-0eb6b35eaee3 from June 15, 2016, we can run:

   $ curl --key /etc/elasticsearch/secret/admin-key \
     --cert /etc/elasticsearch/secret/admin-cert \
     --cacert /etc/elasticsearch/secret/admin-ca -XDELETE \
     "https://localhost:9200/project.logging.3b3594fa-2ccd-11e6-acb7-0eb6b35eaee3.2016.06.15"

Redeploying EFK Certificates

You can use an Ansible playbook to perform a certificate rotation for the EFK stack without needing to run the install/upgrade playbook.

This playbook deletes the current certificate files, generates new EFK certificates, updates certificate secrets, and restarts Kibana and Elasticsearch to force those components to read in the updated certificates.

To redeploy EFK certificates:

1. Use the Ansible playbook to redeploy the EFK certificates:

   $ ansible-playbook playbooks/openshift-logging/redeploy-certificates.yml

Changing the Aggregated Logging Driver

For aggregated logging, it is recommended to use the json-file log driver.

Fluentd determines the driver Docker is using by checking the /etc/docker/daemon.json and /etc/sysconfig/docker files.

You can determine which driver Docker is using with the docker info command:

# docker info | grep Logging
Logging Driver: journald

To change to json-file:

1. Modify either the /etc/sysconfig/docker or /etc/docker/daemon.json files.

   For example:

   # cat /etc/sysconfig/docker
   OPTIONS=' --selinux-enabled --log-driver=json-file --log-opt max-size=1M --log-opt max-file=3 --signature-verification=False'
   cat /etc/docker/daemon.json
   {
   "log-driver": "json-file",
   "log-opts": {
   "max-size": "1M",
   "max-file": "1"
   }
   }

2. Restart the Docker service:

   systemctl restart docker

3. Restart Fluentd.

   Restarting Fluentd on more than a dozen nodes at once will create a large load on the Kubernetes scheduler. Exercise caution when using the following the directions to restart Fluentd.

   There are two methods for restarting Fluentd. You can restart the Fluentd on one node or a set of nodes, or on all nodes.

   1. The following steps demonstrate how to restart Fluentd on one node or a set of nodes.

      1. List the nodes where Fluentd is running:

         $ oc get nodes -l logging-infra-fluentd=true

      2. For each node, remove the label and turn off Fluentd:

         $ oc label node $node logging-infra-fluentd-

      3. Verify Fluentd is off:

         $ oc get pods -l component=fluentd

      4. For each node, restart Fluentd:

         $ oc label node $node logging-infra-fluentd=true

2.  The following steps demonstrate how to restart the Fluentd all nodes.
    1.  Turn off Fluentd on all nodes:
        ```
        $ oc label node -l logging-infra-fluentd=true --overwrite logging-infra-fluentd=false
        ```
    2.  Verify Fluentd is off:
        ```
        $ oc get pods -l component=fluentd
        ```
    3.  Restart Fluentd on all nodes:
        ```
        $ oc label node -l logging-infra-fluentd=false --overwrite logging-infra-fluentd=true
        ```
    4.  Verify Fluentd is on:
        ```
        $ oc get pods -l component=fluentd
        ```

Exported Fields

These are the fields exported by the logging system and available for searching from Elasticsearch and Kibana. Use the full, dotted field name when searching. For example, for an Elasticsearch /_search URL, to look for a Kubernetes pod name, use /_search/q=kubernetes.pod_name:name-of-my-pod. The following sections describe fields that may not be present in your logging store. Not all of these fields are present in every record. The fields are grouped in the following categories:

• exported-fields-Default

• exported-fields-rsyslog

• exported-fields-systemd

• exported-fields-kubernetes

• exported-fields-docker

• exported-fields-pipeline_metadata

• exported-fields-ovirt

• exported-fields-aushape

• exported-fields-tlog

Top Level Fields

The top level fields are common to every application, and may be present in every record. For the Elasticsearch template, top level fields populate the actual mappings of default in the template’s mapping section.

collectd Fields

The following fields represent namespace metrics metadata.

collectd.processes Fields

The following field corresponds to the collectd processes plug-in.

collectd.processes.ps_disk_ops Fields

The collectd ps_disk_ops type of processes plug-in.

collectd.processes.ps_cputime Fields

The collectd ps_cputime type of processes plug-in.

collectd.processes.ps_count Fields

The collectd ps_count type of processes plug-in.

collectd.processes.ps_pagefaults Fields

The collectd ps_pagefaults type of processes plug-in.

collectd.processes.ps_disk_octets Fields

The collectd ps_disk_octets type of processes plug-in.

collectd.disk Fields

Corresponds to collectd disk plug-in.

collectd.disk.disk_merged Fields

The collectd disk_merged type of disk plug-in.

collectd.disk.disk_octets Fields

The collectd disk_octets type of disk plug-in.

collectd.disk.disk_time Fields

The collectd disk_time type of disk plug-in.

collectd.disk.disk_ops Fields

The collectd disk_ops type of disk plug-in.

collectd.disk.disk_io_time Fields

The collectd disk_io_time type of disk plug-in.

collectd.interface Fields

Corresponds to the collectd interface plug-in.

collectd.interface.if_octets Fields

The collectd if_octets type of interface plug-in.

collectd.interface.if_packets Fields

The collectd if_packets type of interface plug-in.

collectd.interface.if_errors Fields

The collectd if_errors type of interface plug-in.

collectd.interface.if_dropped Fields

The collectd if_dropped type of interface plug-in.

collectd.virt Fields

Corresponds to collectd virt plug-in.

collectd.virt.if_octets Fields

The collectd if_octets type of virt plug-in.

collectd.virt.if_packets Fields

The collectd if_packets type of virt plug-in.

collectd.virt.if_errors Fields

The collectd if_errors type of virt plug-in.

collectd.virt.if_dropped Fields

The collectd if_dropped type of virt plug-in.

collectd.virt.disk_ops Fields

The collectd disk_ops type of virt plug-in.

collectd.virt.disk_octets Fields

The collectd disk_octets type of virt plug-in.

collectd.CPU Fields

Corresponds to the collectd CPU plug-in.

collectd.df Fields

Corresponds to the collectd df plug-in.

collectd.entropy Fields

Corresponds to the collectd entropy plug-in.

collectd.nfs Fields

Corresponds to the collectd NFS plug-in.

collectd.memory Fields

Corresponds to the collectd memory plug-in.

collectd.swap Fields

Corresponds to the collectd swap plug-in.

collectd.load Fields

Corresponds to the collectd load plug-in.

collectd.load.load Fields

The collectd load type of load plug-in

collectd.aggregation Fields

Corresponds to collectd aggregation plug-in.

collectd.statsd Fields

Corresponds to collectd statsd plug-in.

collectd.postgresql Fields

Corresponds to collectd postgresql plug-in.

rsyslog Fields

The following fields are RFC5424 based metadata.

systemd Fields

Contains common fields specific to systemd journal. Applications may write their own fields to the journal. These will be available under the systemd.u namespace. RESULT and UNIT are two such fields.

systemd.k Fields

The following table contains systemd kernel-specific metadata.