Viewing cluster logs by using Kibana

OpenShift Logging includes a web console for visualizing collected log data. Currently, OKD deploys the Kibana console for visualization.

Using the log visualizer, you can do the following with your data:

• search and browse the data using the Discover tab.

• chart and map the data using the Visualize tab.

• create and view custom dashboards using the Dashboard tab.

Use and configuration of the Kibana interface is beyond the scope of this documentation. For more information, on using the interface, see the Kibana documentation.

Defining Kibana index patterns

An index pattern defines the Elasticsearch indices that you want to visualize. To explore and visualize data in Kibana, you must create an index pattern.

Prerequisites

• A user must have the cluster-admin role, the cluster-reader role, or both roles to view the infra and audit indices in Kibana. The default kubeadmin user has proper permissions to view these indices.

  If you can view the pods and logs in the default, kube- and openshift- projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:

  $ oc auth can-i get pods/log -n <project>

  Example output

  yes

  The audit logs are not stored in the internal OKD Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default output for audit logs.

• Elasticsearch documents must be indexed before you can create index patterns. This is done automatically, but it might take a few minutes in a new or updated cluster.

Procedure

To define index patterns and create visualizations in Kibana:

1. In the OKD console, click the Application Launcher and select Logging.

2. Create your Kibana index patterns by clicking Management → Index Patterns → Create index pattern:

   • Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named app and use the @timestamp time field to view their container logs.

   • Each admin user must create index patterns when logged into Kibana the first time for the app, infra, and audit indices using the @timestamp time field.

3. Create Kibana Visualizations from the new index patterns.

Viewing cluster logs in Kibana

You view cluster logs in the Kibana web console. The methods for viewing and visualizing your data in Kibana that are beyond the scope of this documentation. For more information, refer to the Kibana documentation.

Prerequisites

• OpenShift Logging and Elasticsearch must be installed.

• Kibana index patterns must exist.

• A user must have the cluster-admin role, the cluster-reader role, or both roles to view the infra and audit indices in Kibana. The default kubeadmin user has proper permissions to view these indices.

  If you can view the pods and logs in the default, kube- and openshift- projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:

  $ oc auth can-i get pods/log -n <project>

  Example output

  yes

  The audit logs are not stored in the internal OKD Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default output for audit logs.

Procedure

To view logs in Kibana:

1. In the OKD console, click the Application Launcher and select Logging.

2. Log in using the same credentials you use to log in to the OKD console.

   The Kibana interface launches.

3. In Kibana, click Discover.

4. Select the index pattern you created from the drop-down menu in the top-left corner: app, audit, or infra.

   The log data displays as time-stamped documents.

5. Expand one of the time-stamped documents.

6. Click the JSON tab to display the log entry for that document.

   Sample infrastructure log entry in Kibana

   {
     "_index": "infra-000001",
     "_type": "_doc",
     "_id": "YmJmYTBlNDkZTRmLTliMGQtMjE3NmFiOGUyOWM3",
     "_version": 1,
     "_score": null,
     "_source": {
       "docker": {
         "container_id": "f85fa55bbef7bb783f041066be1e7c267a6b88c4603dfce213e32c1"
       },
       "kubernetes": {
         "container_name": "registry-server",
         "namespace_name": "openshift-marketplace",
         "pod_name": "redhat-marketplace-n64gc",
         "container_image": "registry.redhat.io/redhat/redhat-marketplace-index:v4.7",
         "container_image_id": "registry.redhat.io/redhat/redhat-marketplace-index@sha256:65fc0c45aabb95809e376feb065771ecda9e5e59cc8b3024c4545c168f",
         "pod_id": "8f594ea2-c866-4b5c-a1c8-a50756704b2a",
         "host": "ip-10-0-182-28.us-east-2.compute.internal",
         "master_url": "https://kubernetes.default.svc",
         "namespace_id": "3abab127-7669-4eb3-b9ef-44c04ad68d38",
         "namespace_labels": {
           "openshift_io/cluster-monitoring": "true"
         },
         "flat_labels": [
           "catalogsource_operators_coreos_com/update=redhat-marketplace"
         ]
       },
       "message": "time=\"2020-09-23T20:47:03Z\" level=info msg=\"serving registry\" database=/database/index.db port=50051",
       "level": "unknown",
       "hostname": "ip-10-0-182-28.internal",
       "pipeline_metadata": {
         "collector": {
           "ipaddr4": "10.0.182.28",
           "inputname": "fluent-plugin-systemd",
           "name": "fluentd",
           "received_at": "2020-09-23T20:47:15.007583+00:00",
           "version": "1.7.4 1.6.0"
         }
       },
       "@timestamp": "2020-09-23T20:47:03.422465+00:00",
       "viaq_msg_id": "YmJmYTBlNDktMDMGQtMjE3NmFiOGUyOWM3",
       "openshift": {
         "labels": {
           "logging": "infra"
         }
       }
     },
     "fields": {
       "@timestamp": [
         "2020-09-23T20:47:03.422Z"
       ],
       "pipeline_metadata.collector.received_at": [
         "2020-09-23T20:47:15.007Z"
       ]
     },
     "sort": [
       1600894023422
     ]
   }