Viewing cluster logging status

You can view the status of the Cluster Logging Operator and for a number of cluster logging components.

Viewing the status of the Cluster Logging Operator

You can view the status of your Cluster Logging Operator.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Change to the openshift-logging project.

    1. $ oc project openshift-logging

  2. To view the cluster logging status:

    1. Get the cluster logging status:

      1. $ oc get clusterlogging instance -o yaml

      Example output

      1. apiVersion: logging.openshift.io/v1
      2. kind: ClusterLogging
      4. ....
      6. status:  (1)
      7.   collection:
      8.     logs:
      9.       fluentdStatus:
      10.         daemonSet: fluentd  (2)
      11.         nodes:
      12.           fluentd-2rhqp: ip-10-0-169-13.ec2.internal
      13.           fluentd-6fgjh: ip-10-0-165-244.ec2.internal
      14.           fluentd-6l2ff: ip-10-0-128-218.ec2.internal
      15.           fluentd-54nx5: ip-10-0-139-30.ec2.internal
      16.           fluentd-flpnn: ip-10-0-147-228.ec2.internal
      17.           fluentd-n2frh: ip-10-0-157-45.ec2.internal
      18.         pods:
      19.           failed: []
      20.           notReady: []
      21.           ready:
      22.           - fluentd-2rhqp
      23.           - fluentd-54nx5
      24.           - fluentd-6fgjh
      25.           - fluentd-6l2ff
      26.           - fluentd-flpnn
      27.           - fluentd-n2frh
      28.   logstore: (3)
      29.     elasticsearchStatus:
      30.     - ShardAllocationEnabled:  all
      31.       cluster:
      32.         activePrimaryShards:    5
      33.         activeShards:           5
      34.         initializingShards:     0
      35.         numDataNodes:           1
      36.         numNodes:               1
      37.         pendingTasks:           0
      38.         relocatingShards:       0
      39.         status:                 green
      40.         unassignedShards:       0
      41.       clusterName:             elasticsearch
      42.       nodeConditions:
      43.         elasticsearch-cdm-mkkdys93-1:
      44.       nodeCount:  1
      45.       pods:
      46.         client:
      47.           failed:
      48.           notReady:
      49.           ready:
      50.           - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
      51.         data:
      52.           failed:
      53.           notReady:
      54.           ready:
      55.           - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
      56.         master:
      57.           failed:
      58.           notReady:
      59.           ready:
      60.           - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
      61. visualization:  (4)
      62.     kibanaStatus:
      63.     - deployment: kibana
      64.       pods:
      65.         failed: []
      66.         notReady: []
      67.         ready:
      68.         - kibana-7fb4fd4cc9-f2nls
      69.       replicaSets:
      70.       - kibana-7fb4fd4cc9
      71.       replicas: 1
      1In the output, the cluster status fields appear in the status stanza.
      2Information on the Fluentd pods.
      3Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.
      4Information on the Kibana pods.

Example condition messages

The following are examples of some condition messages from the Status.Nodes section of the cluster logging instance.

A status message similar to the following indicates a node has exceeded the configured low watermark and no shard will be allocated to this node:

Example output

  1.   nodes:
  2.   - conditions:
  3.     - lastTransitionTime: 2019-03-15T15:57:22Z
  4.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
  5.         be allocated on this node.
  6.       reason: Disk Watermark Low
  7.       status: "True"
  8.       type: NodeStorage
  9.     deploymentName: example-elasticsearch-clientdatamaster-0-1
  10.     upgradeStatus: {}

A status message similar to the following indicates a node has exceeded the configured high watermark and shards will be relocated to other nodes:

Example output

  1.   nodes:
  2.   - conditions:
  3.     - lastTransitionTime: 2019-03-15T16:04:45Z
  4.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
  5.         from this node.
  6.       reason: Disk Watermark High
  7.       status: "True"
  8.       type: NodeStorage
  9.     deploymentName: cluster-logging-operator
  10.     upgradeStatus: {}

A status message similar to the following indicates the Elasticsearch node selector in the CR does not match any nodes in the cluster:

Example output

  1.     Elasticsearch Status:
  2.       Shard Allocation Enabled:  shard allocation unknown
  3.       Cluster:
  4.         Active Primary Shards:  0
  5.         Active Shards:          0
  6.         Initializing Shards:    0
  7.         Num Data Nodes:         0
  8.         Num Nodes:              0
  9.         Pending Tasks:          0
  10.         Relocating Shards:      0
  11.         Status:                 cluster health unknown
  12.         Unassigned Shards:      0
  13.       Cluster Name:             elasticsearch
  14.       Node Conditions:
  15.         elasticsearch-cdm-mkkdys93-1:
  16.           Last Transition Time:  2019-06-26T03:37:32Z
  17.           Message:               0/5 nodes are available: 5 node(s) didn't match node selector.
  18.           Reason:                Unschedulable
  19.           Status:                True
  20.           Type:                  Unschedulable
  21.         elasticsearch-cdm-mkkdys93-2:
  22.       Node Count:  2
  23.       Pods:
  24.         Client:
  25.           Failed:
  26.           Not Ready:
  27.             elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49
  28.             elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl
  29.           Ready:
  30.         Data:
  31.           Failed:
  32.           Not Ready:
  33.             elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49
  34.             elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl
  35.           Ready:
  36.         Master:
  37.           Failed:
  38.           Not Ready:
  39.             elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49
  40.             elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl
  41.           Ready:

A status message similar to the following indicates that the requested PVC could not bind to PV:

Example output

  1.       Node Conditions:
  2.         elasticsearch-cdm-mkkdys93-1:
  3.           Last Transition Time:  2019-06-26T03:37:32Z
  4.           Message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
  5.           Reason:                Unschedulable
  6.           Status:                True
  7.           Type:                  Unschedulable

A status message similar to the following indicates that the Fluentd pods cannot be scheduled because the node selector did not match any nodes:

Example output

  1. Status:
  2.   Collection:
  3.     Logs:
  4.       Fluentd Status:
  5.         Daemon Set:  fluentd
  6.         Nodes:
  7.         Pods:
  8.           Failed:
  9.           Not Ready:
  10.           Ready:

Viewing the status of cluster logging components

You can view the status for a number of cluster logging components.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Change to the openshift-logging project.

    1. $ oc project openshift-logging

  2. View the status of the cluster logging environment:

    1. $ oc describe deployment cluster-logging-operator

    Example output

    1. Name:                   cluster-logging-operator
    3. ....
    5. Conditions:
    6.   Type           Status  Reason
    7.   ----           ------  ------
    8.   Available      True    MinimumReplicasAvailable
    9.   Progressing    True    NewReplicaSetAvailable
    11. ....
    13. Events:
    14.   Type    Reason             Age   From                   Message
    15.   ----    ------             ----  ----                   -------
    16.   Normal  ScalingReplicaSet  62m   deployment-controller  Scaled up replica set cluster-logging-operator-574b8987df to 1----

  3. View the status of the cluster logging replica set:

    1. Get the name of a replica set:

      Example output

      1. $ oc get replicaset

      Example output

      1. NAME                                      DESIRED   CURRENT   READY   AGE
      2. cluster-logging-operator-574b8987df       1         1         1       159m
      3. elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
      4. elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
      5. elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
      6. kibana-5bd5544f87                         1         1         1       157m

    2. Get the status of the replica set:

      1. $ oc describe replicaset cluster-logging-operator-574b8987df

      Example output

      1. Name:           cluster-logging-operator-574b8987df
      3. ....
      5. Replicas:       1 current / 1 desired
      6. Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
      8. ....
      10. Events:
      11.   Type    Reason            Age   From                   Message
      12.   ----    ------            ----  ----                   -------
      13.   Normal  SuccessfulCreate  66m   replicaset-controller  Created pod: cluster-logging-operator-574b8987df-qjhqv----