Observability

Ozone provides multiple tools to get more information about the current state of the cluster.

Prometheus

Ozone has native support for Prometheus integration. All internal metrics (collected by Hadoop metrics framework) are published under the /prom HTTP endpoint. (For example under http://localhost:9876/prom for SCM).

The Prometheus endpoint is turned on by default but can be turned off by the hdds.prometheus.endpoint.enabled configuration variable.

In a secure environment the page is guarded with SPNEGO authentication which is not supported by Prometheus. To enable monitoring in a secure environment, a specific authentication token can be configured

Example ozone-site.xml:

  1. <property>
  2.    <name>hdds.prometheus.endpoint.token</name>
  3.    <value>putyourtokenhere</value>
  4. </property>

Example prometheus configuration:

  1. scrape_configs:
  2.   - job_name: ozone
  3.     bearer_token: <putyourtokenhere>
  4.     metrics_path: /prom
  5.     static_configs:
  6.      - targets:
  7.          - "127.0.0.1:9876"

Grafana

Once Prometheus is up and running, Grana can be configured to monitor and visualize Ozone metrics.

Add Prometheus as a data source

In the Grafana web UI, go to Add Data Sources and then select Prometheus.

Enter the Prometheus hostname/port in the HTTP. For example, http://localhost:9094 (verify the port used by looking at Prometheus command line flags -web.listen-address. The port can also be found from Prometheus web UI → Status → Command-Line Flags.)

Choose Prometheus type: Prometheus

Choose Prometheus version: 2.37.x

Finish the setup by clicking on Save and Test.

Import a Grafana dashboard for Ozone

Apache Ozone comes with a default Grafana dashboard. Follow the instructions below to import it:

Download dashboard json:

  1. wget https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone%20-%20Overall%20Metrics.json

Open Grafana portal and click on Dashboards on the left and select Import.

Click at Upload JSON file and select the file Ozone - Overall Metrics.json that was just downloaded.

The dashboard is now imported.

Overall dashboard

Repeat the same for Object Metrics dashboard and RPC Metrics dashboard.

Object dashboard

RPC dashboard

Distributed tracing

Distributed tracing can help to understand performance bottleneck with visualizing end-to-end performance.

Ozone uses jaeger tracing library to collect traces which can send tracing data to any compatible backend (Zipkin, …).

Tracing is turned off by default, but can be turned on with hdds.tracing.enabled from ozone-site.xml

  1. <property>
  2.    <name>hdds.tracing.enabled</name>
  3.    <value>true</value>
  4. </property>

Jaeger client can be configured with environment variables as documented here:

For example:

  1. JAEGER_SAMPLER_PARAM=0.01
  2. JAEGER_SAMPLER_TYPE=probabilistic
  3. JAEGER_AGENT_HOST=jaeger

This configuration will record 1% of the requests to limit the performance overhead. For more information about jaeger sampling check the documentation

ozone insight

Ozone insight is a swiss-army-knife tool to for checking the current state of Ozone cluster. It can show logging, metrics and configuration for a particular component.

To check the available components use ozone insight list:

  1. > ozone insight list
  3. Available insight points:
  5.   scm.node-manager                     SCM Datanode management related information.
  6.   scm.replica-manager                  SCM closed container replication manager
  7.   scm.event-queue                      Information about the internal async event delivery
  8.   scm.protocol.block-location          SCM Block location protocol endpoint
  9.   scm.protocol.container-location      SCM Container location protocol endpoint
  10.   scm.protocol.security                SCM Block location protocol endpoint
  11.   om.key-manager                       OM Key Manager
  12.   om.protocol.client                   Ozone Manager RPC endpoint
  13.   datanode.pipeline                    More information about one ratis datanode ring.

Configuration

ozone insight config can show configuration related to a specific component (supported only for selected components).

  1. > ozone insight config scm.replica-manager
  3. Configuration for `scm.replica-manager` (SCM closed container replication manager)
  5. >>> hdds.scm.replication.thread.interval
  6.        default: 300s
  7.        current: 300s
  9. There is a replication monitor thread running inside SCM which takes care of replicating the containers in the cluster. This property is used to configure the interval in which that thread runs.
  12. >>> hdds.scm.replication.event.timeout
  13.        default: 30m
  14.        current: 30m
  16. Timeout for the container replication/deletion commands sent  to datanodes. After this timeout the command will be retried.

Metrics

ozone insight metrics can show metrics related to a specific component (supported only for selected components).

  1. > ozone insight metrics scm.protocol.block-location
  2. Metrics for `scm.protocol.block-location` (SCM Block location protocol endpoint)
  4. RPC connections
  6.   Open connections: 0
  7.   Dropped connections: 0
  8.   Received bytes: 1267
  9.   Sent bytes: 2420
  12. RPC queue
  14.   RPC average queue time: 0.0
  15.   RPC call queue length: 0
  18. RPC performance
  20.   RPC processing time average: 0.0
  21.   Number of slow calls: 0
  24. Message type counters
  26.   Number of AllocateScmBlock: ???
  27.   Number of DeleteScmKeyBlocks: ???
  28.   Number of GetScmInfo: ???
  29.   Number of SortDatanodes: ???

Logs

ozone insight logs can connect to the required service and show the DEBUG/TRACE log related to one specific component. For example to display RPC message:

  1. >ozone insight logs om.protocol.client
  3. [OM] 2020-07-28 12:31:49,988 [DEBUG|org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB|OzoneProtocolMessageDispatcher] OzoneProtocol ServiceList request is received
  4. [OM] 2020-07-28 12:31:50,095 [DEBUG|org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB|OzoneProtocolMessageDispatcher] OzoneProtocol CreateVolume request is received

Using -v flag the content of the protobuf message can also be displayed (TRACE level log):

  1. ozone insight logs -v om.protocol.client
  3. [OM] 2020-07-28 12:33:28,463 [TRACE|org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB|OzoneProtocolMessageDispatcher] [service=OzoneProtocol] [type=CreateVolume] request is received:
  4. cmdType: CreateVolume
  5. traceID: ""
  6. clientId: "client-A31DF5C6ECF2"
  7. createVolumeRequest {
  8.   volumeInfo {
  9.     adminName: "hadoop"
  10.     ownerName: "hadoop"
  11.     volume: "vol1"
  12.     quotaInBytes: 1152921504606846976
  13.     volumeAcls {
  14.       type: USER
  15.       name: "hadoop"
  16.       rights: "200"
  17.       aclScope: ACCESS
  18.     }
  19.     volumeAcls {
  20.       type: GROUP
  21.       name: "users"
  22.       rights: "200"
  23.       aclScope: ACCESS
  24.     }
  25.     creationTime: 1595939608460
  26.     objectID: 0
  27.     updateID: 0
  28.     modificationTime: 0
  29.   }
  30. }
  32. [OM] 2020-07-28 12:33:28,474 [TRACE|org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB|OzoneProtocolMessageDispatcher] [service=OzoneProtocol] [type=CreateVolume] request is processed. Response:
  33. cmdType: CreateVolume
  34. traceID: ""
  35. success: false
  36. message: "Volume already exists"
  37. status: VOLUME_ALREADY_EXISTS

Under the hood ozone insight uses HTTP endpoints to retrieve the required information (/conf, /prom and /logLevel endpoints). It’s not yet supported in secure environment.