Along with IoTDB running, we hope to observe the status of IoTDB, so as to troubleshoot system problems or discover potential system risks in time. A series of metrics that can reflect the operating status of the system are system monitoring metrics.

1. When to use metric framework?

Belows are some typical application scenarios

1. System is running slowly

   When system is running slowly, we always hope to have information about system’s running status as detail as possible, such as:

   • JVM：Is there FGC? How long does it cost? How much does the memory usage decreased after GC? Are there lots of threads?
   • System：Is the CPU usage too hi？Are there many disk IOs？
   • Connections：How many connections are there in the current time？
   • Interface：What is the TPS and latency of every interface？
   • Thread Pool：Are there many pending tasks？
   • Cache Hit Ratio

2. No space left on device

   When meet a “no space left on device” error, we really want to know which kind of data file had a rapid rise in the past hours.

3. Is the system running in abnormal status

   We could use the count of error logs、the alive status of nodes in cluster, etc, to determine whether the system is running abnormally.

2. Who will use metric framework?

Any person cares about the system’s status, including but not limited to RD, QA, SRE, DBA, can use the metrics to work more efficiently.

3. What is metrics?

3.1. Key Concept

In IoTDB’s metric module, each metrics is uniquely identified by Metric Name and Tags.

• Metric Name: Metric type name, such as logback_events means log events.
• Tags: indicator classification, in the form of Key-Value pairs, each indicator can have 0 or more categories, common Key-Value pairs:
  • name = xxx: The name of the monitored object, which is the description of business logic. For example, for a monitoring item of type Metric Name = entry_seconds_count, the meaning of name refers to the monitored business interface.
  • type = xxx: Monitoring indicator type subdivision, which is a description of monitoring indicator itself. For example, for monitoring items of type Metric Name = point, the meaning of type refers to the specific type of monitoring points.
  • status = xxx: The status of the monitored object is a description of business logic. For example, for monitoring items of type Metric Name = Task, this parameter can be used to distinguish the status of the monitored object.
  • user = xxx: The relevant user of the monitored object is a description of business logic. For example, count the total points written by the root user.
  • Customize according to the specific situation: For example, there is a level classification under logback_events_total, which is used to indicate the number of logs under a specific level.
• Metric Level: The level of metric managing level, The default startup level is Core level, the recommended startup level is Important level, and the audit strictness is Core > Important > Normal > All
  • Core: Core metrics of the system, used by the operation and maintenance personnel, which is related to the * performance, stability, and security* of the system, such as the status of the instance, the load of the system, etc.
  • Important: Important metrics of the module, which is used by operation and maintenance and testers, and is directly related to the running status of each module, such as the number of merged files, execution status, etc.
  • Normal: Normal metrics of the module, used by developers to facilitate locating the module when problems occur, such as specific key operation situations in the merger.
  • All: All metrics of the module, used by module developers, often used when the problem is reproduced, so as to solve the problem quickly.

3.2. External data format for metrics

• IoTDB provides metrics in JMX, Prometheus and IoTDB formats:
  • For JMX, metrics can be obtained through org.apache.iotdb.metrics.
  • For Prometheus, the value of the metrics can be obtained through the externally exposed port
  • External exposure in IoTDB mode: metrics can be obtained by executing IoTDB queries

4. The detail of metrics

Currently, IoTDB provides metrics for some main modules externally, and with the development of new functions and system optimization or refactoring, metrics will be added and updated synchronously.

If you want to add your own metrics data in IoTDB, please see the [IoTDB Metric Framework] (https://github.com/apache/iotdb/tree/master/metricsopen in new window) document.

4.1. Core level metrics

Core-level metrics are enabled by default during system operation. The addition of each Core-level metrics needs to be carefully evaluated. The current Core-level metrics are as follows:

4.1.1. Cluster

4.1.2. Interface

4.1.2. Node Statistics

4.1.3. Cluster Tracing

4.1.5. Task Statistics

4.1.6. IoTDB process

4.1.7. System

4.1.8. Log

4.1.9. File

4.1.10. JVM Memory

4.1.11. JVM Thread

4.1.12. JVM GC

4.2. Important level metrics

4.2.1. Node

4.2.2. RatisConsensus

4.2.3. IoTConsensus

4.2.4. Cache

4.2.5. Memory

4.2.6. Compaction

4.2.7. IoTDB Process

4.2.8. JVM Class

4.2.9. JVM Compilation

4.2.10. Query Planning

4.2.11. Plan Dispatcher

4.2.12. Query Resource

4.2.13. Data Exchange

4.2.14. Query Task Schedule

4.2.15. Query Execution

4.2.16 Schema Engine

4.2.17 Write Performance

4.3. Normal level Metrics

4.3.1. Cluster

4.4. All Metric

Currently there is no All level metrics, and it will continue to be added in the future.

5. How to get these metrics？

The relevant configuration of the metric module is in conf/iotdb-{datanode/confignode}.properties, and all configuration items support hot loading through the load configuration command.

5.1. JMX

For metrics exposed externally using JMX, you can view them through Jconsole. After entering the Jconsole monitoring page, you will first see an overview of various running conditions of IoTDB. Here you can see heap memory information, thread information, class information, and the server’s CPU usage.

5.1.1. Obtain metric data

After connecting to JMX, you can find the “MBean” named “org.apache.iotdb.metrics” through the “MBeans” tab, and you can view the specific values of all monitoring metrics in the sidebar.

5.1.2. Get other relevant data

After connecting to JMX, you can find the “MBean” named “org.apache.iotdb.service” through the “MBeans” tab, as shown in the image below, to understand the basic status of the service

In order to improve query performance, IOTDB caches ChunkMetaData and TsFileMetaData. Users can use MXBean and expand the sidebar org.apache.iotdb.db.service to view the cache hit ratio:

5.2. Prometheus

5.2.1. The mapping from metric type to prometheus format

For metrics whose Metric Name is name and Tags are K1=V1, …, Kn=Vn, the mapping is as follows, where value is a specific value

5.2.2. Config File

1. Taking DataNode as an example, modify the iotdb-datanode.properties configuration file as follows:

dn_metric_reporter_list=PROMETHEUS
dn_metric_level=CORE
dn_metric_prometheus_reporter_port=9091

Then you can get metrics data as follows

1. Start IoTDB DataNodes
2. Open a browser or use curl to visit http://servier_ip:9091/metrics, you can get the following metric data:

...
# HELP file_count
# TYPE file_count gauge
file_count{name="wal",} 0.0
file_count{name="unseq",} 0.0
file_count{name="seq",} 2.0
...

5.2.3. Prometheus + Grafana

As shown above, IoTDB exposes monitoring metrics data in the standard Prometheus format to the outside world. Prometheus can be used to collect and store monitoring indicators, and Grafana can be used to visualize monitoring indicators.

The following picture describes the relationships among IoTDB, Prometheus and Grafana

iotdb_prometheus_grafana

1. Along with running, IoTDB will collect its metrics continuously.
2. Prometheus scrapes metrics from IoTDB at a constant interval (can be configured).
3. Prometheus saves these metrics to its inner TSDB.
4. Grafana queries metrics from Prometheus at a constant interval (can be configured) and then presents them on the graph.

So, we need to do some additional works to configure and deploy Prometheus and Grafana.

For instance, you can config your Prometheus as follows to get metrics data from IoTDB:

job_name: pull-metrics
honor_labels: true
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
static_configs:
  - targets:
      - localhost:9091

The following documents may help you have a good journey with Prometheus and Grafana.

Prometheus getting_startedopen in new window

Prometheus scrape metricsopen in new window

Grafana getting_startedopen in new window

Grafana query metrics from Prometheusopen in new window

5.2.4. Apache IoTDB Dashboard

We provide the Apache IoTDB Dashboard, and the rendering shown in Grafana is as follows:

Apache IoTDB Dashboard

You can obtain the json files of Dashboards in enterprise version.

5.3. IoTDB

5.3.1. IoTDB mapping relationship of metrics

For metrics whose Metric Name is name and Tags are K1=V1, …, Kn=Vn, the mapping is as follows, taking root.__ system.metric.clusterName.nodeType.nodeId as an example by default

5.3.2. Obtain metrics

According to the above mapping relationship, related IoTDB query statements can be formed to obtain metrics