JMX monitoring

The JMX monitoring feature exposes query metrics via the JMX API.

Table of contents

• Setup

  • Enable collecting stats

  • Enable the JMX API

• Using Docker

• JMX Beans

  • QueryStats MBean

  • NodeStatus MBean

  • NodeInfo MXBean

  • Connections MBean

  • ThreadPools MXBean

  • CircuitBreakers MXBean

• Exposing JMX via HTTP

Setup

Enable collecting stats

By default, Collecting stats is enabled. You can disable collecting stats via the CrateDB configuration file or by running this statement:

cr> SET GLOBAL "stats.enabled" = FALSE;

Enable the JMX API

To monitor CrateDB using the JMX API, you must set the following system properties before you start CrateDB:

com.sun.management.jmxremote
com.sun.management.jmxremote.port=<JMX_PORT>
com.sun.management.jmxremote.ssl=false
com.sun.management.jmxremote.authenticate=false

Here, <JMX_PORT> sets the port number of your JMX server. JMX SSL and authentication are currently not supported.

More information about the JMX monitoring properties can be found in the JMX documentation.

You can set the Java system properties with the -D option:

sh$ ./bin/crate -Dcom.sun.management.jmxremote \
...             -Dcom.sun.management.jmxremote.port=7979 \
...             -Dcom.sun.management.jmxremote.ssl=false \
...             -Dcom.sun.management.jmxremote.authenticate=false

However, the recommended way to set system properties is via the CRATE_JAVA_OPTS environment variable, like so:

sh$ export CRATE_JAVA_OPTS="$CRATE_JAVA_OPTS \
      -Dcom.sun.management.jmxremote \
      -Dcom.sun.management.jmxremote.port=7979 \
      -Dcom.sun.management.jmxremote.ssl=false \
      -Dcom.sun.management.jmxremote.authenticate=false"
sh$ ./bin/crate

If you’re using the CrateDB Debian or RPM packages, you can set this environment variable via the /etc/default/crate configuration file.

Using Docker

To enable JMX monitoring when running CrateDB in a Docker container you have to set the following additional Java system properties:

-Djava.rmi.server.hostname=<RMI_HOSTNAME>
-Dcom.sun.management.jmxremote.rmi.port=<RMI_PORT>

Here, <RMI_HOSTNAME> is the IP address or hostname of the Docker host and <RMI_PORT> is the statically assigned port of the RMI server. For convenience, <RMI_PORT> can be set to the same port the JMX server listens on.

The <RMI_HOSTNAME> and <RMI_PORT> can be used by JMX clients (e.g. JConsole or VisualVM) to connect to the JMX server.

Here’s an example Docker command:

sh> docker run -d -e CRATE_JAVA_OPTS="\
      -Dcom.sun.management.jmxremote
      -Dcom.sun.management.jmxremote.port=7979 \
      -Dcom.sun.management.jmxremote.ssl=false \
      -Dcom.sun.management.jmxremote.authenticate=false \
      -Dcom.sun.management.jmxremote.rmi.port=7979 \
      -Djava.rmi.server.hostname=<RMI_HOSTNAME>" \
      -p 7979:7979 crate -Cnetwork.host=_site_

Here, again, <RMI_HOSTNAME> is the IP address or hostname of the Docker host.

JMX Beans

QueryStats MBean

The QueryStats MBean exposes the sum of durations, in milliseconds, total and failed count of all statements executed since the node was started, grouped by type, for SELECT, UPDATE, DELETE, INSERT, MANAGEMENT, DDL, COPY and UNDEFINED queries.

Metrics can be accessed using the JMX MBean object name io.crate.monitoring:type=QueryStats and the following attributes:

Statements total count since the node was started:

• SelectQueryTotalCount

• InsertQueryTotalCount

• UpdateQueryTotalCount

• DeleteQueryTotalCount

• ManagementQueryTotalCount

• DDLQueryTotalCount

• CopyQueryTotalCount

• UndefinedQueryTotalCount

Statements failed count since the node was started:

• SelectQueryFailedCount

• InsertQueryFailedCount

• UpdateQueryFailedCount

• DeleteQueryFailedCount

• ManagementQueryFailedCount

• DDLQueryFailedCount

• CopyQueryFailedCount

• UndefinedQueryFailedCount

The sum of the durations, in milliseconds, since the node was started, of all statement executions grouped by type:

• SelectQuerySumOfDurations

• InsertQuerySumOfDurations

• UpdateQuerySumOfDurations

• DeleteQuerySumOfDurations

• ManagementQuerySumOfDurations

• DDLQuerySumOfDurations

• CopyQuerySumOfDurations

• UndefinedQuerySumOfDurations

NodeStatus MBean

The NodeStatus JMX MBean exposes the status of the current node as boolean values.

NodeStatus can be accessed using the JMX MBean object name io.crate.monitoring:type=NodeStatus and the following attributes:

• Ready

  Defines if the node is able to process SQL statements.

NodeInfo MXBean

The NodeInfo JMX MXBean exposes information about the current node.

NodeInfo can be accessed using the JMX MXBean object name io.crate.monitoring:type=NodeInfo and the following attributes:

ShardStats returns a CompositeData object containing statistics about the number of shards located on the node with the following attributes:

ShardInfo returns an Array of CompositeData objects containing detailed information about the shards located on the node with the following attributes:

Connections MBean

The Connections MBean exposes information about any open connections to a CrateDB node.

It can be accessed using the io.crate.monitoring:type=Connections object name and has the following attributes:

ThreadPools MXBean

The ThreadPools MXBean exposes statistical information about the used thread pools of a CrateDB node.

It can be accessed using the io.crate.monitoring:type=ThreadPools object name and has following attributes:

Each of them returns a CompositeData object containing detailed statistics of each thread pool with the following attributes:

CircuitBreakers MXBean

The CircuitBreaker MXBean exposes statistical information about all availabe circuit breakers of a CrateDB node.

It can be accessed using the io.crate.monitoring:type=CircuitBreakers object name and has following attributes:

Each of them returns a CompositeData object containing detailed statistics of each circuit breaker with the following attributes:

Exposing JMX via HTTP

The JMX metrics and a readiness endpoint can be exposed via HTTP (e.g. to be used by Prometheus) by using the Crate JMX HTTP Exporter Java agent. See the README in the Crate JMX HTTP Exporter repository for more information.