Monitoring Debezium

Metrics for monitoring Debezium connectors

In addition to the built-in support for JMX metrics in Kafka, Zookeeper, and Kafka Connect, each connector provides additional metrics that you can use to monitor their activities.

• Db2 connector metrics

• MongoDB connector metrics

• MySQL connector metrics

• Oracle connector metrics

• PostgreSQL connector metrics

• SQL Server connector metrics

• Cassandra connector metrics

• Vitess connector metrics

• Spanner connector metrics

• Informix connector metrics

Enabling JMX in local installations

With Zookeeper, Kafka, and Kafka Connect, you enable JMX by setting the appropriate environment variables when you start each service.

Zookeeper JMX environment variables

Zookeeper has built-in support for JMX. When running Zookeeper using a local installation, the zkServer.sh script recognizes the following environment variables:

JMXPORT

Enables JMX and specifies the port number that will be used for JMX. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.port=$JMXPORT.

JMXAUTH

Whether JMX clients must use password authentication when connecting. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

Whether JMX clients connect using SSL/TLS. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

JMXLOG4J

Whether the Log4J JMX MBeans should be disabled. Must be either true (default) or false. The default is true. The value is used to specify the JVM parameter -Dzookeeper.jmx.log4j.disable=$JMXLOG4J.

Kafka JMX environment variables

When running Kafka using a local installation, the kafka-server-start.sh script recognizes the following environment variables:

JMX_PORT

Enables JMX and specifies the port number that will be used for JMX. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.port=$JMX_PORT.

KAFKA_JMX_OPTS

The JMX options, which are passed directly to the JVM during startup. The default options are:

• -Dcom.sun.management.jmxremote

• -Dcom.sun.management.jmxremote.authenticate=false

• -Dcom.sun.management.jmxremote.ssl=false

Kafka Connect JMX environment variables

When running Kafka using a local installation, the connect-distributed.sh script recognizes the following environment variables:

JMX_PORT

Enables JMX and specifies the port number that will be used for JMX. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.port=$JMX_PORT.

KAFKA_JMX_OPTS

The JMX options, which are passed directly to the JVM during startup. The default options are:

• -Dcom.sun.management.jmxremote

• -Dcom.sun.management.jmxremote.authenticate=false

• -Dcom.sun.management.jmxremote.ssl=false

Enabling JMX in Docker

If you are running Apache Zookeeper, Kafka, and Kafka Connect in via the Debezium container images, enabling JMX requires several additional environment variables that are not typically needed when running on a local machine. This is because the JVM requires the host name to which it will advertise itself to JMX clients. Thus, Debezium’s container images for Zookeeper, Kafka, and Kafka Connect use several environment variables to enable and configure JMX. Most of the environment variables are the same for all of the images, but there are some minor differences.

Zookeeper JMX environment variables in Docker

The debezium/zookeeper image recognizes the following JMX-related environment variables:

JMXPORT (Required)

The port number that will be used for JMX. The value is used to specify the following JVM parameters:

• -Dcom.sun.management.jmxremote.port=$JMXPORT

• -Dcom.sun.management.jmxremote.rmi.port=$JMXPORT

JMXHOST (Required)

The IP address or resolvable host name of the Docker host, which JMX uses to construct a URL sent to the JMX client. A value of localhost or 127.0.0.1 will not work. Typically, 0.0.0.0 can be used. The value is used to specify the JVM parameter -Djava.rmi.server.hostname=$JMXHOST.

JMXAUTH

Whether JMX clients must use password authentication when connecting. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

Whether JMX clients connect using SSL/TLS. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

JMXLOG4J

Whether the Log4J JMX MBeans should be disabled. Must be either true or false. The default is true. The value is used to specify the JVM parameter -Dzookeeper.jmx.log4j.disable=$JMXLOG4J.

Example 1. Enabling JMX in a Zookeeper Docker container

The following example Docker command starts a container using the debezium/zookeeper image with values for the JMXPORT and JMXHOST environment variables, and maps the Docker host’s port 9010 to the container’s JMX port:

$ docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 -p 9010:9010 -e JMXPORT=9010 -e JMXHOST=10.0.1.10 quay.io/debezium/zookeeper:latest

Kafka JMX environment variables in Docker

The debezium/kafka image recognizes the following JMX-related environment variables:

JMXPORT (Required)

The port number that will be used for JMX. The value is used to specify the following JVM parameters:

• -Dcom.sun.management.jmxremote.port=$JMXPORT

• -Dcom.sun.management.jmxremote.rmi.port=$JMXPORT

JMXHOST (Required)

The IP address or resolvable host name of the Docker host, which JMX uses to construct a URL sent to the JMX client. A value of localhost or 127.0.0.1 will not work. Typically, 0.0.0.0 can be used. The value is used to specify the JVM parameter -Djava.rmi.server.hostname=$JMXHOST.

JMXAUTH

Whether JMX clients must use password authentication when connecting. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

Whether JMX clients connect using SSL/TLS. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

Example 2. Enabling JMX in a Kafka Docker container

The following example Docker command starts a container using the debezium/kafka image with values for the JMXPORT and HOST_NAME environment variables, and maps the Docker host’s port 9011 to the container’s JMX port:

$ docker run -it --rm --name kafka -p 9092:9092 -p 9011:9011 -e JMXPORT=9011 -e JMXHOST=10.0.1.10 --link zookeeper:zookeeper quay.io/debezium/kafka:latest

Kafka Connect JMX environment variables in Docker

The debezium/connect image recognizes the following JMX-related environment variables:

JMXPORT (Required)

The port number that will be used for JMX. The value is used to specify the following JVM parameters:

• -Dcom.sun.management.jmxremote.port=$JMXPORT

• -Dcom.sun.management.jmxremote.rmi.port=$JMXPORT

JMXHOST (Required)

The IP address or resolvable host name of the Docker host, which JMX uses to construct a URL sent to the JMX client. A value of localhost or 127.0.0.1 will not work. Typically, 0.0.0.0 can be used. The value is used to specify the JVM parameter -Djava.rmi.server.hostname=$JMXHOST.

JMXAUTH

Whether JMX clients must use password authentication when connecting. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

Whether JMX clients connect using SSL/TLS. Must be either true or false. The default is false. The value is used to specify the JVM parameter -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

Example 3. Enabling JMX in a Kafka Connect Docker container

The following example Docker command starts a container using the debezium/connect image with values for the JMXPORT and JMXHOST environment variables, and maps the Docker host’s port 9012 to the container’s JMX port:

$ docker run -it --rm --name connect \
  -p 8083:8083 -p 9012:9012 \
  -e JMXPORT=9012 -e JMXHOST=10.0.1.10 \
  -e GROUP_ID=1 \
  -e CONFIG_STORAGE_TOPIC=my_connect_configs \
  -e OFFSET_STORAGE_TOPIC=my_connect_offsets \
  -e STATUS_STORAGE_TOPIC=my_connect_statuses \
  --link zookeeper:zookeeper \
  --link kafka:kafka \
  --link mysql:mysql \
  quay.io/debezium/connect:latest

Using Prometheus and Grafana

The metrics exposed by Debezium and Kafka can be exported and displayed with Prometheus and Grafana. You can find an example for the required configuration and example dashboards for different connectors in the Debezium examples repository.

Java Flight Recording

The Debezium container image for Kafka Connect also provides easy-to-use support for JDK Flight Recorder. To enable the recording it is necessary to set the environment variable ENABLE_JFR=true. The recording will then be started automatically upon container start. By default, the recording is enabled, i.e. a continuous recording with default settings is done, which you then can dump using jcmd or JDK Mission Control.

It is possible to modify the recorder’s behaviour. For that, you need to set either environment variables prefixed with JFR_RECORDING_ to set recording parameters. The part after the prefix is converted to lowercase and underscores are replaced with dashes so JFR_RECORDING_PATH_TO_GC_ROOTS=true becomes path-to-gc-roots=true.

Flight Recorder control options can be configured via JFR_OPT_ prefixed environment variables.