Monitoring

There are different ways to monitor a Pulsar cluster, exposing both metrics relative to the usage of topics and the overall health of the individual components of the cluster.

Collecting metrics

Broker stats

Pulsar broker metrics can be collected from brokers and exported in JSON format. There are two main types of metrics:

  • Destination dumps, which containing stats for each individual topic. They can be fetched using
  1. bin/pulsar-admin broker-stats destinations
  • Broker metrics, containing broker info and topics stats aggregated at namespacelevel:
  1. bin/pulsar-admin broker-stats monitoring-metrics

All the message rates are updated every 1min.

The aggregated broker metrics are also exposed in the Prometheus format at:

  1. http://$BROKER_ADDRESS:8080/metrics

ZooKeeper stats

The local Zookeeper/configuration store server and clients that are shipped with Pulsar have been instrumented to exposedetailed stats through Prometheus as well.

  1. http://$LOCAL_ZK_SERVER:8000/metrics
  2. http://$GLOBAL_ZK_SERVER:8001/metrics

The default port of local ZooKeeper is 8000 and that of configuration store is 8001.These can be changed by specifying system property stats_server_port.

BookKeeper stats

For BookKeeper you can configure the stats frameworks by changing the statsProviderClass inconf/bookkeeper.conf.

By default, the default BookKeeper configuration included with Pulsar distribution will enablethe Prometheus exporter.

  1. http://$BOOKIE_ADDRESS:8000/metrics

For bookies, the default port is 8000 (instead of 8080) and that can be configured by changingthe prometheusStatsHttpPort in conf/bookkeeper.conf.

Configuring Prometheus

You can configure Prometheus to collect and store the metrics data by following the PrometheusGetting started guide.

When running on bare metal, you can provide the list of nodes that needs to be probed. When deployingin a Kubernetes cluster, the monitoring is automatically setup with the providedinstructions.

Dashboards

When collecting time series statistics, the major problem is to make sure the number of dimensionsattached to the data does not explode.

For that reason we only collect time series of metrics aggregated at the namespace level.

Pulsar per-topic dashboard

The per-topic dashboard instructions are available at Dashboard.

Grafana

You can use grafana to easily create dashboard driven by the data stored in Prometheus.

There is a pulsar-grafana Docker image that is ready to use with the principal dashboards alreadyin place. This is enabled by default when deploying Pulsar on Kubernetes.

To use the dashboard manually:

  1. docker run -p3000:3000 \
  2.         -e PROMETHEUS_URL=http://$PROMETHEUS_HOST:9090/ \
  3.         apachepulsar/pulsar-grafana:latest