Monitor

You can use different ways to monitor a Pulsar cluster, exposing both metrics related to the usage of topics and the overall health of the individual components of the cluster.

Collect metrics

You can collect broker stats, ZooKeeper stats, and BookKeeper stats.

Broker stats

You can collect Pulsar broker metrics from brokers and export the metrics in JSON format. The Pulsar broker metrics mainly have two types:

  • Destination dumps, which contain stats for each individual topic. You can fetch the destination dumps using the command below:

    2. bin/pulsar-admin broker-stats destinations

  • Broker metrics, which contain the broker information and topics stats aggregated at namespace level. You can fetch the broker metrics by using the following command:

    2. bin/pulsar-admin broker-stats monitoring-metrics

All the message rates are updated every minute.

The aggregated broker metrics are also exposed in the Prometheus format at:

  2. http://$BROKER_ADDRESS:8080/metrics/

ZooKeeper stats

The local ZooKeeper, configuration store server and clients that are shipped with Pulsar can expose detailed stats through Prometheus.

  2. http://$LOCAL_ZK_SERVER:8000/metrics
  3. http://$GLOBAL_ZK_SERVER:8001/metrics

The default port of local ZooKeeper is 8000 and the default port of the configuration store is 8001. You can use a different stats port by configuring metricsProvider.httpPort in the conf/zookeeper.conf file.

BookKeeper stats

You can configure the stats frameworks for BookKeeper by modifying the statsProviderClass in the conf/bookkeeper.conf file.

The default BookKeeper configuration enables the Prometheus exporter. The configuration is included with Pulsar distribution.

  2. http://$BOOKIE_ADDRESS:8000/metrics

The default port for bookie is 8000. You can change the port by configuring prometheusStatsHttpPort in the conf/bookkeeper.conf file.

Managed cursor acknowledgment state

The acknowledgment state is persistent to the ledger first. When the acknowledgment state fails to be persistent to the ledger, they are persistent to ZooKeeper. To track the stats of acknowledgement, you can configure the metrics for the managed cursor.

  2. brk_ml_cursor_persistLedgerSucceed(namespace=", ledger_name="", cursor_name:")
  3. brk_ml_cursor_persistLedgerErrors(namespace="", ledger_name="", cursor_name:"")
  4. brk_ml_cursor_persistZookeeperSucceed(namespace="", ledger_name="", cursor_name:"")
  5. brk_ml_cursor_persistZookeeperErrors(namespace="", ledger_name="", cursor_name:"")
  6. brk_ml_cursor_nonContiguousDeletedMessagesRange(namespace="", ledger_name="", cursor_name:"")

Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.

Function and connector stats

You can collect functions worker stats from functions-worker and export the metrics in JSON formats, which contain functions worker JVM metrics.

  2. pulsar-admin functions-worker monitoring-metrics

You can collect functions and connectors metrics from functions-worker and export the metrics in JSON formats.

  2. pulsar-admin functions-worker function-stats

The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get FUNCTIONS_WORKER_ADDRESS and WORKER_PORT from the functions_worker.yml file.

  2. http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics:

Configure Prometheus

You can use Prometheus to collect all the metrics exposed for Pulsar components and set up Grafana dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to Prometheus guide.

When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to Kubernetes instructions.

Dashboards

When you collect time series statistics, the major problem is to make sure the number of dimensions attached to the data does not explode. Thus you only need to collect time series of metrics aggregated at the namespace level.

Pulsar per-topic dashboard

The per-topic dashboard instructions are available at Pulsar manager.

Grafana

You can use grafana to create dashboard driven by the data that is stored in Prometheus.

When you deploy Pulsar on Kubernetes with the Pulsar Helm Chart, a pulsar-grafana Docker image is enabled by default. You can use the docker image with the principal dashboards.

The following are some Grafana dashboards examples:

  • pulsar-grafana: a Grafana dashboard that displays metrics collected in Prometheus for Pulsar clusters running on Kubernetes.
  • apache-pulsar-grafana-dashboard: a collection of Grafana dashboard templates for different Pulsar components running on both Kubernetes and on-premise machines.

Alerting rules

You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to alerting rules.