Pulsar Metrics

Pulsar exposes the following metrics in Prometheus format. You can monitor your clusters with those metrics.

• ZooKeeper
• BookKeeper
• Broker
• Pulsar Functions
• Proxy
• Pulsar SQL Worker
• Pulsar transaction

The following types of metrics are available:

• Counter: a cumulative metric that represents a single monotonically increasing counter. The value increases by default. You can reset the value to zero or restart your cluster.
• Gauge: a metric that represents a single numerical value that can arbitrarily go up and down.
• Histogram: a histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. The _bucket suffix is the number of observations within a histogram bucket, configured with parameter {le="<upper inclusive bound>"}. The _count suffix is the number of observations, shown as a time series and behaves like a counter. The _sum suffix is the sum of observed values, also shown as a time series and behaves like a counter. These suffixes are together denoted by _* in this doc.
• Summary: similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

ZooKeeper

The ZooKeeper metrics are exposed under “/metrics” at port 8000. You can use a different port by configuring the metricsProvider.httpPort in conf/zookeeper.conf.

ZooKeeper provides a New Metrics System since 3.6.0, more detailed metrics can refer ZooKeeper Monitor Guide.

BookKeeper

The BookKeeper metrics are exposed under “/metrics” at port 8000. You can change the port by updating prometheusStatsHttpPort in the bookkeeper.conf configuration file.

Server metrics

Journal metrics

Storage metrics

Broker

The broker metrics are exposed under “/metrics” at port 8080. You can change the port by updating webServicePort to a different port in the broker.conf configuration file.

All the metrics exposed by a broker are labelled with cluster=${pulsar_cluster}. The name of Pulsar cluster is the value of ${pulsar_cluster}, which you have configured in the broker.conf file.

The following metrics are available for broker:

• ZooKeeper
  • Server metrics
  • Request metrics
• BookKeeper
  • Server metrics
  • Journal metrics
  • Storage metrics
• Broker
  • Namespace metrics
  • Replication metrics
  • Topic metrics
  • Replication metrics
  • ManagedLedgerCache metrics
  • ManagedLedger metrics
  • LoadBalancing metrics
  • BundleUnloading metrics
  • BundleSplit metrics
  • Subscription metrics
  • Consumer metrics
  • Managed ledger bookie client metrics
  • Token metrics
  • Authentication metrics
  • Connection metrics
  • Jetty metrics
• Pulsar Functions
• Proxy
• Pulsar SQL Worker
• Pulsar transaction

BookKeeper client metrics

All the BookKeeper client metric are labelled with the following label:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.

Namespace metrics

Namespace metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to false.

All the namespace metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.

Replication metrics

If a namespace is configured to be replicated among multiple Pulsar clusters, the corresponding replication metrics is also exposed when replicationMetricsEnabled is enabled.

All the replication metrics are also labelled with remoteCluster=${pulsar_remote_cluster}.

Topic metrics

Topic metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to true.

All the topic metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
• topic: topic=${pulsar_topic}. ${pulsar_topic} is the topic name.

Replication metrics

If a namespace that a topic belongs to is configured to be replicated among multiple Pulsar clusters, the corresponding replication metrics is also exposed when replicationMetricsEnabled is enabled.

All the replication metrics are labelled with remoteCluster=${pulsar_remote_cluster}.

Topic lookup metrics

ManagedLedgerCache metrics

All the ManagedLedgerCache metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.

ManagedLedger metrics

All the managedLedger metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
• quantile: quantile=${quantile}. Quantile is only for Histogram type metric, and represents the threshold for given Buckets.

Managed cursor acknowledgment state

The acknowledgment state is persistent to the ledger first. When the acknowledgment state fails to be persistent to the ledger, they are persistent to ZooKeeper. To track the stats of acknowledgment, you can configure the metrics for the managed cursor.

All the cursor acknowledgment state metrics are labelled with the following labels:

• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.

• ledger_name: ledger_name=${pulsar_ledger_name}. ${pulsar_ledger_name} is the ledger name.

• cursor_name: ledger_name=${pulsar_cursor_name}. ${pulsar_cursor_name} is the cursor name.

LoadBalancing metrics

All the loadbalancing metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• broker: broker=${broker}. ${broker} is the IP address of the broker
• metric: metric=”loadBalancing”.

BundleUnloading metrics

All the bundleUnloading metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• metric: metric=”bundleUnloading”.

BundleSplit metrics

All the bundleUnloading metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• metric: metric=”bundlesSplit”.

Bundle metrics

All the bundle metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• broker: broker=${broker}. ${broker} is the IP address of the broker
• bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker
• metric: metric=”bundle”.

Subscription metrics

Subscription metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to true.

All the subscription metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
• topic: topic=${pulsar_topic}. ${pulsar_topic} is the topic name.
• subscription: subscription=${subscription}. ${subscription} is the topic subscription name.

Consumer metrics

Consumer metrics are only exposed when both exposeTopicLevelMetricsInPrometheus and exposeConsumerLevelMetricsInPrometheus are set to true.

All the consumer metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
• topic: topic=${pulsar_topic}. ${pulsar_topic} is the topic name.
• subscription: subscription=${subscription}. ${subscription} is the topic subscription name.
• consumer_name: consumer_name=${consumer_name}. ${consumer_name} is the topic consumer name.
• consumer_id: consumer_id=${consumer_id}. ${consumer_id} is the topic consumer id.

Managed ledger bookie client metrics

All the managed ledger bookie client metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.

Token metrics

All the token metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.

Authentication metrics

All the authentication metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• provider_name: provider_name=${provider_name}. ${provider_name} is the class name of the authentication provider.
• auth_method: auth_method=${auth_method}. ${auth_method} is the authentication method of the authentication provider.
• reason: reason=${reason}. ${reason} is the reason for failing authentication operation. (This label is only for pulsar_authentication_failures_count.)

Connection metrics

All the connection metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• broker: broker=${advertised_address}. ${advertised_address} is the advertised address of the broker.
• metric: metric=${metric}. ${metric} is the connection metric collective name.

Jetty metrics

For a functions-worker running separately from brokers, its Jetty metrics are only exposed when includeStandardPrometheusMetrics is set to true.

All the jetty metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.

Pulsar Functions

All the Pulsar Functions metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.

Connectors

All the Pulsar connector metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.

Connector metrics contain source metrics and sink metrics.

• Source metrics

  Name	Type	Description
  pulsar_source_written_total	Counter	The total number of records written to a Pulsar topic.
  pulsar_source_written_total_1min	Counter	The total number of records written to a Pulsar topic in the last 1 minute.
  pulsar_source_received_total	Counter	The total number of records received from source.
  pulsar_source_received_total_1min	Counter	The total number of records received from source in the last 1 minute.
  pulsar_source_last_invocation	Gauge	The timestamp of the last invocation of the source.
  pulsar_source_source_exception	Gauge	The exception from a source.
  pulsar_source_source_exceptions_total	Counter	The total number of source exceptions.
  pulsar_source_source_exceptions_total_1min	Counter	The total number of source exceptions in the last 1 minute.
  pulsar_source_system_exception	Gauge	The exception from system code.
  pulsar_source_system_exceptions_total	Counter	The total number of system exceptions.
  pulsar_source_system_exceptions_total_1min	Counter	The total number of system exceptions in the last 1 minute.
  pulsarsource_user_metric	Summary	The user-defined metrics.

• Sink metrics

  Name	Type	Description
  pulsar_sink_written_total	Counter	The total number of records processed by a sink.
  pulsar_sink_written_total_1min	Counter	The total number of records processed by a sink in the last 1 minute.
  pulsar_sink_received_total_1min	Counter	The total number of messages that a sink has received from Pulsar topics in the last 1 minute.
  pulsar_sink_received_total	Counter	The total number of records that a sink has received from Pulsar topics.
  pulsar_sink_last_invocation	Gauge	The timestamp of the last invocation of the sink.
  pulsar_sink_sink_exception	Gauge	The exception from a sink.
  pulsar_sink_sink_exceptions_total	Counter	The total number of sink exceptions.
  pulsar_sink_sink_exceptions_total_1min	Counter	The total number of sink exceptions in the last 1 minute.
  pulsar_sink_system_exception	Gauge	The exception from system code.
  pulsar_sink_system_exceptions_total	Counter	The total number of system exceptions.
  pulsar_sink_system_exceptions_total_1min	Counter	The total number of system exceptions in the last 1 minute.
  pulsarsink_user_metric	Summary	The user-defined metrics.

Proxy

All the proxy metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• kubernetes_pod_name: kubernetes_pod_name=${kubernetes_pod_name}. ${kubernetes_pod_name} is the kubernetes pod name.

Pulsar SQL Worker

Pulsar transaction

All the transaction metrics are labelled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• coordinator_id: coordinator_id=${coordinator_id}. ${coordinator_id} is the coordinator id.