Pulsar metrics

Pulsar exposes the following metrics in Prometheus format. You can monitor your clusters with those metrics.

• • Pulsar Functions
  • Connectors
  • Proxy
  • Pulsar transaction

The following types of metrics are available:

• Counter: a cumulative metric that represents a single monotonically increasing counter. The value increases by default. You can reset the value to zero or restart your cluster.
• Gauge: a metric that represents a single numerical value that can arbitrarily go up and down.
• Histogram: a histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. The _bucket suffix is the number of observations within a histogram bucket, configured with parameter {le="<upper inclusive bound>"}. The _count suffix is the number of observations, shown as a time series and behaves like a counter. The _sum suffix is the sum of observed values, also shown as a time series and behaves like a counter. These suffixes are together denoted by _* in this doc.
• Summary: similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

ZooKeeper

The ZooKeeper metrics are exposed under “/metrics” at port 8000. You can use a different port by configuring the metricsProvider.httpPort in conf/zookeeper.conf.

ZooKeeper provides a New Metrics System since 3.6.0. For more detailed metrics, refer to the ZooKeeper Monitor Guide.

BookKeeper

The BookKeeper metrics are exposed under “/metrics” at port 8000. You can change the port by updating prometheusStatsHttpPort in the bookkeeper.conf configuration file.

Server metrics

Journal metrics

Storage metrics

Replication metrics

Broker

The broker metrics are exposed under “/metrics/“ at port 8080. You can change the port by updating webServicePort to a different port in the broker.conf configuration file.

All the metrics exposed by a broker are labeled with cluster=\$\{pulsar_cluster\}. The name of Pulsar cluster is the value of \$\{pulsar_cluster\}, which you have configured in the broker.conf file.

Broker metrics

All the broker metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.

BookKeeper client metrics

The BookKeeper client metrics is disabled by default. Set bookkeeperClientExposeStatsToPrometheus=true in broker.conf to expose the BookKeeper client metrics. Suppose you want to expose PerChannel BookKeeper client metrics, which allows you to get the metrics for each bookie that the BookKeeper client connected. You can set bookkeeperClientLimitStatsLogging=true in broker.conf. Note that enabling the PerChannel BookKeeper client metrics might introduce performance issues in high-load situations.

All the BookKeeper client metrics are labeled with the following label:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you configured in broker.conf.

Namespace metrics

Namespace metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to false.

All the namespace metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you configured in broker.conf.
• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.

Topic metrics

Topic metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to true.

All the topic metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you configured in broker.conf.
• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.
• topic: topic=\$\{pulsar_topic\}. \$\{pulsar_topic\} is the topic name.

Replication metrics

If a namespace is configured to be replicated among multiple Pulsar clusters, the corresponding replication metrics is also exposed when replicationMetricsEnabled is enabled.

All the replication metrics are also labelled with remoteCluster=\$\{pulsar_remote_cluster\}.

Topic lookup metrics

ManagedLedger metrics

All the managedLedger metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
• quantile: quantile=${quantile}. Quantile is only for Histogram type metric, and represents the threshold for given Buckets.

Managed cursor acknowledgment state

The acknowledgment state is persistent to the ledger first. When the acknowledgment state fails to be persistent to the ledger, they are persistent to ZooKeeper. To track the stats of acknowledgment, you can configure the metrics for the managed cursor.

All the cursor acknowledgment state metrics are labeled with the following labels:

• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.

• ledger_name: ledger_name=\$\{pulsar_ledger_name\}. \$\{pulsar_ledger_name\} is the ledger name.

• cursor_name: ledger_name=\$\{pulsar_cursor_name\}. \$\{pulsar_cursor_name\} is the cursor name.

Subscription metrics

Subscription metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to true.

All the subscription metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.
• topic: topic=\$\{pulsar_topic\}. \$\{pulsar_topic\} is the topic name.
• subscription: subscription=\$\{subscription\}. \$\{subscription\} is the topic subscription name.

Consumer metrics

Consumer metrics are only exposed when both exposeTopicLevelMetricsInPrometheus and exposeConsumerLevelMetricsInPrometheus are set to true.

All the consumer metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.
• topic: topic=\$\{pulsar_topic\}. \$\{pulsar_topic\} is the topic name.
• subscription: subscription=\$\{subscription\}. \$\{subscription\} is the topic subscription name.
• consumer_name: consumer_name=\$\{consumer_name\}. \$\{consumer_name\} is the topic consumer name.
• consumer_id: consumer_id=\$\{consumer_id\}. \$\{consumer_id\} is the topic consumer id.

Managed ledger bookie client metrics

All the managed ledger bookie client metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.

Token metrics

All the token metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.

Authentication metrics

All the authentication metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• provider_name: provider_name=\$\{provider_name\}. \$\{provider_name\} is the class name of the authentication provider.
• auth_method: auth_method=\$\{auth_method\}. \$\{auth_method\} is the authentication method of the authentication provider.
• reason: reason=\$\{reason\}. \$\{reason\} is the reason for failing authentication operation. (This label is only for pulsar_authentication_failures_total.)

Connection metrics

All the connection metrics are labelled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• broker: broker=\$\{advertised_address\}. \$\{advertised_address\} is the advertised address of the broker.
• metric: metric=\$\{metric\}. \$\{metric\} is the connection metric collective name.

Jetty metrics

For a functions-worker running separately from brokers, its Jetty metrics are only exposed when includeStandardPrometheusMetrics is set to true.

All the jetty metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.

Schema metrics

For a functions-worker running separately from brokers, its schema metrics are only exposed when includeStandardPrometheusMetrics is set to true.

All the schema metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.

Offload metrics

All the offload metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you configured in broker.conf.
• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.
• topic: topic=\$\{pulsar_topic\}. \$\{pulsar_topic\} is the topic name.

Web service executor metrics

For functions workers running separately from brokers, their Jetty metrics are only exposed when includeStandardPrometheusMetrics is set to true.

All the web service executor metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.

Metadata store metrics

All the metadata store metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you configured in broker.conf.
• name: name=\$\{metadata-store\|configuration-metadata-store\|state-metadata-store\}. \$\{name\} is the metadata store name.

JVM Metrics

Process Metrics

Memory Metrics

Buffer Pools Metrics

Garbage Collectors Metrics

Threads Metrics

Classloaders Metrics

Load balancing

This section shows all metrics related to broker load balancing.

note

• Load balancing metrics are not exposed by default. If you want to access load balancing metrics, you need to expose them by setting the following configurations in the broker.conf or standalone.conf file and ensure that your cluster has an active producer or consumer.

  loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl
  loadBalancerEnabled=true
  exposeBundlesMetricsInPrometheus=true // Add this configuration to standalone.conf

• Metrics with an asterisk (*) are only available in the extensible load balancer.

LoadBalancing metrics

All the loadbalancing metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• broker: broker=${broker}. ${broker} is the IP address of the broker
• metric: metric=”loadBalancing”.

BundleUnloading metrics

All the bundleUnloading metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker.
• metric: metric=”bundleUnloading”.

BundleSplit metrics

All the bundleUnloading metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker.
• metric: metric=”bundlesSplit”.

Bundle metrics

All the bundle metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
• broker: broker=${broker}. ${broker} is the IP address of the broker
• bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker
• metric: metric=”bundle”.

Bundle assign metrics

All the bundle assign metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name you have configured in the broker.conf file.
• broker: broker=${broker}. ${broker} is the IP address of the broker.
• bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker.
• metric: metric=”assign”.

Service unit state channel metrics

All the service unit state channel metrics are labeled with the following labels:

• cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name you have configured in the broker.conf file.
• metric: metric=”sunitStateChn”.

Pulsar Functions

All the Pulsar Functions metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.

Connectors

All the Pulsar connector metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• namespace: namespace=\$\{pulsar_namespace\}. \$\{pulsar_namespace\} is the namespace name.

Connector metrics contain source metrics and sink metrics.

• Source metrics

  Name	Type	Description
  pulsarsource_written_total	Counter	The total number of records written to a Pulsar topic.
  pulsar_source_written_1min_total	Counter	The total number of records written to a Pulsar topic in the last 1 minute.
  pulsar_source_received_total	Counter	The total number of records received from source.
  pulsar_source_received_1min_total	Counter	The total number of records received from source in the last 1 minute.
  pulsar_source_last_invocation	Gauge	The timestamp of the last invocation of the source.
  pulsar_source_source_exception	Gauge	The exception from a source.
  pulsar_source_source_exceptions_total	Counter	The total number of source exceptions.
  pulsar_source_source_exceptions_1min_total	Counter	The total number of source exceptions in the last 1 minute.
  pulsar_source_system_exception	Gauge	The exception from system code.
  pulsar_source_system_exceptions_total	Counter	The total number of system exceptions.
  pulsar_source_system_exceptions_1min_total	Counter	The total number of system exceptions in the last 1 minute.
  pulsar_source_user_metric	Summary	The user-defined metrics.

• Sink metrics

  Name	Type	Description
  pulsarsink_written_total	Counter	The total number of records processed by a sink.
  pulsar_sink_written_1min_total	Counter	The total number of records processed by a sink in the last 1 minute.
  pulsar_sink_received_1min_total	Counter	The total number of messages that a sink has received from Pulsar topics in the last 1 minute.
  pulsar_sink_received_total	Counter	The total number of records that a sink has received from Pulsar topics.
  pulsar_sink_last_invocation	Gauge	The timestamp of the last invocation of the sink.
  pulsar_sink_sink_exception	Gauge	The exception from a sink.
  pulsar_sink_sink_exceptions_total	Counter	The total number of sink exceptions.
  pulsar_sink_sink_exceptions_1min_total	Counter	The total number of sink exceptions in the last 1 minute.
  pulsar_sink_system_exception	Gauge	The exception from system code.
  pulsar_sink_system_exceptions_total	Counter	The total number of system exceptions.
  pulsar_sink_system_exceptions_1min_total	Counter	The total number of system exceptions in the last 1 minute.
  pulsar_sink_user_metric	Summary	The user-defined metrics.

Proxy

All the proxy metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• kubernetes_pod_name: kubernetes_pod_name=\$\{kubernetes_pod_name\}. \$\{kubernetes_pod_name\} is the Kubernetes pod name.

Pulsar transaction

All the transaction metrics are labeled with the following labels:

• cluster: cluster=\$\{pulsar_cluster\}. \$\{pulsar_cluster\} is the cluster name that you have configured in the broker.conf file.
• coordinator_id: coordinator_id=\$\{coordinator_id\}. \$\{coordinator_id\} is the coordinator id.