Configure metadata store

Pulsar metadata store maintains all the metadata, configuration, and coordination of a Pulsar cluster, such as topic metadata, schema, broker load data, and so on.

The metadata store of each Pulsar instance should contain the following two components:

  • A local metadata store ensemble (metadataStoreUrl) that stores cluster-specific configuration and coordination, such as which brokers are responsible for which topics as well as ownership metadata, broker load reports, and BookKeeper ledger metadata.
  • A configuration store quorum (configurationMetadataStoreUrl) stores configuration for clusters, tenants, namespaces, topics, and other entities that need to be globally consistent.

Configure metadata store - 图1note

If you are using a standalone Pulsar or a single Pulsar cluster, you only need to configure one metadata store (via metadataStoreUrl) and it also serves as a configuration store.

Pulsar supports the following metadata store services:

Configure metadata store - 图2note

RocksDB and local memory are only applicable to standalone Pulsar or single-node Pulsar clusters.

Use ZooKeeper as metadata store

Pulsar metadata store can be deployed on a separate ZooKeeper cluster or deployed on an existing ZooKeeper cluster.

To use ZooKeeper as the metadata store, add the following parameters to the conf/broker.conf or conf/standalone.conf file.

  1. metadataStoreUrl=zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
  2. configurationMetadataStoreUrl=zk:my-global-zk-1:2181,my-global-zk-2:2181,my-global-zk-3:2181

Use etcd as metadata store

To use etcd as the metadata store, add the following parameters to the conf/broker.conf or conf/standalone.conf file.

  1. metadataStoreUrl=etcd:http://my-etcd-1:2379,http://my-etcd-2:2379,http://my-etcd-3:2379
  2. configurationMetadataStoreUrl=etcd:my-global-etcd-1:2379,my-global-etcd-2:2379,my-global-etcd-3:2379
  3. # metadataStoreConfigPath=/path/to/file

Configure metadata store - 图3tip

The metadataStoreConfigPath parameter is required when you want to use the following advanced configurations.

  1. useTls=false
  2. tlsProvider=JDK
  3. tlsTrustCertsFilePath=
  4. tlsKeyFilePath=
  5. tlsCertificateFilePath=
  6. authority=

Use RocksDB as metadata store

To use RocksDB as the metadata store, add the following parameters to the conf/broker.conf or conf/standalone.conf file.

  1. metadataStoreUrl=rocksdb://data/metadata
  2. # metadataStoreConfigPath=/path/to/file

Configure metadata store - 图4tip

The metadataStoreConfigPath parameter is required when you want to use advanced configurations. See this example for more information.

Use local memory as metadata store

To use local memory as the metadata store, add the following parameters to the conf/broker.conf or conf/standalone.conf file.

  1. metadataStoreUrl=memory://local

Enable batch operations on metadata store

Pulsar metadata store supports batch operations and caching to meet low latency and high throughput and improve performance.

To enable batch operations on the metadata store, you can configure the following parameters in the conf/broker.conf or conf/standalone.conf file.

  1. # Whether we should enable metadata operations batching
  2. metadataStoreBatchingEnabled=true
  4. # Maximum delay to impose on batching grouping
  5. metadataStoreBatchingMaxDelayMillis=5
  7. # Maximum number of operations to include in a singular batch
  8. metadataStoreBatchingMaxOperations=1000
  10. # Maximum size of a batch
  11. metadataStoreBatchingMaxSizeKb=128