Run a Pulsar cluster locally with Docker Compose

To run Pulsar loccally with Docker Compose, follow the steps below.

Step 1: Configure the compose.yml template

To get up and run a Pulsar cluster quickly, you can use the following template to create a compose.yml file by modifying or adding the configurations in the environment section.

  1. version: '3'
  2. networks:
  3.   pulsar:
  4.     driver: bridge
  5. services:
  6.   # Start zookeeper
  7.   zookeeper:
  8.     image: apachepulsar/pulsar:latest
  9.     container_name: zookeeper
  10.     restart: on-failure
  11.     networks:
  12.       - pulsar
  13.     volumes:
  14.       - ./data/zookeeper:/pulsar/data/zookeeper
  15.     environment:
  16.       - metadataStoreUrl=zk:zookeeper:2181
  17.       - PULSAR_MEM=-Xms256m -Xmx256m -XX:MaxDirectMemorySize=256m
  18.     command: >
  19.       bash -c "bin/apply-config-from-env.py conf/zookeeper.conf && \
  20.              bin/generate-zookeeper-config.sh conf/zookeeper.conf && \
  21.              exec bin/pulsar zookeeper"
  22.     healthcheck:
  23.       test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"]
  24.       interval: 10s
  25.       timeout: 5s
  26.       retries: 30
  28.   # Init cluster metadata
  29.   pulsar-init:
  30.     container_name: pulsar-init
  31.     hostname: pulsar-init
  32.     image: apachepulsar/pulsar:latest
  33.     networks:
  34.       - pulsar
  35.     command: >
  36.       bin/pulsar initialize-cluster-metadata \
  37.                --cluster cluster-a \
  38.                --zookeeper zookeeper:2181 \
  39.                --configuration-store zookeeper:2181 \
  40.                --web-service-url http://broker:8080 \
  41.                --broker-service-url pulsar://broker:6650
  42.     depends_on:
  43.       zookeeper:
  44.         condition: service_healthy
  46.   # Start bookie
  47.   bookie:
  48.     image: apachepulsar/pulsar:latest
  49.     container_name: bookie
  50.     restart: on-failure
  51.     networks:
  52.       - pulsar
  53.     environment:
  54.       - clusterName=cluster-a
  55.       - zkServers=zookeeper:2181
  56.       - metadataServiceUri=metadata-store:zk:zookeeper:2181
  57.       # otherwise every time we run docker compose uo or down we fail to start due to Cookie
  58.       # See: https://github.com/apache/bookkeeper/blob/405e72acf42bb1104296447ea8840d805094c787/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Cookie.java#L57-68
  59.       - advertisedAddress=bookie
  60.       - BOOKIE_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
  61.     depends_on:
  62.       zookeeper:
  63.         condition: service_healthy
  64.       pulsar-init:
  65.         condition: service_completed_successfully
  66.     # Map the local directory to the container to avoid bookie startup failure due to insufficient container disks.
  67.     volumes:
  68.       - ./data/bookkeeper:/pulsar/data/bookkeeper
  69.     command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf && exec bin/pulsar bookie"
  71.   # Start broker
  72.   broker:
  73.     image: apachepulsar/pulsar:latest
  74.     container_name: broker
  75.     hostname: broker
  76.     restart: on-failure
  77.     networks:
  78.       - pulsar
  79.     environment:
  80.       - metadataStoreUrl=zk:zookeeper:2181
  81.       - zookeeperServers=zookeeper:2181
  82.       - clusterName=cluster-a
  83.       - managedLedgerDefaultEnsembleSize=1
  84.       - managedLedgerDefaultWriteQuorum=1
  85.       - managedLedgerDefaultAckQuorum=1
  86.       - advertisedAddress=broker
  87.       - advertisedListeners=external:pulsar://127.0.0.1:6650
  88.       - PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
  89.     depends_on:
  90.       zookeeper:
  91.         condition: service_healthy
  92.       bookie:
  93.         condition: service_started
  94.     ports:
  95.       - "6650:6650"
  96.       - "8080:8080"
  97.     command: bash -c "bin/apply-config-from-env.py conf/broker.conf && exec bin/pulsar broker"

Step 2: Create a Pulsar cluster

As preparation, create data directories and change the data directory ownership to uid(10000) which is the default user id used in the Pulsar Docker container.

  1. sudo mkdir -p ./data/zookeeper ./data/bookkeeper
  2. # this step might not be necessary on other than Linux platforms
  3. sudo chown -R 10000 data

To create a Pulsar cluster by using the compose.yml file, run the following command.

  1. docker compose up -d

Step 3: Destroy the Pulsar cluster

If you want to destroy the Pulsar cluster with all the containers, run the following command. It will also delete the network that the containers are connected to.

  1. docker compose down