Pulsar C++ client

You can use Pulsar C++ client to create Pulsar producers and consumers in C++.

All the methods in producer, consumer, and reader of a C++ client are thread-safe.

Supported platforms

Pulsar C++ client is supported on Linux ,MacOS and Windows platforms.

Doxygen-generated API docs for the C++ client are available here.

Linux

note

You can choose one of the following installation methods based on your needs: Compilation, Install RPM or Install Debian.

Compilation

System requirements

You need to install the following components before using the C++ client:

  1. Clone the Pulsar repository.
  2. $ git clone https://github.com/apache/pulsar
  1. Install all necessary dependencies.
  2. $ apt-get install cmake libssl-dev libcurl4-openssl-dev liblog4cxx-dev \
  3.   libprotobuf-dev protobuf-compiler libboost-all-dev google-mock libgtest-dev libjsoncpp-dev
  1. Compile and install Google Test.
  2. # libgtest-dev version is 1.18.0 or above
  3. $ cd /usr/src/googletest
  4. $ sudo cmake .
  5. $ sudo make
  6. $ sudo cp ./googlemock/libgmock.a ./googlemock/gtest/libgtest.a /usr/lib/
  8. # less than 1.18.0
  9. $ cd /usr/src/gtest
  10. $ sudo cmake .
  11. $ sudo make
  12. $ sudo cp libgtest.a /usr/lib
  14. $ cd /usr/src/gmock
  15. $ sudo cmake .
  16. $ sudo make
  17. $ sudo cp libgmock.a /usr/lib
  1. Compile the Pulsar client library for C++ inside the Pulsar repository.
  2. $ cd pulsar-client-cpp
  3. $ cmake .
  4. $ make

After you install the components successfully, the files libpulsar.so and libpulsar.a are in the lib folder of the repository. The tools perfProducer and perfConsumer are in the perf directory.

Install Dependencies

Since 2.1.0 release, Pulsar ships pre-built RPM and Debian packages. You can download and install those packages directly.

After you download and install RPM or DEB, the libpulsar.so, libpulsarnossl.so, libpulsar.a, and libpulsarwithdeps.a libraries are in your /usr/lib directory.

By default, they are built in code path ${PULSAR_HOME}/pulsar-client-cpp. You can build with the command below.

cmake . -DBUILD_TESTS=OFF -DLINK_STATIC=ON && make pulsarShared pulsarSharedNossl pulsarStatic pulsarStaticWithDeps -j 3.

These libraries rely on some other libraries. If you want to get detailed version of dependencies, see RPM or DEB files.

  1. libpulsar.so is a shared library, containing statically linked boost and openssl. It also dynamically links all other necessary libraries. You can use this Pulsar library with the command below.
  2.  g++ --std=c++11  PulsarTest.cpp -o test /usr/lib/libpulsar.so -I/usr/local/ssl/include
  1. libpulsarnossl.so is a shared library, similar to libpulsar.so except that the libraries openssl and crypto are dynamically linked. You can use this Pulsar library with the command below.
  2.  g++ --std=c++11  PulsarTest.cpp -o test /usr/lib/libpulsarnossl.so -lssl -lcrypto -I/usr/local/ssl/include -L/usr/local/ssl/lib
  1. libpulsar.a is a static library. You need to load dependencies before using this library. You can use this Pulsar library with the command below.
  2.  g++ --std=c++11  PulsarTest.cpp -o test /usr/lib/libpulsar.a -lssl -lcrypto -ldl -lpthread  -I/usr/local/ssl/include -L/usr/local/ssl/lib -lboost_system -lboost_regex -lcurl -lprotobuf -lzstd -lz
  1. libpulsarwithdeps.a is a static library, based on libpulsar.a. It is archived in the dependencies of libboost_regex, libboost_system, libcurl, libprotobuf, libzstd and libz. You can use this Pulsar library with the command below.
  2.  g++ --std=c++11  PulsarTest.cpp -o test /usr/lib/libpulsarwithdeps.a -lssl -lcrypto -ldl -lpthread  -I/usr/local/ssl/include -L/usr/local/ssl/lib

The libpulsarwithdeps.a does not include library openssl related libraries libssl and libcrypto, because these two libraries are related to security. It is more reasonable and easier to use the versions provided by the local system to handle security issues and upgrade libraries.

Install RPM

  1. Download a RPM package from the links in the table.
LinkCrypto files
clientasc, sha512
client-debuginfoasc, sha512
client-develasc, sha512
  1. Install the package using the following command.
  2. $ rpm -ivh apache-pulsar-client*.rpm

After you install RPM successfully, Pulsar libraries are in the /usr/lib directory, for example：

  2. lrwxrwxrwx 1 root root 18 Dec 30 22:21 libpulsar.so -> libpulsar.so.2.9.1
  3. lrwxrwxrwx 1 root root 23 Dec 30 22:21 libpulsarnossl.so -> libpulsarnossl.so.2.9.1
note

If you get the error that libpulsar.so: cannot open shared object file: No such file or directory when starting Pulsar client, you may need to run ldconfig first.

  1. Install the GCC and g++ using the following command, otherwise errors would occur in installing Node.js.
  2. $ sudo yum -y install gcc automake autoconf libtool make
  3. $ sudo yum -y install gcc-c++

Install Debian

  1. Download a Debian package from the links in the table.
LinkCrypto files
clientasc, sha512
client-develasc, sha512
  1. Install the package using the following command.
  2. $ apt install ./apache-pulsar-client*.deb

After you install DEB successfully, Pulsar libraries are in the /usr/lib directory.

Build

If you want to build RPM and Debian packages from the latest master, follow the instructions below. You should run all the instructions at the root directory of your cloned Pulsar repository.

There are recipes that build RPM and Debian packages containing a statically linked libpulsar.so / libpulsarnossl.so / libpulsar.a / libpulsarwithdeps.a with all required dependencies.

To build the C++ library packages, you need to build the Java packages first.

  2. mvn install -DskipTests

RPM

To build the RPM inside a Docker container, use the command below. The RPMs are in the pulsar-client-cpp/pkg/rpm/RPMS/x86_64/ path.

  2. pulsar-client-cpp/pkg/rpm/docker-build-rpm.sh
Package nameContent
pulsar-clientShared library libpulsar.so and libpulsarnossl.so
pulsar-client-develStatic library libpulsar.a, libpulsarwithdeps.aand C++ and C headers
pulsar-client-debuginfoDebug symbols for libpulsar.so

Debian

To build Debian packages, enter the following command.

  2. pulsar-client-cpp/pkg/deb/docker-build-deb.sh

Debian packages are created in the pulsar-client-cpp/pkg/deb/BUILD/DEB/ path.

Package nameContent
pulsar-clientShared library libpulsar.so and libpulsarnossl.so
pulsar-client-devStatic library libpulsar.a, libpulsarwithdeps.a and C++ and C headers

MacOS

Compilation

  1. Clone the Pulsar repository.
  2. $ git clone https://github.com/apache/pulsar
  1. Install all necessary dependencies.
  2. # OpenSSL installation
  3. $ brew install openssl
  4. $ export OPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include/
  5. $ export OPENSSL_ROOT_DIR=/usr/local/opt/openssl/
  7. # Protocol Buffers installation
  8. $ brew install protobuf boost boost-python log4cxx
  9. # If you are using python3, you need to install boost-python3 
  11. # Google Test installation
  12. $ git clone https://github.com/google/googletest.git
  13. $ cd googletest
  14. $ cmake .
  15. $ make install
  1. Compile the Pulsar client library in the repository that you cloned.
  2. $ cd pulsar-client-cpp
  3. $ cmake .
  4. $ make

Install libpulsar

Pulsar releases are available in the Homebrew core repository. You can install the C++ client library with the following command. The package is installed with the library and headers.

  2. brew install libpulsar

Windows (64-bit)

Compilation

  1. Clone the Pulsar repository.
  2. $ git clone https://github.com/apache/pulsar
  1. Install all necessary dependencies.
  2. cd ${PULSAR_HOME}/pulsar-client-cpp
  3. vcpkg install --feature-flags=manifests --triplet x64-windows
  1. Build C++ libraries.
  2. cmake -B ./build -A x64 -DBUILD_PYTHON_WRAPPER=OFF -DBUILD_TESTS=OFF -DVCPKG_TRIPLET=x64-windows -DCMAKE_BUILD_TYPE=Release -S .
  3. cmake --build ./build --config Release

NOTE

  1. For Windows 32-bit, you need to use -A Win32 and -DVCPKG_TRIPLET=x86-windows.
  2. For MSVC Debug mode, you need to replace Release with Debug for both CMAKE_BUILD_TYPE variable and --config option.
  1. Client libraries are available in the following places.
  2. ${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.lib
  3. ${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.dll

Connection URLs

To connect Pulsar using client libraries, you need to specify a Pulsar protocol URL.

Pulsar protocol URLs are assigned to specific clusters, you can use the Pulsar URI scheme. The default port is 6650. The following is an example for localhost.

  2. pulsar://localhost:6650

In a Pulsar cluster in production, the URL looks as follows.

  2. pulsar://pulsar.us-west.example.com:6650

If you use TLS authentication, you need to add ssl, and the default port is 6651. The following is an example.

  2. pulsar+ssl://pulsar.us-west.example.com:6651

Create a producer

To use Pulsar as a producer, you need to create a producer on the C++ client. There are two main ways of using a producer:

Simple blocking example

This example sends 100 messages using the blocking style. While simple, it does not produce high throughput as it waits for each ack to come back before sending the next message.

  2. #include <pulsar/Client.h>
  3. #include <thread>
  5. using namespace pulsar;
  7. int main() {
  8.     Client client("pulsar://localhost:6650");
  10.     Result result = client.createProducer("persistent://public/default/my-topic", producer);
  11.     if (result != ResultOk) {
  12.         std::cout << "Error creating producer: " << result << std::endl;
  13.         return -1;
  14.     }
  16.     // Send 100 messages synchronously
  17.     int ctr = 0;
  18.     while (ctr < 100) {
  19.         std::string content = "msg" + std::to_string(ctr);
  20.         Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
  21.         Result result = producer.send(msg);
  22.         if (result != ResultOk) {
  23.             std::cout << "The message " << content << " could not be sent, received code: " << result << std::endl;
  24.         } else {
  25.             std::cout << "The message " << content << " sent successfully" << std::endl;
  26.         }
  28.         std::this_thread::sleep_for(std::chrono::milliseconds(100));
  29.         ctr++;
  30.     }
  32.     std::cout << "Finished producing synchronously!" << std::endl;
  34.     client.close();
  35.     return 0;
  36. }

Non-blocking example

This example sends 100 messages using the non-blocking style calling sendAsync instead of send. This allows the producer to have multiple messages inflight at a time which increases throughput.

The producer configuration blockIfQueueFull is useful here to avoid ResultProducerQueueIsFull errors when the internal queue for outgoing send requests becomes full. Once the internal queue is full, sendAsync becomes blocking which can make your code simpler.

Without this configuration, the result code ResultProducerQueueIsFull is passed to the callback. You must decide how to deal with that (retry, discard etc).

  2. #include <pulsar/Client.h>
  3. #include <thread>
  5. using namespace pulsar;
  7. std::atomic<uint32_t> acksReceived;
  9. void callback(Result code, const MessageId& msgId, std::string msgContent) {
  10.     // message processing logic here
  11.     std::cout << "Received ack for msg: " << msgContent << " with code: "
  12.         << code << " -- MsgID: " << msgId << std::endl;
  13.     acksReceived++;
  14. }
  16. int main() {
  17.     Client client("pulsar://localhost:6650");
  19.     ProducerConfiguration producerConf;
  20.     producerConf.setBlockIfQueueFull(true);
  21.     Producer producer;
  22.     Result result = client.createProducer("persistent://public/default/my-topic",
  23.                                           producerConf, producer);
  24.     if (result != ResultOk) {
  25.         std::cout << "Error creating producer: " << result << std::endl;
  26.         return -1;
  27.     }
  29.     // Send 100 messages asynchronously
  30.     int ctr = 0;
  31.     while (ctr < 100) {
  32.         std::string content = "msg" + std::to_string(ctr);
  33.         Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
  34.         producer.sendAsync(msg, std::bind(callback,
  35.                                           std::placeholders::_1, std::placeholders::_2, content));
  37.         std::this_thread::sleep_for(std::chrono::milliseconds(100));
  38.         ctr++;
  39.     }
  41.     // wait for 100 messages to be acked
  42.     while (acksReceived < 100) {
  43.         std::this_thread::sleep_for(std::chrono::milliseconds(100));
  44.     }
  46.     std::cout << "Finished producing asynchronously!" << std::endl;
  48.     client.close();
  49.     return 0;
  50. }

Partitioned topics and lazy producers

When scaling out a Pulsar topic, you may configure a topic to have hundreds of partitions. Likewise, you may have also scaled out your producers so there are hundreds or even thousands of producers. This can put some strain on the Pulsar brokers as when you create a producer on a partitioned topic, internally it creates one internal producer per partition which involves communications to the brokers for each one. So for a topic with 1000 partitions and 1000 producers, it ends up creating 1,000,000 internal producers across the producer applications, each of which has to communicate with a broker to find out which broker it should connect to and then perform the connection handshake.

You can reduce the load caused by this combination of a large number of partitions and many producers by doing the following:

  • use SinglePartition partition routing mode (this ensures that all messages are only sent to a single, randomly selected partition)
  • use non-keyed messages (when messages are keyed, routing is based on the hash of the key and so messages will end up being sent to multiple partitions)
  • use lazy producers (this ensures that an internal producer is only created on demand when a message needs to be routed to a partition)

With our example above, that reduces the number of internal producers spread out over the 1000 producer apps from 1,000,000 to just 1000.

Note that there can be extra latency for the first message sent. If you set a low send timeout, this timeout could be reached if the initial connection handshake is slow to complete.

  2. ProducerConfiguration producerConf;
  3. producerConf.setPartitionsRoutingMode(ProducerConfiguration::UseSinglePartition);
  4. producerConf.setLazyStartPartitionedProducers(true);

Enable chunking

Message chunking enables Pulsar to process large payload messages by splitting the message into chunks at the producer side and aggregating chunked messages at the consumer side.

The message chunking feature is OFF by default. The following is an example about how to enable message chunking when creating a producer.

  2. ProducerConfiguration conf;
  3. conf.setBatchingEnabled(false);
  4. conf.setChunkingEnabled(true);
  5. Producer producer;
  6. client.createProducer("my-topic", conf, producer);

Note: To enable chunking, you need to disable batching (setBatchingEnabled\=false) concurrently.

Create a consumer

To use Pulsar as a consumer, you need to create a consumer on the C++ client. There are two main ways of using the consumer:

Blocking example

The benefit of this approach is that it is the simplest code. Simply keeps calling receive(msg) which blocks until a message is received.

This example starts a subscription at the earliest offset and consumes 100 messages.

  2. #include <pulsar/Client.h>
  4. using namespace pulsar;
  6. int main() {
  7.     Client client("pulsar://localhost:6650");
  9.     Consumer consumer;
  10.     ConsumerConfiguration config;
  11.     config.setSubscriptionInitialPosition(InitialPositionEarliest);
  12.     Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
  13.     if (result != ResultOk) {
  14.         std::cout << "Failed to subscribe: " << result << std::endl;
  15.         return -1;
  16.     }
  18.     Message msg;
  19.     int ctr = 0;
  20.     // consume 100 messages
  21.     while (ctr < 100) {
  22.         consumer.receive(msg);
  23.         std::cout << "Received: " << msg
  24.             << "  with payload '" << msg.getDataAsString() << "'" << std::endl;
  26.         consumer.acknowledge(msg);
  27.         ctr++;
  28.     }
  30.     std::cout << "Finished consuming synchronously!" << std::endl;
  32.     client.close();
  33.     return 0;
  34. }

Consumer with a message listener

You can avoid running a loop with blocking calls with an event based style by using a message listener which is invoked for each message that is received.

This example starts a subscription at the earliest offset and consumes 100 messages.

  2. #include <pulsar/Client.h>
  3. #include <atomic>
  4. #include <thread>
  6. using namespace pulsar;
  8. std::atomic<uint32_t> messagesReceived;
  10. void handleAckComplete(Result res) {
  11.     std::cout << "Ack res: " << res << std::endl;
  12. }
  14. void listener(Consumer consumer, const Message& msg) {
  15.     std::cout << "Got message " << msg << " with content '" << msg.getDataAsString() << "'" << std::endl;
  16.     messagesReceived++;
  17.     consumer.acknowledgeAsync(msg.getMessageId(), handleAckComplete);
  18. }
  20. int main() {
  21.     Client client("pulsar://localhost:6650");
  23.     Consumer consumer;
  24.     ConsumerConfiguration config;
  25.     config.setMessageListener(listener);
  26.     config.setSubscriptionInitialPosition(InitialPositionEarliest);
  27.     Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
  28.     if (result != ResultOk) {
  29.         std::cout << "Failed to subscribe: " << result << std::endl;
  30.         return -1;
  31.     }
  33.     // wait for 100 messages to be consumed
  34.     while (messagesReceived < 100) {
  35.         std::this_thread::sleep_for(std::chrono::milliseconds(100));
  36.     }
  38.     std::cout << "Finished consuming asynchronously!" << std::endl;
  40.     client.close();
  41.     return 0;
  42. }

Configure chunking

You can limit the maximum number of chunked messages a consumer maintains concurrently by configuring the setMaxPendingChunkedMessage and setAutoAckOldestChunkedMessageOnQueueFull parameters. When the threshold is reached, the consumer drops pending messages by silently acknowledging them or asking the broker to redeliver them later.

The following is an example of how to configure message chunking.

  2. ConsumerConfiguration conf;
  3. conf.setAutoAckOldestChunkedMessageOnQueueFull(true);
  4. conf.setMaxPendingChunkedMessage(100);
  5. Consumer consumer;
  6. client.subscribe("my-topic", "my-sub", conf, consumer);

Enable authentication in connection URLs

If you use TLS authentication when connecting to Pulsar, you need to add ssl in the connection URLs, and the default port is 6651. The following is an example.

  2. ClientConfiguration config = ClientConfiguration();
  3. config.setUseTls(true);
  4. config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
  5. config.setTlsAllowInsecureConnection(false);
  6. config.setAuth(pulsar::AuthTls::create(
  7.             "/path/to/client-cert.pem", "/path/to/client-key.pem"););
  9. Client client("pulsar+ssl://my-broker.com:6651", config);

For complete examples, refer to C++ client examples.

Schema

This section describes some examples about schema. For more information about schema, see Pulsar schema.

Avro schema

  • The following example shows how to create a producer with an Avro schema.

    2. static const std::string exampleSchema =
    3.     "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
    4.     "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
    5. Producer producer;
    6. ProducerConfiguration producerConf;
    7. producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
    8. client.createProducer("topic-avro", producerConf, producer);

  • The following example shows how to create a consumer with an Avro schema.

    2. static const std::string exampleSchema =
    3.     "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
    4.     "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
    5. ConsumerConfiguration consumerConf;
    6. Consumer consumer;
    7. consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
    8. client.subscribe("topic-avro", "sub-2", consumerConf, consumer)

ProtobufNative schema

The following example shows how to create a producer and a consumer with a ProtobufNative schema. ​

  1. Generate the User class using Protobuf3.

    note

    You need to use Protobuf3 or later versions.

​

  2. syntax = "proto3";
  4. message User {
  5.     string name = 1;
  6.     int32 age = 2;
  7. }

​ 2. Include the ProtobufNativeSchema.h in your source code. Ensure the Protobuf dependency has been added to your project. ​

  2. #include <pulsar/ProtobufNativeSchema.h>

​ 3. Create a producer to send a User instance. ​

  2. ProducerConfiguration producerConf;
  3. producerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
  4. Producer producer;
  5. client.createProducer("topic-protobuf", producerConf, producer);
  6. User user;
  7. user.set_name("my-name");
  8. user.set_age(10);
  9. std::string content;
  10. user.SerializeToString(&content);
  11. producer.send(MessageBuilder().setContent(content).build());

​ 4. Create a consumer to receive a User instance. ​

  2. ConsumerConfiguration consumerConf;
  3. consumerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
  4. consumerConf.setSubscriptionInitialPosition(InitialPositionEarliest);
  5. Consumer consumer;
  6. client.subscribe("topic-protobuf", "my-sub", consumerConf, consumer);
  7. Message msg;
  8. consumer.receive(msg);
  9. User user2;
  10. user2.ParseFromArray(msg.getData(), msg.getLength());