Apache Kafka

Detailed documentation on the Apache Kafka pubsub component

Component format

To setup Apache Kafka pubsub create a component of type pubsub.kafka. See this guide on how to create and apply a pubsub configuration.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4.   name: kafka-pubsub
  5.   namespace: default
  6. spec:
  7.   type: pubsub.kafka
  8.   version: v1
  9.   metadata:
  10.       # Kafka broker connection setting
  11.     - name: brokers
  12.       value: "dapr-kafka.myapp.svc.cluster.local:9092"
  13.     - name: authRequired
  14.       value: "true"
  15.     - name: saslUsername
  16.       value: "adminuser"
  17.     - name: saslPassword
  18.       value: "KeFg23!"
  19.     - name: maxMessageBytes
  20.       value: 1024

Warning

The above example uses secrets as plain strings. It is recommended to use a secret store for the secrets as described here.

Spec metadata fields

FieldRequiredDetailsExample
brokersYComma separated list of kafka brokerslocalhost:9092, dapr-kafka.myapp.svc.cluster.local:9092
authRequiredNEnable authentication on the Kafka broker. Defaults to “false”.“true”, “false”
saslUsernameNUsername used for authentication. Only required if authRequired is set to true.“adminuser”
saslPasswordNPassword used for authentication. Can be secretKeyRef to use a secret reference. Only required if authRequired is set to true. Can be secretKeyRef to use a secret reference“”, “KeFg23!”
maxMessageBytesNThe maximum message size allowed for a single Kafka message. Default is 1024.2048

Per-call metadata fields

Partition Key

When invoking the Kafka pub/sub, its possible to provide an optional partition key by using the metadata query param in the request url.

The param name is partitionKey.

Example:

  1. curl -X POST http://localhost:3500/v1.0/publish/myKafka/myTopic?metadata.partitionKey=key1 \
  2.   -H "Content-Type: application/json" \
  3.   -d '{
  4.         "data": {
  5.           "message": "Hi"
  6.         }
  7.       }'

Create a Kafka instance

You can run Kafka locally using this Docker image. To run without Docker, see the getting started guide here.

To run Kafka on Kubernetes, you can use any Kafka operator, such as Strimzi.

Last modified May 26, 2021: Update to point to 1.2 (#1518) (c690379)