Apache Kafka

Detailed documentation on the Apache Kafka pubsub component

Component format

To set up Apache Kafka pub/sub, create a component of type pubsub.kafka. See the pub/sub broker component file to learn how ConsumerID is automatically generated. Read the How-to: Publish and Subscribe guide on how to create and apply a pub/sub configuration.

All component metadata field values can carry templated metadata values, which are resolved on Dapr sidecar startup. For example, you can choose to use {namespace} as the consumerGroup to enable using the same appId in different namespaces using the same topics as described in this article.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: kafka-pubsub
  5. spec:
  6. type: pubsub.kafka
  7. version: v1
  8. metadata:
  9. - name: brokers # Required. Kafka broker connection setting
  10. value: "dapr-kafka.myapp.svc.cluster.local:9092"
  11. - name: consumerGroup # Optional. Used for input bindings.
  12. value: "{namespace}"
  13. - name: consumerID # Optional. If not supplied, runtime will create one.
  14. value: "channel1"
  15. - name: clientID # Optional. Used as client tracing ID by Kafka brokers.
  16. value: "my-dapr-app-id"
  17. - name: authType # Required.
  18. value: "password"
  19. - name: saslUsername # Required if authType is `password`.
  20. value: "adminuser"
  21. - name: saslPassword # Required if authType is `password`.
  22. secretKeyRef:
  23. name: kafka-secrets
  24. key: saslPasswordSecret
  25. - name: saslMechanism
  26. value: "SHA-512"
  27. - name: maxMessageBytes # Optional.
  28. value: 1024
  29. - name: consumeRetryInterval # Optional.
  30. value: 200ms
  31. - name: version # Optional.
  32. value: 0.10.2.0
  33. - name: disableTls # Optional. Disable TLS. This is not safe for production!! You should read the `Mutual TLS` section for how to use TLS.
  34. value: "true"

For details on using secretKeyRef, see the guide on how to reference secrets in components.

Spec metadata fields

FieldRequiredDetailsExample
brokersYA comma-separated list of Kafka brokers.“localhost:9092,dapr-kafka.myapp.svc.cluster.local:9093”
consumerGroupNA kafka consumer group to listen on. Each record published to a topic is delivered to one consumer within each consumer group subscribed to the topic.“group1”
consumerIDNConsumer ID (consumer tag) organizes one or more consumers into a group. Consumers with the same consumer ID work as one virtual consumer; for example, a message is processed only once by one of the consumers in the group. If the consumerID is not provided, the Dapr runtime set it to the Dapr application ID (appID) value.“channel1”
clientIDNA user-provided string sent with every request to the Kafka brokers for logging, debugging, and auditing purposes. Defaults to “sarama”.“my-dapr-app”
authRequiredNDeprecated Enable SASL authentication with the Kafka brokers.“true”, “false”
authTypeYConfigure or disable authentication. Supported values: none, password, mtls, or oidc“password”, “none”
saslUsernameNThe SASL username used for authentication. Only required if authType is set to “password”.“adminuser”
saslPasswordNThe SASL password used for authentication. Can be secretKeyRef to use a secret reference. Only required if authType is set to “password”`.“”, “KeFg23!”
saslMechanismNThe SASL Authentication Mechanism you wish to use. Only required if authType is set to “password”. Defaults to PLAINTEXT“SHA-512”, “SHA-256”, “PLAINTEXT”
initialOffsetNThe initial offset to use if no offset was previously committed. Should be “newest” or “oldest”. Defaults to “newest”.“oldest”
maxMessageBytesNThe maximum size in bytes allowed for a single Kafka message. Defaults to 1024.2048
consumeRetryIntervalNThe interval between retries when attempting to consume topics. Treats numbers without suffix as milliseconds. Defaults to 100ms.200ms
consumeRetryEnabledNDisable consume retry by setting “false”“true”, “false”
versionNKafka cluster version. Defaults to 2.0.0. Note that this must be set to 1.0.0 if you are using Azure EventHubs with Kafka.0.10.2.0
caCertNCertificate authority certificate, required for using TLS. Can be secretKeyRef to use a secret reference“——-BEGIN CERTIFICATE——-\n<base64-encoded DER>\n——-END CERTIFICATE——-“
clientCertNClient certificate, required for authType mtls. Can be secretKeyRef to use a secret reference“——-BEGIN CERTIFICATE——-\n<base64-encoded DER>\n——-END CERTIFICATE——-“
clientKeyNClient key, required for authType mtls Can be secretKeyRef to use a secret reference“——-BEGIN RSA PRIVATE KEY——-\n<base64-encoded PKCS8>\n——-END RSA PRIVATE KEY——-“
skipVerifyNSkip TLS verification, this is not recommended for use in production. Defaults to “false”“true”, “false”
disableTlsNDisable TLS for transport security. To disable, you’re not required to set value to “true”. This is not recommended for use in production. Defaults to “false”.“true”, “false”
oidcTokenEndpointNFull URL to an OAuth2 identity provider access token endpoint. Required when authType is set to oidchttps://identity.example.com/v1/token”
oidcClientIDNThe OAuth2 client ID that has been provisioned in the identity provider. Required when authType is set to oidcdapr-kafka
oidcClientSecretNThe OAuth2 client secret that has been provisioned in the identity provider: Required when authType is set to oidc“KeFg23!”
oidcScopesNComma-delimited list of OAuth2/OIDC scopes to request with the access token. Recommended when authType is set to oidc. Defaults to “openid”“openid,kafka-prod”

The secretKeyRef above is referencing a kubernetes secrets store to access the tls information. Visit here to learn more about how to configure a secret store component.

Note

The metadata version must be set to 1.0.0 when using Azure EventHubs with Kafka.

Authentication

Kafka supports a variety of authentication schemes and Dapr supports several: SASL password, mTLS, OIDC/OAuth2. With the added authentication methods, the authRequired field has been deprecated from the v1.6 release and instead the authType field should be used. If authRequired is set to true, Dapr will attempt to configure authType correctly based on the value of saslPassword. There are four valid values for authType: none, password, certificate, mtls, and oidc. Note this is authentication only; authorization is still configured within Kafka.

None

Setting authType to none will disable any authentication. This is NOT recommended in production.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: kafka-pubsub-noauth
  5. spec:
  6. type: pubsub.kafka
  7. version: v1
  8. metadata:
  9. - name: brokers # Required. Kafka broker connection setting
  10. value: "dapr-kafka.myapp.svc.cluster.local:9092"
  11. - name: consumerGroup # Optional. Used for input bindings.
  12. value: "group1"
  13. - name: clientID # Optional. Used as client tracing ID by Kafka brokers.
  14. value: "my-dapr-app-id"
  15. - name: authType # Required.
  16. value: "none"
  17. - name: maxMessageBytes # Optional.
  18. value: 1024
  19. - name: consumeRetryInterval # Optional.
  20. value: 200ms
  21. - name: version # Optional.
  22. value: 0.10.2.0
  23. - name: disableTls
  24. value: "true"

SASL Password

Setting authType to password enables SASL authentication. This requires setting the saslUsername and saslPassword fields.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: kafka-pubsub-sasl
  5. spec:
  6. type: pubsub.kafka
  7. version: v1
  8. metadata:
  9. - name: brokers # Required. Kafka broker connection setting
  10. value: "dapr-kafka.myapp.svc.cluster.local:9092"
  11. - name: consumerGroup # Optional. Used for input bindings.
  12. value: "group1"
  13. - name: clientID # Optional. Used as client tracing ID by Kafka brokers.
  14. value: "my-dapr-app-id"
  15. - name: authType # Required.
  16. value: "password"
  17. - name: saslUsername # Required if authType is `password`.
  18. value: "adminuser"
  19. - name: saslPassword # Required if authType is `password`.
  20. secretKeyRef:
  21. name: kafka-secrets
  22. key: saslPasswordSecret
  23. - name: saslMechanism
  24. value: "SHA-512"
  25. - name: maxMessageBytes # Optional.
  26. value: 1024
  27. - name: consumeRetryInterval # Optional.
  28. value: 200ms
  29. - name: version # Optional.
  30. value: 0.10.2.0
  31. - name: caCert
  32. secretKeyRef:
  33. name: kafka-tls
  34. key: caCert

Mutual TLS

Setting authType to mtls uses a x509 client certificate (the clientCert field) and key (the clientKey field) to authenticate. Note that mTLS as an authentication mechanism is distinct from using TLS to secure the transport layer via encryption. mTLS requires TLS transport (meaning disableTls must be false), but securing the transport layer does not require using mTLS. See Communication using TLS for configuring underlying TLS transport.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: kafka-pubsub-mtls
  5. spec:
  6. type: pubsub.kafka
  7. version: v1
  8. metadata:
  9. - name: brokers # Required. Kafka broker connection setting
  10. value: "dapr-kafka.myapp.svc.cluster.local:9092"
  11. - name: consumerGroup # Optional. Used for input bindings.
  12. value: "group1"
  13. - name: clientID # Optional. Used as client tracing ID by Kafka brokers.
  14. value: "my-dapr-app-id"
  15. - name: authType # Required.
  16. value: "mtls"
  17. - name: caCert
  18. secretKeyRef:
  19. name: kafka-tls
  20. key: caCert
  21. - name: clientCert
  22. secretKeyRef:
  23. name: kafka-tls
  24. key: clientCert
  25. - name: clientKey
  26. secretKeyRef:
  27. name: kafka-tls
  28. key: clientKey
  29. - name: maxMessageBytes # Optional.
  30. value: 1024
  31. - name: consumeRetryInterval # Optional.
  32. value: 200ms
  33. - name: version # Optional.
  34. value: 0.10.2.0

OAuth2 or OpenID Connect

Setting authType to oidc enables SASL authentication via the OAUTHBEARER mechanism. This supports specifying a bearer token from an external OAuth2 or OIDC identity provider. Currently, only the client_credentials grant is supported.

Configure oidcTokenEndpoint to the full URL for the identity provider access token endpoint.

Set oidcClientID and oidcClientSecret to the client credentials provisioned in the identity provider.

If caCert is specified in the component configuration, the certificate is appended to the system CA trust for verifying the identity provider certificate. Similarly, if skipVerify is specified in the component configuration, verification will also be skipped when accessing the identity provider.

By default, the only scope requested for the token is openid; it is highly recommended that additional scopes be specified via oidcScopes in a comma-separated list and validated by the Kafka broker. If additional scopes are not used to narrow the validity of the access token, a compromised Kafka broker could replay the token to access other services as the Dapr clientID.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: kafka-pubsub
  5. spec:
  6. type: pubsub.kafka
  7. version: v1
  8. metadata:
  9. - name: brokers # Required. Kafka broker connection setting
  10. value: "dapr-kafka.myapp.svc.cluster.local:9092"
  11. - name: consumerGroup # Optional. Used for input bindings.
  12. value: "group1"
  13. - name: clientID # Optional. Used as client tracing ID by Kafka brokers.
  14. value: "my-dapr-app-id"
  15. - name: authType # Required.
  16. value: "oidc"
  17. - name: oidcTokenEndpoint # Required if authType is `oidc`.
  18. value: "https://identity.example.com/v1/token"
  19. - name: oidcClientID # Required if authType is `oidc`.
  20. value: "dapr-myapp"
  21. - name: oidcClientSecret # Required if authType is `oidc`.
  22. secretKeyRef:
  23. name: kafka-secrets
  24. key: oidcClientSecret
  25. - name: oidcScopes # Recommended if authType is `oidc`.
  26. value: "openid,kafka-dev"
  27. - name: caCert # Also applied to verifying OIDC provider certificate
  28. secretKeyRef:
  29. name: kafka-tls
  30. key: caCert
  31. - name: maxMessageBytes # Optional.
  32. value: 1024
  33. - name: consumeRetryInterval # Optional.
  34. value: 200ms
  35. - name: version # Optional.
  36. value: 0.10.2.0

Communication using TLS

By default TLS is enabled to secure the transport layer to Kafka. To disable TLS, set disableTls to true. When TLS is enabled, you can control server certificate verification using skipVerify to disable verification (NOT recommended in production environments) and caCert to specify a trusted TLS certificate authority (CA). If no caCert is specified, the system CA trust will be used. To also configure mTLS authentication, see the section under Authentication. Below is an example of a Kafka pubsub component configured to use transport layer TLS:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: kafka-pubsub
  5. spec:
  6. type: pubsub.kafka
  7. version: v1
  8. metadata:
  9. - name: brokers # Required. Kafka broker connection setting
  10. value: "dapr-kafka.myapp.svc.cluster.local:9092"
  11. - name: consumerGroup # Optional. Used for input bindings.
  12. value: "group1"
  13. - name: clientID # Optional. Used as client tracing ID by Kafka brokers.
  14. value: "my-dapr-app-id"
  15. - name: authType # Required.
  16. value: "certificate"
  17. - name: consumeRetryInterval # Optional.
  18. value: 200ms
  19. - name: version # Optional.
  20. value: 0.10.2.0
  21. - name: maxMessageBytes # Optional.
  22. value: 1024
  23. - name: caCert # Certificate authority certificate.
  24. secretKeyRef:
  25. name: kafka-tls
  26. key: caCert
  27. auth:
  28. secretStore: <SECRET_STORE_NAME>

Sending and receiving multiple messages

Apache Kafka component supports sending and receiving multiple messages in a single operation using the bulk Pub/sub API.

Configuring bulk subscribe

When subscribing to a topic, you can configure bulkSubscribe options. Refer to Subscribing messages in bulk for more details. Learn more about the bulk subscribe API.

Apache Kafka supports the following bulk metadata options:

ConfigurationDefault
maxBulkAwaitDurationMs10000 (10s)
maxBulkSubCount80

Per-call metadata fields

Partition Key

When invoking the Kafka pub/sub, its possible to provide an optional partition key by using the metadata query param in the request url.

The param name is partitionKey.

Example:

  1. curl -X POST http://localhost:3500/v1.0/publish/myKafka/myTopic?metadata.partitionKey=key1 \
  2. -H "Content-Type: application/json" \
  3. -d '{
  4. "data": {
  5. "message": "Hi"
  6. }
  7. }'

Message headers

All other metadata key/value pairs (that are not partitionKey) are set as headers in the Kafka message. Here is an example setting a correlationId for the message.

  1. curl -X POST http://localhost:3500/v1.0/publish/myKafka/myTopic?metadata.correlationId=myCorrelationID&metadata.partitionKey=key1 \
  2. -H "Content-Type: application/json" \
  3. -d '{
  4. "data": {
  5. "message": "Hi"
  6. }
  7. }'

Create a Kafka instance

You can run Kafka locally using this Docker image. To run without Docker, see the getting started guide here.

To run Kafka on Kubernetes, you can use any Kafka operator, such as Strimzi.

Last modified June 19, 2023: Merge pull request #3565 from dapr/aacrawfi/skip-secrets-close (b1763bf)