Azure Event Hubs

Detailed documentation on the Azure Event Hubs pubsub component

Component format

To set up an Azure Event Hubs pub/sub, create a component of type pubsub.azure.eventhubs. See the pub/sub broker component file to learn how ConsumerID is automatically generated. Read the How-to: Publish and Subscribe guide on how to create and apply a pub/sub configuration.

Apart from the configuration metadata fields shown below, Azure Event Hubs also supports Azure Authentication mechanisms.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: eventhubs-pubsub
  5. spec:
  6. type: pubsub.azure.eventhubs
  7. version: v1
  8. metadata:
  9. # Either connectionString or eventHubNamespace is required
  10. # Use connectionString when *not* using Microsoft Entra ID
  11. - name: connectionString
  12. value: "Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}"
  13. # Use eventHubNamespace when using Microsoft Entra ID
  14. - name: eventHubNamespace
  15. value: "namespace"
  16. - name: consumerID # Optional. If not supplied, the runtime will create one.
  17. value: "channel1"
  18. - name: enableEntityManagement
  19. value: "false"
  20. # The following four properties are needed only if enableEntityManagement is set to true
  21. - name: resourceGroupName
  22. value: "test-rg"
  23. - name: subscriptionID
  24. value: "value of Azure subscription ID"
  25. - name: partitionCount
  26. value: "1"
  27. - name: messageRetentionInDays
  28. value: "3"
  29. # Checkpoint store attributes
  30. - name: storageAccountName
  31. value: "myeventhubstorage"
  32. - name: storageAccountKey
  33. value: "112233445566778899"
  34. - name: storageContainerName
  35. value: "myeventhubstoragecontainer"
  36. # Alternative to passing storageAccountKey
  37. - name: storageConnectionString
  38. value: "DefaultEndpointsProtocol=https;AccountName=<account>;AccountKey=<account-key>"

Warning

The above example uses secrets as plain strings. It is recommended to use a secret store for the secrets as described here.

Spec metadata fields

FieldRequiredDetailsExample
connectionStringYConnection string for the Event Hub or the Event Hub namespace.
Mutally exclusive with eventHubNamespace field.
Required when not using Microsoft Entra ID Authentication
“Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}” or “Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key}”
eventHubNamespaceYThe Event Hub Namespace name.
Mutally exclusive with connectionString field.
Required when using Microsoft Entra ID Authentication
“namespace”
consumerIDNConsumer ID (consumer tag) organizes one or more consumers into a group. Consumers with the same consumer ID work as one virtual consumer; for example, a message is processed only once by one of the consumers in the group. If the consumerID is not provided, the Dapr runtime set it to the Dapr application ID (appID) value.Can be set to string value (such as “channel1” in the example above) or string format value (such as “{podName}”, etc.). See all of template tags you can use in your component metadata.
storageAccountNameYStorage account name to use for the checkpoint store.“myeventhubstorage”
storageAccountKeyYStorage account key for the checkpoint store account.
When using Microsoft Entra ID, it’s possible to omit this if the service principal has access to the storage account too.
“112233445566778899”
storageConnectionStringY*Connection string for the checkpoint store, alternative to specifying storageAccountKey“DefaultEndpointsProtocol=https;AccountName=myeventhubstorage;AccountKey=<account-key>”
storageContainerNameYStorage container name for the storage account name.“myeventhubstoragecontainer”
enableEntityManagementNBoolean value to allow management of the EventHub namespace and storage account. Default: false“true”, “false”
resourceGroupNameNName of the resource group the Event Hub namespace is part of. Required when entity management is enabled“test-rg”
subscriptionIDNAzure subscription ID value. Required when entity management is enabled“azure subscription id”
partitionCountNNumber of partitions for the new Event Hub namespace. Used only when entity management is enabled. Default: “1”“2”
messageRetentionInDaysNNumber of days to retain messages for in the newly created Event Hub namespace. Used only when entity management is enabled. Default: “1”“90”

Microsoft Entra ID authentication

The Azure Event Hubs pub/sub component supports authentication using all Microsoft Entra ID mechanisms. For further information and the relevant component metadata fields to provide depending on the choice of Microsoft Entra ID authentication mechanism, see the docs for authenticating to Azure.

Example Configuration

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: eventhubs-pubsub
  5. spec:
  6. type: pubsub.azure.eventhubs
  7. version: v1
  8. metadata:
  9. # Azure Authentication Used
  10. - name: azureTenantId
  11. value: "***"
  12. - name: azureClientId
  13. value: "***"
  14. - name: azureClientSecret
  15. value: "***"
  16. - name: eventHubNamespace
  17. value: "namespace"
  18. - name: enableEntityManagement
  19. value: "false"
  20. # The following four properties are needed only if enableEntityManagement is set to true
  21. - name: resourceGroupName
  22. value: "test-rg"
  23. - name: subscriptionID
  24. value: "value of Azure subscription ID"
  25. - name: partitionCount
  26. value: "1"
  27. - name: messageRetentionInDays
  28. # Checkpoint store attributes
  29. # In this case, we're using Microsoft Entra ID to access the storage account too
  30. - name: storageAccountName
  31. value: "myeventhubstorage"
  32. - name: storageContainerName
  33. value: "myeventhubstoragecontainer"

Sending and receiving multiple messages

Azure Eventhubs supports sending and receiving multiple messages in a single operation using the bulk pub/sub API.

Configuring bulk publish

To set the metadata for bulk publish operation, set the query parameters on the HTTP request or the gRPC metadata, as documented in the API reference.

MetadataDefault
metadata.maxBulkPubBytes1000000

Configuring bulk subscribe

When subscribing to a topic, you can configure bulkSubscribe options. Refer to Subscribing messages in bulk for more details and to learn more about the bulk subscribe API.

ConfigurationDefault
maxMessagesCount100
maxAwaitDurationMs10000

Configuring checkpoint frequency

When subscribing to a topic, you can configure the checkpointing frequency in a partition by setting the metadata in the HTTP or gRPC subscribe request. This metadata enables checkpointing after the configured number of events within a partition event sequence. Disable checkpointing by setting the frequency to 0.

Learn more about checkpointing.

MetadataDefault
metadata.checkPointFrequencyPerPartition1

Following example shows a sample subscription file for Declarative subscription using checkPointFrequencyPerPartition metadata. Similarly, you can also pass the metadata in Programmatic subscriptions as well.

  1. apiVersion: dapr.io/v2alpha1
  2. kind: Subscription
  3. metadata:
  4. name: order-pub-sub
  5. spec:
  6. topic: orders
  7. routes:
  8. default: /checkout
  9. pubsubname: order-pub-sub
  10. metadata:
  11. checkPointFrequencyPerPartition: 1
  12. scopes:
  13. - orderprocessing
  14. - checkout

Note

When subscribing to a topic using BulkSubscribe, you configure the checkpointing to occur after the specified number of batches, instead of events, where batch means the collection of events received in a single request.

Create an Azure Event Hub

Follow the instructions on the documentation to set up Azure Event Hubs.

Because this component uses Azure Storage as checkpoint store, you will also need an Azure Storage Account. Follow the instructions on the documentation to manage the storage account access keys.

See the documentation on how to get the Event Hubs connection string (note this is not for the Event Hubs namespace).

Create consumer groups for each subscriber

For every Dapr app that wants to subscribe to events, create an Event Hubs consumer group with the name of the Dapr app ID. For example, a Dapr app running on Kubernetes with dapr.io/app-id: "myapp" will need an Event Hubs consumer group named myapp.

Note: Dapr passes the name of the consumer group to the Event Hub, so this is not supplied in the metadata.

Entity Management

When entity management is enabled in the metadata, as long as the application has the right role and permissions to manipulate the Event Hub namespace, Dapr can automatically create the Event Hub and consumer group for you.

The Evet Hub name is the topic field in the incoming request to publish or subscribe to, while the consumer group name is the name of the Dapr app which subscribes to a given Event Hub. For example, a Dapr app running on Kubernetes with name dapr.io/app-id: "myapp" requires an Event Hubs consumer group named myapp.

Entity management is only possible when using Microsoft Entra ID Authentication and not using a connection string.

Dapr passes the name of the consumer group to the Event Hub, so this is not supplied in the metadata.

Subscribing to Azure IoT Hub Events

Azure IoT Hub provides an endpoint that is compatible with Event Hubs, so the Azure Event Hubs pubsub component can also be used to subscribe to Azure IoT Hub events.

The device-to-cloud events created by Azure IoT Hub devices will contain additional IoT Hub System Properties, and the Azure Event Hubs pubsub component for Dapr will return the following as part of the response metadata:

System Property NameDescription & Routing Query Keyword
iothub-connection-auth-generation-idThe connectionDeviceGenerationId of the device that sent the message. See IoT Hub device identity properties.
iothub-connection-auth-methodThe connectionAuthMethod used to authenticate the device that sent the message.
iothub-connection-device-idThe deviceId of the device that sent the message. See IoT Hub device identity properties.
iothub-connection-module-idThe moduleId of the device that sent the message. See IoT Hub device identity properties.
iothub-enqueuedtimeThe enqueuedTime in RFC3339 format that the device-to-cloud message was received by IoT Hub.
message-idThe user-settable AMQP messageId.

For example, the headers of a delivered HTTP subscription message would contain:

  1. {
  2. 'user-agent': 'fasthttp',
  3. 'host': '127.0.0.1:3000',
  4. 'content-type': 'application/json',
  5. 'content-length': '120',
  6. 'iothub-connection-device-id': 'my-test-device',
  7. 'iothub-connection-auth-generation-id': '637618061680407492',
  8. 'iothub-connection-auth-method': '{"scope":"module","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}',
  9. 'iothub-connection-module-id': 'my-test-module-a',
  10. 'iothub-enqueuedtime': '2021-07-13T22:08:09Z',
  11. 'message-id': 'my-custom-message-id',
  12. 'x-opt-sequence-number': '35',
  13. 'x-opt-enqueued-time': '2021-07-13T22:08:09Z',
  14. 'x-opt-offset': '21560',
  15. 'traceparent': '00-4655608164bc48b985b42d39865f3834-ed6cf3697c86e7bd-01'
  16. }

Last modified October 11, 2024: Fixed typo (#4389) (fe17926)