GCP

Detailed documentation on the GCP Pub/Sub component

Create a Dapr component

To set up GCP pub/sub, create a component of type pubsub.gcp.pubsub. See the pub/sub broker component file to learn how ConsumerID is automatically generated. Read the How-to: Publish and Subscribe guide on how to create and apply a pub/sub configuration.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4.   name: gcp-pubsub
  5. spec:
  6.   type: pubsub.gcp.pubsub
  7.   version: v1
  8.   metadata:
  9.   - name: type
  10.     value: service_account
  11.   - name: projectId
  12.     value: <PROJECT_ID> # replace
  13.   - name: endpoint # Optional. 
  14.     value: "http://localhost:8085"
  15.   - name: consumerID # Optional - defaults to the app's own ID
  16.     value: <CONSUMER_ID> 
  17.   - name: identityProjectId
  18.     value: <IDENTITY_PROJECT_ID> # replace
  19.   - name: privateKeyId
  20.     value: <PRIVATE_KEY_ID> #replace
  21.   - name: clientEmail
  22.     value: <CLIENT_EMAIL> #replace
  23.   - name: clientId
  24.     value: <CLIENT_ID> # replace
  25.   - name: authUri
  26.     value: https://accounts.google.com/o/oauth2/auth
  27.   - name: tokenUri
  28.     value: https://oauth2.googleapis.com/token
  29.   - name: authProviderX509CertUrl
  30.     value: https://www.googleapis.com/oauth2/v1/certs
  31.   - name: clientX509CertUrl
  32.     value: https://www.googleapis.com/robot/v1/metadata/x509/<PROJECT_NAME>.iam.gserviceaccount.com #replace PROJECT_NAME
  33.   - name: privateKey
  34.     value: <PRIVATE_KEY> # replace x509 cert
  35.   - name: disableEntityManagement
  36.     value: "false"
  37.   - name: enableMessageOrdering
  38.     value: "false"  
  39.   - name: orderingKey # Optional
  40.     value: <ORDERING_KEY>
  41.   - name: maxReconnectionAttempts # Optional
  42.     value: 30
  43.   - name: connectionRecoveryInSec # Optional
  44.     value: 2
  45.   - name: deadLetterTopic # Optional
  46.     value: <EXISTING_PUBSUB_TOPIC>
  47.   - name: maxDeliveryAttempts # Optional
  48.     value: 5

Warning

The above example uses secrets as plain strings. It is recommended to use a secret store for the secrets as described here.

Spec metadata fields

FieldRequiredDetailsExample
projectIdYGCP project idmyproject-123
endpointNGCP endpoint for the component to use. Only used for local development (for example) with GCP Pub/Sub Emulator. The endpoint is unnecessary when running against the GCP production API.“http://localhost:8085“
consumerIDNThe Consumer ID organizes one or more consumers into a group. Consumers with the same consumer ID work as one virtual consumer; for example, a message is processed only once by one of the consumers in the group. If the consumerID is not provided, the Dapr runtime set it to the Dapr application ID (appID) value. The consumerID, along with the topic provided as part of the request, are used to build the Pub/Sub subscription ID
identityProjectIdNIf the GCP pubsub project is different from the identity project, specify the identity project using this attribute“myproject-123”
privateKeyIdNIf using explicit credentials, this field should contain the private_key_id field from the service account json document“my-private-key”
privateKeyNIf using explicit credentials, this field should contain the private_key field from the service account json——-BEGIN PRIVATE KEY——-MIIBVgIBADANBgkqhkiG9w0B
clientEmailNIf using explicit credentials, this field should contain the client_email field from the service account json“myservice@myproject-123.iam.gserviceaccount.com”
clientIdNIf using explicit credentials, this field should contain the client_id field from the service account json106234234234
authUriNIf using explicit credentials, this field should contain the auth_uri field from the service account jsonhttps://accounts.google.com/o/oauth2/auth
tokenUriNIf using explicit credentials, this field should contain the token_uri field from the service account jsonhttps://oauth2.googleapis.com/token
authProviderX509CertUrlNIf using explicit credentials, this field should contain the auth_provider_x509_cert_url field from the service account jsonhttps://www.googleapis.com/oauth2/v1/certs
clientX509CertUrlNIf using explicit credentials, this field should contain the client_x509_cert_url field from the service account jsonhttps://www.googleapis.com/robot/v1/metadata/x509/myserviceaccount%40myproject.iam.gserviceaccount.com
disableEntityManagementNWhen set to “true”, topics and subscriptions do not get created automatically. Default: “false”“true”, “false”
enableMessageOrderingNWhen set to “true”, subscribed messages will be received in order, depending on publishing and permissions configuration.“true”, “false”
orderingKeyNThe key provided in the request. It’s used when enableMessageOrdering is set to true to order messages based on such key.“my-orderingkey”
maxReconnectionAttemptsNDefines the maximum number of reconnect attempts. Default: 3030
connectionRecoveryInSecNTime in seconds to wait between connection recovery attempts. Default: 22
deadLetterTopicNName of the GCP Pub/Sub Topic. This topic must exist before using this component.“myapp-dlq”
maxDeliveryAttemptsNMaximum number of attempts to deliver the message. If deadLetterTopic is specified, maxDeliveryAttempts is the maximum number of attempts for failed processing of messages. Once that number is reached, the message will be moved to the dead-letter topic. Default: 55
typeNDEPRECATED GCP credentials type. Only service_account is supported. Defaults to service_accountservice_account

Warning

If enableMessageOrdering is set to “true”, the roles/viewer or roles/pubsub.viewer role will be required on the service account in order to guarantee ordering in cases where order tokens are not embedded in the messages. If this role is not given, or the call to Subscription.Config() fails for any other reason, ordering by embedded order tokens will still function correctly.

GCP Credentials

Since the GCP Pub/Sub component uses the GCP Go Client Libraries, by default it authenticates using Application Default Credentials. This is explained further in the Authenticate to GCP Cloud services using client libraries guide.

Create a GCP Pub/Sub

For local development, the GCP Pub/Sub Emulator is used to test the GCP Pub/Sub Component. Follow these instructions to run the GCP Pub/Sub Emulator.

To run the GCP Pub/Sub Emulator locally using Docker, use the following docker-compose.yaml:

  1. version: '3'
  2. services:
  3.   pubsub:
  4.     image: gcr.io/google.com/cloudsdktool/cloud-sdk:422.0.0-emulators
  5.     ports:
  6.       - "8085:8085"
  7.     container_name: gcp-pubsub
  8.     entrypoint: gcloud beta emulators pubsub start --project local-test-prj --host-port 0.0.0.0:8085

In order to use the GCP Pub/Sub Emulator with your pub/sub binding, you need to provide the endpoint configuration in the component metadata. The endpoint is unnecessary when running against the GCP Production API.

The projectId attribute must match the --project used in either the docker-compose.yaml or Docker command.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4.   name: gcp-pubsub
  5. spec:
  6.   type: pubsub.gcp.pubsub
  7.   version: v1
  8.   metadata:
  9.   - name: projectId
  10.     value: "local-test-prj"
  11.   - name: consumerID
  12.     value: "testConsumer"
  13.   - name: endpoint
  14.     value: "localhost:8085"

You can use either “explicit” or “implicit” credentials to configure access to your GCP pubsub instance. If using explicit, most fields are required. Implicit relies on dapr running under a Kubernetes service account (KSA) mapped to a Google service account (GSA) which has the necessary permissions to access pubsub. In implicit mode, only the projectId attribute is needed, all other are optional.

Follow the instructions here on setting up Google Cloud Pub/Sub system.

Last modified June 19, 2023: Merge pull request #3565 from dapr/aacrawfi/skip-secrets-close (b1763bf)