AWS SNS/SQS

Detailed documentation on the AWS SNS/SQS pubsub component

Component format

To setup AWS SNS/SQS for pub/sub, you create a component of type pubsub.snssqs. See this guide on how to create and apply a pubsub configuration.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4.   name: snssqs-pubsub
  5.   namespace: default
  6. spec:
  7.   type: pubsub.snssqs
  8.   version: v1
  9.   metadata:
  10.     - name: accessKey
  11.       value: "AKIAIOSFODNN7EXAMPLE"
  12.     - name: secretKey
  13.       value: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  14.     - name: region
  15.       value: "us-east-1"
  16.     - name: sessionToken
  17.       value: "TOKEN"
  18.     - name: messageVisibilityTimeout
  19.       value: 10
  20.     - name: messageRetryLimit
  21.       value: 10
  22.     - name: messageWaitTimeSeconds
  23.       value: 1
  24.     - name: messageMaxNumber
  25.       value: 10

Warning

The above example uses secrets as plain strings. It is recommended to use a secret store for the secrets as described here.

Spec metadata fields

FieldRequiredDetailsExample
accessKeyYID of the AWS account with appropriate permissions to SNS and SQS. Can be secretKeyRef to use a secret reference“AKIAIOSFODNN7EXAMPLE”
secretKeyYSecret for the AWS user. Can be secretKeyRef to use a secret reference“wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY”
regionYThe AWS region to the instance. See this page for valid regions: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html. Ensure that SNS and SQS are available in that region.“us-east-1”
endpointNAWS endpoint for the component to use. Only used for local development. The endpoint is unncessary when running against production AWS“http://localhost:4566“
sessionTokenNAWS session token to use. A session token is only required if you are using temporary security credentials“TOKEN”
messageVisibilityTimeoutNAmount of time in seconds that a message is hidden from receive requests after it is sent to a subscriber. Default: 1010
messageRetryLimitNNumber of times to resend a message after processing of that message fails before removing that message from the queue. Default: 1010
messageWaitTimeSecondsNamount of time to await receipt of a message before making another request. Default: 11
messageMaxNumberNmaximum number of messages to receive from the queue at a time. Default: 10, Maximum: 1010

Create an SNS/SQS instance

For local development the localstack project is used to integrate AWS SNS/SQS. Follow the instructions here to install the localstack CLI.

In order to use localstack with your pubsub binding, you need to provide the endpoint configuration in the component metadata. The endpoint is unncessary when running against production AWS.

See Authenticating to AWS for information about authentication-related attributes

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4.   name: snssqs-pubsub
  5. spec:
  6.   type: pubsub.snssqs
  7.   version: v1
  8.   metadata:
  9.     - name: endpoint
  10.       value: http://localhost:4566
  11.     # Use us-east-1 for localstack
  12.     - name: region
  13.       value: us-east-1

To run localstack on Kubernetes, you can apply the configuration below. Localstack is then reachable at the DNS name http://localstack.default.svc.cluster.local:4566 (assuming this was applied to the default namespace) and this should be used as the endpoint

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: localstack
  5.   namespace: default
  6. spec:
  7.   # using the selector, we will expose the running deployments
  8.   # this is how Kubernetes knows, that a given service belongs to a deployment
  9.   selector:
  10.     matchLabels:
  11.       app: localstack
  12.   replicas: 1
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: localstack
  17.     spec:
  18.       containers:
  19.       - name: localstack
  20.         image: localstack/localstack:latest
  21.         ports:
  22.           # Expose the edge endpoint
  23.           - containerPort: 4566
  24. ---
  25. kind: Service
  26. apiVersion: v1
  27. metadata:
  28.   name: localstack
  29.   labels:
  30.     app: localstack
  31. spec:
  32.   selector:
  33.     app: localstack
  34.   ports:
  35.   - protocol: TCP
  36.     port: 4566
  37.     targetPort: 4566
  38.   type: LoadBalancer

In order to run in AWS, you should create an IAM user with permissions to the SNS and SQS services. Use the AWS account ID and AWS account secret and plug them into the accessKey and secretKey in the component metadata using Kubernetes secrets and secretKeyRef.

Last modified September 20, 2021 : Merge pull request #1800 from greenie-msft/gRPC_proxying_video (36dff3c)