Overview

Configure Dapr retries, timeouts, and circuit breakers

Note

Resiliency is currently a preview feature. Before you can utilize a resiliency spec, you must first enable the resiliency preview feature.

Distributed applications are commonly comprised of many microservices, with dozens, even hundreds, of instances for any given application. With so many microservices, the likelihood of a system failure increases. For example, an instance can fail or be unresponsive due to hardware, an overwhelming number of requests, application restarts/scale outs, or several other reasons. These events can cause a network call between services to fail. Designing and implementing your application with fault tolerance, the ability to detect, mitigate, and respond to failures, allows your application to recover to a functioning state and become self healing.

Dapr provides a capability for defining and applying fault tolerance resiliency policies via a resiliency spec. Resiliency specs are saved in the same location as components specs and are applied when the Dapr sidecar starts. The sidecar determines how to apply resiliency policies to your Dapr API calls. In self-hosted mode, the resiliency spec must be named resiliency.yaml. In Kubernetes Dapr finds the named resiliency specs used by your application. Within the resiliency spec, you can define policies for popular resiliency patterns, such as:

Policies can then be applied to targets, which include:

Additionally, resiliency policies can be scoped to specific apps.

Below is the general structure of a resiliency policy:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Resiliency
  3. metadata:
  4.   name: myresiliency
  5. scopes:
  6.   # optionally scope the policy to specific apps
  7. spec:
  8.   policies:
  9.     timeouts:
  10.       # timeout policy definitions
  12.     retries:
  13.       # retry policy definitions
  15.     circuitBreakers:
  16.       # circuit breaker policy definitions
  18.   targets:
  19.     apps:
  20.       # apps and their applied policies here
  22.     actors:
  23.       # actor types and their applied policies here
  25.     components:
  26.       # components and their applied policies here

Complete example policy

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Resiliency
  3. metadata:
  4.   name: myresiliency
  5. # similar to subscription and configuration specs, scopes lists the Dapr App IDs that this
  6. # resiliency spec can be used by.
  7. scopes:
  8.   - app1
  9.   - app2
  10. spec:
  11.   # policies is where timeouts, retries and circuit breaker policies are defined. 
  12.   # each is given a name so they can be referred to from the targets section in the resiliency spec.
  13.   policies:
  14.     # timeouts are simple named durations.
  15.     timeouts:
  16.       general: 5s
  17.       important: 60s
  18.       largeResponse: 10s
  20.     # retries are named templates for retry configurations and are instantiated for life of the operation.
  21.     retries:
  22.       pubsubRetry:
  23.         policy: constant
  24.         duration: 5s
  25.         maxRetries: 10
  27.       retryForever:
  28.         policy: exponential
  29.         maxInterval: 15s
  30.         maxRetries: -1 # retry indefinitely
  32.       important:
  33.         policy: constant
  34.         duration: 5s
  35.         maxRetries: 30
  37.       someOperation:
  38.         policy: exponential
  39.         maxInterval: 15s
  41.       largeResponse:
  42.         policy: constant
  43.         duration: 5s
  44.         maxRetries: 3
  46.     # circuit breakers are automatically instantiated per component and app instance.
  47.     # circuit breakers maintain counters that live as long as the Dapr sidecar is running. They are not persisted.
  48.     circuitBreakers:
  49.       simpleCB:
  50.         maxRequests: 1
  51.         timeout: 30s 
  52.         trip: consecutiveFailures >= 5
  54.       pubsubCB:
  55.         maxRequests: 1
  56.         interval: 8s
  57.         timeout: 45s
  58.         trip: consecutiveFailures > 8
  60.   # targets are what named policies are applied to. Dapr supports 3 target types - apps, components and actors
  61.   targets:
  62.     apps:
  63.       appB:
  64.         timeout: general
  65.         retry: important
  66.         # circuit breakers for services are scoped app instance.
  67.         # when a breaker is tripped, that route is removed from load balancing for the configured `timeout` duration.
  68.         circuitBreaker: simpleCB
  70.     actors:
  71.       myActorType: # custom Actor Type Name
  72.         timeout: general
  73.         retry: important
  74.         # circuit breakers for actors are scoped by type, id, or both.
  75.         # when a breaker is tripped, that type or id is removed from the placement table for the configured `timeout` duration.
  76.         circuitBreaker: simpleCB
  77.         circuitBreakerScope: both ## 
  78.         circuitBreakerCacheSize: 5000
  80.     components:
  81.       # for state stores, policies apply to saving and retrieving state.
  82.       statestore1: # any component name -- happens to be a state store here
  83.         outbound:
  84.           timeout: general
  85.           retry: retryForever
  86.           # circuit breakers for components are scoped per component configuration/instance. For example myRediscomponent.
  87.           # when this breaker is tripped, all interaction to that component is prevented for the configured `timeout` duration.
  88.           circuitBreaker: simpleCB
  90.       pubsub1: # any component name -- happens to be a pubsub broker here
  91.         outbound:
  92.           retry: pubsubRetry
  93.           circuitBreaker: pubsubCB
  95.       pubsub2: # any component name -- happens to be another pubsub broker here
  96.         outbound:
  97.           retry: pubsubRetry
  98.           circuitBreaker: pubsubCB
  99.         inbound: # inbound only applies to delivery from sidecar to app
  100.           timeout: general
  101.           retry: important
  102.           circuitBreaker: pubsubCB

Watch this video for how to use resiliency:

Last modified October 12, 2022: Update resiliency-overview.md (e87fa09f)