Circuit Breakers

Circuit breaking is a powerful feature where Linkerd will temporarily stop routing requests to an endpoint if that endpoint is deemed to be unhealthy, instead routing that request to other replicas in the Service.

In this tutoral, we’ll see how to enable circuit breaking on a Service to improve client success rate when a backend replica is unhealthy.

See the reference documentation for more details on how Linkerd implements circuit breaking.

Circuit Breakers - 图1

Linkerd Production Tip

This page contains best-effort instructions by the open source community. Production users with mission-critical applications should familiarize themselves with Linkerd production resources and/or connect with a commercial Linkerd provider.

Prerequisites

To use this guide, you’ll need a Kubernetes cluster running:

Set up the demo

Remember those puzzles where one guard always tells the truth and one guard always lies? This demo involves one pod (named good) which always returns an HTTP 200 and one pod (named bad) which always returns an HTTP 500. We’ll also create a load generator to send traffic to a Service which includes these two pods.

For load generation we’ll use Slow-Cooker and for the backend pods we’ll use BB.

To add these components to your cluster and include them in the Linkerd data plane, run:

  1. cat <<EOF | linkerd inject - | kubectl apply -f -
  2. ---
  3. apiVersion: v1
  4. kind: Namespace
  5. metadata:
  6.   name: circuit-breaking-demo
  7. ---
  8. apiVersion: apps/v1
  9. kind: Deployment
  10. metadata:
  11.   name: good
  12.   namespace: circuit-breaking-demo
  13. spec:
  14.   replicas: 1
  15.   selector:
  16.     matchLabels:
  17.       class: good
  18.   template:
  19.     metadata:
  20.       labels:
  21.         class: good
  22.         app: bb
  23.     spec:
  24.       containers:
  25.       - name: terminus
  26.         image: buoyantio/bb:v0.0.6
  27.         args:
  28.         - terminus
  29.         - "--h1-server-port=8080"
  30.         ports:
  31.         - containerPort: 8080
  32. ---
  33. apiVersion: apps/v1
  34. kind: Deployment
  35. metadata:
  36.   name: bad
  37.   namespace: circuit-breaking-demo
  38. spec:
  39.   replicas: 1
  40.   selector:
  41.     matchLabels:
  42.       class: bad
  43.   template:
  44.     metadata:
  45.       labels:
  46.         class: bad
  47.         app: bb
  48.     spec:
  49.       containers:
  50.       - name: terminus
  51.         image: buoyantio/bb:v0.0.6
  52.         args:
  53.         - terminus
  54.         - "--h1-server-port=8080"
  55.         - "--percent-failure=100"
  56.         ports:
  57.         - containerPort: 8080
  58. ---
  59. apiVersion: v1
  60. kind: Service
  61. metadata:
  62.   name: bb
  63.   namespace: circuit-breaking-demo
  64. spec:
  65.   ports:
  66.   - name: http
  67.     port: 8080
  68.     targetPort: 8080
  69.   selector:
  70.     app: bb
  71. ---
  72. apiVersion: apps/v1
  73. kind: Deployment
  74. metadata:
  75.   name: slow-cooker
  76.   namespace: circuit-breaking-demo
  77. spec:
  78.   replicas: 1
  79.   selector:
  80.     matchLabels:
  81.       app: slow-cooker
  82.   template:
  83.     metadata:
  84.       labels:
  85.         app: slow-cooker
  86.     spec:
  87.       containers:
  88.       - args:
  89.         - -c
  90.         - |
  91.           sleep 5 # wait for pods to start
  92.           /slow_cooker/slow_cooker --qps 10 http://bb:8080
  93.         command:
  94.         - /bin/sh
  95.         image: buoyantio/slow_cooker:1.3.0
  96.         name: slow-cooker
  97. EOF

We can now look at the success rate of the good and bad pods:

  1. > linkerd viz -n circuit-breaking-demo stat deploy
  2. NAME          MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
  3. bad              1/1     6.43%   4.7rps           1ms           1ms           4ms          2
  4. good             1/1   100.00%   5.9rps           1ms           1ms           1ms          3
  5. slow-cooker      1/1   100.00%   0.3rps           1ms           1ms           1ms          1

Here we can see that good and bad deployments are each receiving similar amounts of traffic, but good has a success rate of 100% while the success rate of bad is very low (only healthcheck probes are succeeding). We can also see how this looks from the perspective of the traffic generator:

  1. > linkerd viz -n circuit-breaking-demo stat deploy/slow-cooker --to svc/bb
  2. NAME          MESHED   SUCCESS       RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
  3. slow-cooker      1/1    51.00%   10.0rps           1ms           1ms           2ms          2

From slow-cooker’s perspective, roughly 50% of requests that it sends to the Service are failing. We can use circuit breaking to improve this by cutting off traffic to the bad pod.

Breaking the circuit

Linkerd supports a type of circuit breaking called consecutive failure accrual. This works by tracking consecutive failures from each endpoint in Linkerd’s internal load balancer. If there are ever too many failures in a row, that endpoint is temporarily ignored and Linkerd will only load balance among the remaining endpoints. After a backoff period, the endpoint is re-introduced so that we can determine if it has become healthy.

Let’s enable consecutive failure accrual on the bb Service by adding an annotation:

  1. kubectl annotate -n circuit-breaking-demo svc/bb balancer.linkerd.io/failure-accrual=consecutive

Circuit Breakers - 图2

Warning

Circuit breaking is incompatible with ServiceProfiles. If a ServiceProfile is defined for the annotated Service, proxies will not perform circuit breaking as long as the ServiceProfile exists.

We can check that failure accrual was configured correctly by using a Linkerd diagnostics command. The linkerd diagnostics policy command prints the policy that Linkerd will use when sending traffic to a Service. We’ll use the jq utility to filter the output to focus on failure accrual:

  1. > linkerd diagnostics policy -n circuit-breaking-demo svc/bb 8080 -o json | jq '.protocol.Kind.Detect.http1.failure_accrual'
  2. {
  3.   "Kind": {
  4.     "ConsecutiveFailures": {
  5.       "max_failures": 7,
  6.       "backoff": {
  7.         "min_backoff": {
  8.           "seconds": 1
  9.         },
  10.         "max_backoff": {
  11.           "seconds": 60
  12.         },
  13.         "jitter_ratio": 0.5
  14.       }
  15.     }
  16.   }
  17. }

This tells us that Linkerd will use ConsecutiveFailures failure accrual when talking to the bb Service. It also tells us that the max_failures is 7, meaning that it will trip the circuit breaker once it observes 7 consective failures. We’ll talk more about each of the parameters here at the end of this article.

Let’s look at how much traffic each pod is getting now that the circuit breaker is in place:

  1. > linkerd viz -n circuit-breaking-demo stat deploy
  2. NAME          MESHED   SUCCESS       RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
  3. bad              1/1    94.74%    0.3rps           1ms           1ms           1ms          3
  4. good             1/1   100.00%   10.3rps           1ms           1ms           4ms          4
  5. slow-cooker      1/1   100.00%    0.3rps           1ms           1ms           1ms          1

Notice that the bad pod’s RPS is significantly lower now. The circuit breaker has stopped nearly all of the traffic from slow-cooker to bad.

We can also see how this has affected slow-cooker:

  1. > linkerd viz -n circuit-breaking-demo stat deploy/slow-cooker --to svc/bb
  2. NAME          MESHED   SUCCESS       RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
  3. slow-cooker      1/1    99.83%   10.0rps           1ms           1ms           1ms          4

Nearly all of slow-cooker’s requests are now getting routed to the good pod and succeeding!

Tuning circuit breaking

As we saw when we ran the linkerd diagnostics policy command, consecutive failure accrual is controlled by a number of parameters. Each of these parameters has a default, but can be manually configured using annotations:

  • balancer.linkerd.io/failure-accrual-consecutive-max-failures
    • The number of consecutive failures that Linkerd must observe before tripping the circuit breaker (default: 7). Consider setting a lower value if you want circuit breaks to trip more easily which can lead to better success rate at the expense of less evenly distributed traffic. Consider setting a higher value if you find circuit breakers are tripping too easily, causing traffic to be cut off from healthy endpoints.
  • balancer.linkerd.io/failure-accrual-consecutive-max-penalty
    • The maximum amount of time a circuit breaker will remain tripped before the endpoint is restored (default: 60s). Consider setting a longer duration if you want to reduce the amount of traffic to endpoints which have tripped the circuit breaker. Consider setting a shorter duration if you’d like tripped circuit breakers to recover faster after an endpoint becomes healthy again.
  • balancer.linkerd.io/failure-accrual-consecutive-min-penalty
    • The minimum amount of time a circuit breaker will remain tripped before the endpoints is restored (default: 1s). Consider tuning this in a similar way to failure-accrual-consecutive-max-penalty.
  • balancer.linkerd.io/failure-accrual-consecutive-jitter-ratio
    • The amount of jitter to introduce to circuit breaker backoffs (default: 0.5). You are unlikely to need to tune this but might consider increasing it if you notice many clients are sending requests to a circuit broken endpoint at the same time, leading to spiky traffic patterns.

See the reference documentation for details on failure accrual configuration.