Injecting Faults

It is easy to inject failures into applications by using the HTTPRoute resource to redirect a percentage of traffic to a specific backend. This backend is completely flexible and can return whatever responses you want - 500s, timeouts or even crazy payloads.

The books demo is a great way to show off this behavior. The overall topology looks like:

Topology
Topology

In this guide, you will split some of the requests from webapp to books. Most requests will end up at the correct books destination, however some of them will be redirected to a faulty backend. This backend will return 500s for every request and inject faults into the webapp service. No code changes are required and as this method is configuration driven, it is a process that can be added to integration tests and CI pipelines. If you are really living the chaos engineering lifestyle, fault injection could even be used in production.

Prerequisites

To use this guide, you’ll need a Kubernetes cluster running:

Setup the service

First, add the books sample application to your cluster:

  1. kubectl create ns booksapp && \
  2.   linkerd inject https://run.linkerd.io/booksapp.yml | \
  3.   kubectl -n booksapp apply -f -

As this manifest is used as a demo elsewhere, it has been configured with an error rate. To show how fault injection works, the error rate needs to be removed so that there is a reliable baseline. To increase success rate for booksapp to 100%, run:

  1. kubectl -n booksapp patch deploy authors \
  2.   --type='json' \
  3.   -p='[{"op":"remove", "path":"/spec/template/spec/containers/0/env/2"}]'

After a little while, the stats will show 100% success rate. You can verify this by running:

  1. linkerd viz -n booksapp stat-inbound deploy

The output will end up looking at little like:

  1. NAME     SERVER          ROUTE      TYPE  SUCCESS   RPS  LATENCY_P50  LATENCY_P95  LATENCY_P99  
  2. authors  [default]:4191  [default]        100.00%  0.20          0ms          1ms          1ms  
  3. authors  [default]:7001  [default]        100.00%  3.00          2ms         36ms         43ms  
  4. books    [default]:4191  [default]        100.00%  0.23          4ms          4ms          4ms  
  5. books    [default]:7002  [default]        100.00%  3.60          2ms          2ms          2ms  
  6. traffic  [default]:4191  [default]        100.00%  0.22          0ms          3ms          1ms  
  7. webapp   [default]:4191  [default]        100.00%  0.72          4ms          5ms          1ms  
  8. webapp   [default]:7000  [default]        100.00%  3.25          2ms          2ms         65ms

Create the faulty backend

Injecting faults into booksapp requires a service that is configured to return errors. To do this, you can start NGINX and configure it to return 500s by running:

  1. cat <<EOF | linkerd inject - | kubectl apply -f -
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: error-injector
  6.   namespace: booksapp
  7. data:
  8.  nginx.conf: |-
  9.     events {}
  10.     http {
  11.         server {
  12.           listen 8080;
  13.             location / {
  14.                 return 500;
  15.             }
  16.         }
  17.     }
  18. ---
  19. apiVersion: apps/v1
  20. kind: Deployment
  21. metadata:
  22.   name: error-injector
  23.   namespace: booksapp
  24.   labels:
  25.     app: error-injector
  26. spec:
  27.   selector:
  28.     matchLabels:
  29.       app: error-injector
  30.   replicas: 1
  31.   template:
  32.     metadata:
  33.       labels:
  34.         app: error-injector
  35.     spec:
  36.       containers:
  37.         - name: nginx
  38.           image: nginx:alpine
  39.           volumeMounts:
  40.             - name: nginx-config
  41.               mountPath: /etc/nginx/nginx.conf
  42.               subPath: nginx.conf
  43.       volumes:
  44.         - name: nginx-config
  45.           configMap:
  46.             name: error-injector
  47. ---
  48. apiVersion: v1
  49. kind: Service
  50. metadata:
  51.   name: error-injector
  52.   namespace: booksapp
  53. spec:
  54.   ports:
  55.   - name: service
  56.     port: 8080
  57.   selector:
  58.     app: error-injector
  59. EOF

Inject faults

With booksapp and NGINX running, it is now time to partially split the traffic between an existing backend, books, and the newly created error-injector. This is done by adding an HTTPRoute configuration to your cluster:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: policy.linkerd.io/v1beta2
  3. kind: HTTPRoute
  4. metadata:
  5.   name: error-split
  6.   namespace: booksapp
  7. spec:
  8.   parentRefs:
  9.     - name: books
  10.       kind: Service
  11.       group: core
  12.       port: 7002
  13.   rules:
  14.     - backendRefs:
  15.       - name: books
  16.         port: 7002
  17.         weight: 90
  18.       - name: error-injector
  19.         port: 8080
  20.         weight: 10
  21. EOF

Injecting Faults - 图2

Note

Two versions of the HTTPRoute resource may be used with Linkerd:

  • The upstream version provided by the Gateway API, with the gateway.networking.k8s.io API group
  • A Linkerd-specific CRD provided by Linkerd, with the policy.linkerd.io API group

The two HTTPRoute resource definitions are similar, but the Linkerd version implements experimental features not yet available with the upstream Gateway API resource definition. See the HTTPRoute reference documentation for details.

When Linkerd sees traffic going to the books service, it will send 9/10 requests to the original service and 1/10 to the error injector. You can see what this looks like by running stat-outbound:

  1. linkerd viz stat-outbound -n booksapp deploy/webapp
  2. NAME    SERVICE       ROUTE        TYPE       BACKEND              SUCCESS   RPS  LATENCY_P50  LATENCY_P95  LATENCY_P99  TIMEOUTS  RETRIES  
  3. webapp  authors:7001  [default]                                     98.44%  4.28         25ms         47ms         50ms     0.00%    0.00%  
  4.                       └────────────────────►  authors:7001          98.44%  4.28         15ms         42ms         48ms     0.00%           
  5. webapp  books:7002    error-split  HTTPRoute                        87.76%  7.22         26ms         49ms        333ms     0.00%    0.00%  
  6.                       ├────────────────────►  books:7002           100.00%  6.33         14ms         42ms         83ms     0.00%           
  7.                       └────────────────────►  error-injector:8080    0.00%  0.88         12ms         24ms         25ms     0.00%

We can see here that 0.88 requests per second are being sent to the error injector and that the overall success rate is 87.76%.

Cleanup

To remove everything in this guide from your cluster, run:

  1. kubectl delete ns booksapp