Chaos Testing

Apply some chaos engineering by throwing in some HTTP errors or network delays. Understanding failure scenarios is a critical aspect of microservices architecture (aka distributed computing)

Before StartYou should have NO virtualservice nor destinationrule (in tutorial namespace) kubectl get virtualservice kubectl get destinationruleif so run:
  1. ./scripts/clean.sh tutorial

HTTP Error 503

By default, recommendation v1 and v2 are being randomly load-balanced as that is the default behavior in Kubernetes/OpenShift

  1. $ oc get pods -l app=recommendation -n tutorial
  2. or
  3. $ kubectl get pods -l app=recommendation -n tutorial
  5. NAME                                  READY     STATUS    RESTARTS   AGE
  6. recommendation-v1-3719512284-7mlzw   2/2       Running   6          18h
  7. recommendation-v2-2815683430-vn77w   2/2       Running   0          3h

You can inject 503’s, for approximately 50% of the requests

  1. kubectl create -f istiofiles/destination-rule-recommendation.yml -n tutorial
  2. kubectl create -f istiofiles/virtual-service-recommendation-503.yml -n tutorial
  4. ./scripts/run.sh http://istio-ingressgateway-istio-system.$(minishift ip).nip.io/customer
  6. customer => preference => recommendation v2 from '7778d6fb89-j6lts': 138
  7. customer => Error: 503 - preference => Error: 503 - fault filter abort
  8. customer => preference => recommendation v1 from '6976858b48-dswhz': 139
  9. customer => Error: 503 - preference => Error: 503 - fault filter abort
  10. customer => preference => recommendation v2 from '7778d6fb89-j6lts': 139
  11. customer => preference => recommendation v1 from '6976858b48-dswhz': 140
  12. customer => preference => recommendation v2 from '7778d6fb89-j6lts': 140
  13. customer => preference => recommendation v1 from '6976858b48-dswhz': 141
  14. customer => Error: 503 - preference => Error: 503 - fault filter abort
  15. customer => Error: 503 - preference => Error: 503 - fault filter abort
  16. customer => preference => recommendation v2 from '7778d6fb89-j6lts': 141

Clean up

  1. kubectl delete -f istiofiles/virtual-service-recommendation-503.yml -n tutorial

Delay

The most insidious of possible distributed computing faults is not a "down" service but a service that is responding slowly, potentially causing a cascading failure in your network of services.

  1. kubectl create -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial
  2. kubectl replace -f istiofiles/destination-rule-recommendation.yml -n tutorial

And hit the customer endpoint

  1. ./scripts/run.sh http://istio-ingressgateway-istio-system.$(minishift ip).nip.io/customer

You will notice many requests to the customer endpoint now have a delay.If you are monitoring the logs for recommendation v1 and v2, you will also see the delay happens BEFORE the recommendation service is actually called

  1. stern recommendation -n tutorial

orbash./kubetail.sh recommendation -n tutorial

Clean up

  1. kubectl delete -f istiofiles/destination-rule-recommendation.yml -n tutorial
  2. kubectl delete -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial