Diferencia and Detecting Regressions

Before StartYou should have NO virtualservice nor destinationrule (in tutorial namespace) kubectl get virtualservice kubectl get destinationruleif so run:
  1. ./scripts/clean.sh

What is Diferencia?

Diferencia is designed to act as a proxy which intercepts calls to a service and multicast it to several instances of the same service.These instances are an old and new version of the service. Then when the response is returned from each instance, Diferencia will check if both responses are similar, and if it is the case, then the two implementations might be considered compatible and the new version implementation is regression-free.

Overview of Diferencia

In the previous schema, you can see that a request is multicasted to two instances of Service A, one being V1 and to other V2.Both return a response and then it is compared by Diferencia.Finally, notice that no response is returned (by default) but the result (in form of HTTP Status Code).

Since Diferencia does not return a real response, but a comparison, effectively means that it is a perfect fit for Traffic Shadowing/Mirroring technique.

Selecting Master and Candidate

The first thing you need to do is selecting a master service (the one you know it works) and a candidate service (the one you want to release) and create a Kubernetes service.For this case, master is recommendation:v1 and candidate is recommendation:v2.

  1. oc create -f diferencia/recommendation-master-svc.yml -n tutorial
  2. oc create -f diferencia/recommendation-candidate-svc.yml -n tutorial
  4. or
  6. kubectl create -f diferencia/recommendation-master-svc.yml -n tutorial
  7. kubectl create -f diferencia/recommendation-candidate-svc.yml -n tutorial

Deploying Diferencia

Next thing is to deploy Diferencia.

  1. oc apply -f <(istioctl kube-inject -f diferencia/Deployment.yml) -n tutorial
  2. oc create -f diferencia/Service.yml -n tutorial
  4. or
  6. kubectl apply -f <(istioctl kube-inject -f diferencia/Deployment.yml) -n tutorial
  7. kubectl create -f diferencia/Service.yml -n tutorial

Shadowing Traffic

The last thing is to just start shadowing traffic between recommendation:v1 and diferencia.

  1. istioctl create -f istiofiles/destination-rule-recommendation-v1-v2.yml -n tutorial
  3. istioctl create -f diferencia/virtual-service-recommendation-v1-diferencia-mirror-v2.yml -n tutorial

And then you can send two requests to the application.

  1. curl customer-tutorial.$(minishift ip).nip.io
  2. customer => preference => recommendation v1 from '5f97c568c4-rkb8t': 8
  4. curl customer-tutorial.$(minishift ip).nip.io
  5. customer => preference => recommendation v1 from '5f97c568c4-rkb8t': 10

Things to notice here is that for each curl request, the counter is incremented by 2.Why? Because one request is the one sent to the user and the other one is the one consumed by diferencia.

Then how do you check if there is any regression or not?Diferencia offers different ways:

  • logs

  • If you start Diferencia in debug mode (in this example it is) then you can inspect the results of each request.

  • API

  • There is an endpoint that returns a JSON document with the stats of each request/response. :8082/stats.

  • Dashboard

  • There is a simple HTML dashboard to get the stats. :8082/dashboard.

  • Prometheus

  • All statistics are exposed in Prometheus format. :8081/metrics.

Check the logs of Diferencia.

  1. oc logs -f `oc get pods|grep recommendation-diferencia|awk '{ print $1 }'` -c recommendation-diferencia
  3. or
  5. kubectl logs -f `kubectl get pods|grep recommendation-diferencia|awk '{ print $1 }'` -c recommendation-diferencia

And you’ll see:

  1. time="2018-11-14T12:07:43Z" level=debug msg="URL / is going to be processed"
  2. time="2018-11-14T12:07:43Z" level=debug msg="Forwarding call to http://recommendation-master:8080/"
  3. time="2018-11-14T12:07:43Z" level=debug msg="Forwarding call to http://recommendation-candidate:8080/"
  4. time="2018-11-14T12:07:43Z" level=debug msg="Forwarding call to http://recommendation-master:8080/"
  5. time="2018-11-14T12:07:43Z" level=debug msg="Result of comparing http://recommendation-master:8080/ and http://recommendation-candidate:8080/ is false"
  6. time="2018-11-14T12:07:50Z" level=debug msg="URL / is going to be processed"
  7. time="2018-11-14T12:07:50Z" level=debug msg="Forwarding call to http://recommendation-master:8080/"
  8. time="2018-11-14T12:07:50Z" level=debug msg="Forwarding call to http://recommendation-candidate:8080/"
  9. time="2018-11-14T12:07:50Z" level=debug msg="Forwarding call to http://recommendation-master:8080/"
  10. time="2018-11-14T12:07:50Z" level=debug msg="Result of comparing http://recommendation-master:8080/ and http://recommendation-candidate:8080/ is false"

And notice that in both cases the response is false.This means that recommendation v1 and recommendation v2 are not compatible.The reason of that is because of both responses are not equal (recommendation v1 from '5f97c568c4-rkb8t': 8, recommendation v2 from '5f97cab34c4-r348t': 12).

This example uses a text comparison which is fine for this case but not in a real use case. Diferencia is built to work perfectly fine in different scenarios where JSON is the content type. You can read more about how Diferencia works when content is JSON here.

Noise Detection

Notice that in the previous example we could agree that if recommendation was present, we could say that both services are compatible and they are regression-free.

To not make a simple comparision but something a bit more error-prone, Diferencia implements a Noise cancellation approach to remove these parts that are dynamic (for example the hostname).

And it works like:

Noise Cancellation

This algorithm sends the request to not two instances but three. Two are the old instances (primary and secondary) and one to the new instance (candidate).

The request to primary and secondary returns a valid request (since old instances are the correct ones), then from their responses, Diferencia checks for the differences between both responses.

Since both responses are valid (because the source is the same but different instances), the differences that are between them are just noise that should not take into consideration for response validation.

Then detected noise is removed from primary and candidate response, and they are compared, if they are equal, then you know that everything is fine.

Of course this is a simple example but in case of the JSON is where noise cancellation is really useful, for example for skipping fields such as time, random numbers, …​

Removes Old Diferencia instance

Let’s remove Diferencia and mirroring.

  1. oc delete deployments recommendation-diferencia
  2. or
  3. kubectl delete deployments recommendation-diferencia
  5. istioctl delete -f diferencia/virtual-service-recommendation-v1-diferencia-mirror-v2.yml -n tutorial

Deploy Diferencia with Noise Cancellation

Then let’s add Diferencia with noise cancellation enabled:

  1. oc apply -f <(istioctl kube-inject -f diferencia/Deployment-noise-detection.yml) -n tutorial
  2. oc create -f diferencia/Service.yml -n tutorial
  4. or
  6. kubectl apply -f <(istioctl kube-inject -f diferencia/Deployment-noise-detection.yml) -n tutorial
  7. kubectl create -f diferencia/Service.yml -n tutorial

Then let’s enable mirroring again:

  1. istioctl create -f diferencia/virtual-service-recommendation-v1-diferencia-mirror-v2.yml -n tutorial

And then you can send two requests to the application.

  1. curl customer-tutorial.$(minishift ip).nip.io
  2. customer => preference => recommendation v1 from '5f97c568c4-rkb8t': 11
  4. curl customer-tutorial.$(minishift ip).nip.io
  5. customer => preference => recommendation v1 from '5f97c568c4-rkb8t': 14

Things to notice here is that for each curl request, the counter is incremented by 3.Why? Because one request is the one sent to the user and the other one is the one consumed by diferencia as primary and seconday.

Now if you check the logs of Diferencia, you should see something like:

  1. oc logs -f `oc get pods|grep recommendation-diferencia|awk '{ print $1 }'` -c recommendation-diferencia
  3. or
  5. kubectl logs -f `kubectl get pods|grep recommendation-diferencia|awk '{ print $1 }'` -c recommendation-diferencia
  7. time="2018-11-16T08:57:33Z" level=debug msg="URL / is going to be processed"
  8. time="2018-11-16T08:57:33Z" level=debug msg="Forwarding call to http://recommendation-master:8080/"
  9. time="2018-11-16T08:57:33Z" level=debug msg="Forwarding call to http://recommendation-candidate:8080/"
  10. time="2018-11-16T08:57:33Z" level=debug msg="Forwarding call to http://recommendation-master:8080/"
  11. time="2018-11-16T08:57:33Z" level=debug msg="Result of comparing http://recommendation-master:8080/ and http://recommendation-candidate:8080/ is true"

Notice that now the result is true meaning that both services are equivalent.

This is a simple example using text plain, but Diferencia is really useful when the output is a JSON format where there can be incompatibilities.

Clean Up

  1. oc delete deployments recommendation-diferencia
  2. or
  3. kubectl delete deployments recommendation-diferencia
  5. oc delete service recommendation-candidate
  6. or
  7. kubectl delete service recommendation-candidate
  10. oc delete service recommendation-master
  11. or
  12. kubectl delete service recommendation-master
  14. istioctl delete -f diferencia/virtual-service-recommendation-v1-diferencia-mirror-v2.yml -n tutorial