Mirroring

This task demonstrates the traffic mirroring capabilities of Istio.

Traffic mirroring, also called shadowing, is a powerful concept that allowsfeature teams to bring changes to production with as little risk as possible.Mirroring sends a copy of live traffic to a mirrored service. The mirroredtraffic happens out of band of the critical request path for the primary service.

In this task, you will first force all traffic to v1 of a test service. Then,you will apply a rule to mirror a portion of traffic to v2.

Before you begin

  • Set up Istio by following the instructions in theInstallation guide.

  • Start by deploying two versions of the httpbin service that have access logging enabled:

httpbin-v1:

  1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: httpbin-v1
  6. spec:
  7.   replicas: 1
  8.   selector:
  9.     matchLabels:
  10.       app: httpbin
  11.       version: v1
  12.   template:
  13.     metadata:
  14.       labels:
  15.         app: httpbin
  16.         version: v1
  17.     spec:
  18.       containers:
  19.       - image: docker.io/kennethreitz/httpbin
  20.         imagePullPolicy: IfNotPresent
  21.         name: httpbin
  22.         command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
  23.         ports:
  24.         - containerPort: 80
  25. EOF

httpbin-v2:

  1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: httpbin-v2
  6. spec:
  7.   replicas: 1
  8.   selector:
  9.     matchLabels:
  10.       app: httpbin
  11.       version: v2
  12.   template:
  13.     metadata:
  14.       labels:
  15.         app: httpbin
  16.         version: v2
  17.     spec:
  18.       containers:
  19.       - image: docker.io/kennethreitz/httpbin
  20.         imagePullPolicy: IfNotPresent
  21.         name: httpbin
  22.         command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
  23.         ports:
  24.         - containerPort: 80
  25. EOF

httpbin Kubernetes service:

  1. $ kubectl create -f - <<EOF
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: httpbin
  6.   labels:
  7.     app: httpbin
  8. spec:
  9.   ports:
  10.   - name: http
  11.     port: 8000
  12.     targetPort: 80
  13.   selector:
  14.     app: httpbin
  15. EOF
  • Start the sleep service so you can use curl to provide load:

sleep service:

  1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: sleep
  6. spec:
  7.   replicas: 1
  8.   selector:
  9.     matchLabels:
  10.       app: sleep
  11.   template:
  12.     metadata:
  13.       labels:
  14.         app: sleep
  15.     spec:
  16.       containers:
  17.       - name: sleep
  18.         image: tutum/curl
  19.         command: ["/bin/sleep","infinity"]
  20.         imagePullPolicy: IfNotPresent
  21. EOF

Creating a default routing policy

By default Kubernetes load balances across both versions of the httpbin service.In this step, you will change that behavior so that all traffic goes to v1.

  • Create a default route rule to route all traffic to v1 of the service:

If you installed/configured Istio with mutual TLS authentication enabled, you must add a TLS traffic policy mode: ISTIO_MUTUAL to the DestinationRule before applying it.Otherwise requests will generate 503 errors.

  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: httpbin
  6. spec:
  7.   hosts:
  8.     - httpbin
  9.   http:
  10.   - route:
  11.     - destination:
  12.         host: httpbin
  13.         subset: v1
  14.       weight: 100
  15. ---
  16. apiVersion: networking.istio.io/v1alpha3
  17. kind: DestinationRule
  18. metadata:
  19.   name: httpbin
  20. spec:
  21.   host: httpbin
  22.   subsets:
  23.   - name: v1
  24.     labels:
  25.       version: v1
  26.   - name: v2
  27.     labels:
  28.       version: v2
  29. EOF

Now all traffic goes to the httpbin:v1 service.

  • Send some traffic to the service:
  1. $ export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
  2. $ kubectl exec -it $SLEEP_POD -c sleep -- sh -c 'curl  http://httpbin:8000/headers' | python -m json.tool
  3. {
  4.   "headers": {
  5.     "Accept": "*/*",
  6.     "Content-Length": "0",
  7.     "Host": "httpbin:8000",
  8.     "User-Agent": "curl/7.35.0",
  9.     "X-B3-Sampled": "1",
  10.     "X-B3-Spanid": "eca3d7ed8f2e6a0a",
  11.     "X-B3-Traceid": "eca3d7ed8f2e6a0a",
  12.     "X-Ot-Span-Context": "eca3d7ed8f2e6a0a;eca3d7ed8f2e6a0a;0000000000000000"
  13.   }
  14. }
  • Check the logs for v1 and v2 of the httpbin pods. You should see accesslog entries for v1 and none for v2:
  1. $ export V1_POD=$(kubectl get pod -l app=httpbin,version=v1 -o jsonpath={.items..metadata.name})
  2. $ kubectl logs -f $V1_POD -c httpbin
  3. 127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
  1. $ export V2_POD=$(kubectl get pod -l app=httpbin,version=v2 -o jsonpath={.items..metadata.name})
  2. $ kubectl logs -f $V2_POD -c httpbin
  3. <none>

Mirroring traffic to v2

  • Change the route rule to mirror traffic to v2:
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: httpbin
  6. spec:
  7.   hosts:
  8.     - httpbin
  9.   http:
  10.   - route:
  11.     - destination:
  12.         host: httpbin
  13.         subset: v1
  14.       weight: 100
  15.     mirror:
  16.       host: httpbin
  17.       subset: v2
  18.     mirror_percent: 100
  19. EOF

This route rule sends 100% of the traffic to v1. The last stanza specifiesthat you want to mirror to the httpbin:v2 service. When traffic gets mirrored,the requests are sent to the mirrored service with their Host/Authority headersappended with -shadow. For example, cluster-1 becomes cluster-1-shadow.

Also, it is important to note that these requests are mirrored as “fire andforget”, which means that the responses are discarded.

You can use the mirror_percent field to mirror a fraction of the traffic,instead of mirroring all requests. If this field is absent, for compatibility witholder versions, all traffic will be mirrored.

  • Send in traffic:
  1. $ kubectl exec -it $SLEEP_POD -c sleep -- sh -c 'curl  http://httpbin:8000/headers' | python -m json.tool

Now, you should see access logging for both v1 and v2. The access logscreated in v2 are the mirrored requests that are actually going to v1.

  1. $ kubectl logs -f $V1_POD -c httpbin
  2. 127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
  3. 127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
  1. $ kubectl logs -f $V2_POD -c httpbin
  2. 127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"
  • If you want to examine traffic internals, run the following commands on another console:
  1. $ export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
  2. $ export V1_POD_IP=$(kubectl get pod -l app=httpbin -l version=v1 -o jsonpath={.items..status.podIP})
  3. $ export V2_POD_IP=$(kubectl get pod -l app=httpbin -l version=v2 -o jsonpath={.items..status.podIP})
  4. $ kubectl exec -it $SLEEP_POD -c istio-proxy -- sudo tcpdump -A -s 0 host $V1_POD_IP or host $V2_POD_IP
  5. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  6. listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
  7. 05:47:50.159513 IP sleep-7b9f8bfcd-2djx5.38836 > 10-233-75-11.httpbin.default.svc.cluster.local.80: Flags [P.], seq 4039989036:4039989832, ack 3139734980, win 254, options [nop,nop,TS val 77427918 ecr 76730809], length 796: HTTP: GET /headers HTTP/1.1
  8. E..P2.X.X.X.
  9. .K.
  10. .K....P..W,.$.......+.....
  11. ..t.....GET /headers HTTP/1.1
  12. host: httpbin:8000
  13. user-agent: curl/7.35.0
  14. accept: */*
  15. x-forwarded-proto: http
  16. x-request-id: 571c0fd6-98d4-4c93-af79-6a2fe2945847
  17. x-envoy-decorator-operation: httpbin.default.svc.cluster.local:8000/*
  18. x-b3-traceid: 82f3e0a76dcebca2
  19. x-b3-spanid: 82f3e0a76dcebca2
  20. x-b3-sampled: 0
  21. x-istio-attributes: Cj8KGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBIjEiFodHRwYmluLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKPQoXZGVzdGluYXRpb24uc2VydmljZS51aWQSIhIgaXN0aW86Ly9kZWZhdWx0L3NlcnZpY2VzL2h0dHBiaW4KKgodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USCRIHZGVmYXVsdAolChhkZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWUSCRIHaHR0cGJpbgo6Cgpzb3VyY2UudWlkEiwSKmt1YmVybmV0ZXM6Ly9zbGVlcC03YjlmOGJmY2QtMmRqeDUuZGVmYXVsdAo6ChNkZXN0aW5hdGlvbi5zZXJ2aWNlEiMSIWh0dHBiaW4uZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbA==
  22. content-length: 0
  25. 05:47:50.159609 IP sleep-7b9f8bfcd-2djx5.49560 > 10-233-71-7.httpbin.default.svc.cluster.local.80: Flags [P.], seq 296287713:296288571, ack 4029574162, win 254, options [nop,nop,TS val 77427918 ecr 76732809], length 858: HTTP: GET /headers HTTP/1.1
  26. E.....X.X...
  27. .K.
  28. .G....P......l......e.....
  29. ..t.....GET /headers HTTP/1.1
  30. host: httpbin-shadow:8000
  31. user-agent: curl/7.35.0
  32. accept: */*
  33. x-forwarded-proto: http
  34. x-request-id: 571c0fd6-98d4-4c93-af79-6a2fe2945847
  35. x-envoy-decorator-operation: httpbin.default.svc.cluster.local:8000/*
  36. x-b3-traceid: 82f3e0a76dcebca2
  37. x-b3-spanid: 82f3e0a76dcebca2
  38. x-b3-sampled: 0
  39. x-istio-attributes: Cj8KGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBIjEiFodHRwYmluLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKPQoXZGVzdGluYXRpb24uc2VydmljZS51aWQSIhIgaXN0aW86Ly9kZWZhdWx0L3NlcnZpY2VzL2h0dHBiaW4KKgodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USCRIHZGVmYXVsdAolChhkZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWUSCRIHaHR0cGJpbgo6Cgpzb3VyY2UudWlkEiwSKmt1YmVybmV0ZXM6Ly9zbGVlcC03YjlmOGJmY2QtMmRqeDUuZGVmYXVsdAo6ChNkZXN0aW5hdGlvbi5zZXJ2aWNlEiMSIWh0dHBiaW4uZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbA==
  40. x-envoy-internal: true
  41. x-forwarded-for: 10.233.75.12
  42. content-length: 0
  45. 05:47:50.166734 IP 10-233-75-11.httpbin.default.svc.cluster.local.80 > sleep-7b9f8bfcd-2djx5.38836: Flags [P.], seq 1:472, ack 796, win 276, options [nop,nop,TS val 77427925 ecr 77427918], length 471: HTTP: HTTP/1.1 200 OK
  46. E....3X.?...
  47. .K.
  48. .K..P...$....ZH...........
  49. ..t...t.HTTP/1.1 200 OK
  50. server: envoy
  51. date: Fri, 15 Feb 2019 05:47:50 GMT
  52. content-type: application/json
  53. content-length: 241
  54. access-control-allow-origin: *
  55. access-control-allow-credentials: true
  56. x-envoy-upstream-service-time: 3
  58. {
  59.   "headers": {
  60.     "Accept": "*/*",
  61.     "Content-Length": "0",
  62.     "Host": "httpbin:8000",
  63.     "User-Agent": "curl/7.35.0",
  64.     "X-B3-Sampled": "0",
  65.     "X-B3-Spanid": "82f3e0a76dcebca2",
  66.     "X-B3-Traceid": "82f3e0a76dcebca2"
  67.   }
  68. }
  70. 05:47:50.166789 IP sleep-7b9f8bfcd-2djx5.38836 > 10-233-75-11.httpbin.default.svc.cluster.local.80: Flags [.], ack 472, win 262, options [nop,nop,TS val 77427925 ecr 77427925], length 0
  71. E..42.X.X.\.
  72. .K.
  73. .K....P..ZH.$.............
  74. ..t...t.
  75. 05:47:50.167234 IP 10-233-71-7.httpbin.default.svc.cluster.local.80 > sleep-7b9f8bfcd-2djx5.49560: Flags [P.], seq 1:512, ack 858, win 280, options [nop,nop,TS val 77429926 ecr 77427918], length 511: HTTP: HTTP/1.1 200 OK
  76. E..3..X.>...
  77. .G.
  78. .K..P....l....;...........
  79. ..|...t.HTTP/1.1 200 OK
  80. server: envoy
  81. date: Fri, 15 Feb 2019 05:47:49 GMT
  82. content-type: application/json
  83. content-length: 281
  84. access-control-allow-origin: *
  85. access-control-allow-credentials: true
  86. x-envoy-upstream-service-time: 3
  88. {
  89.   "headers": {
  90.     "Accept": "*/*",
  91.     "Content-Length": "0",
  92.     "Host": "httpbin-shadow:8000",
  93.     "User-Agent": "curl/7.35.0",
  94.     "X-B3-Sampled": "0",
  95.     "X-B3-Spanid": "82f3e0a76dcebca2",
  96.     "X-B3-Traceid": "82f3e0a76dcebca2",
  97.     "X-Envoy-Internal": "true"
  98.   }
  99. }
  101. 05:47:50.167253 IP sleep-7b9f8bfcd-2djx5.49560 > 10-233-71-7.httpbin.default.svc.cluster.local.80: Flags [.], ack 512, win 262, options [nop,nop,TS val 77427926 ecr 77429926], length 0
  102. E..4..X.X...
  103. .K.
  104. .G....P...;..n............
  105. ..t...|.

You can see request and response contents of the traffic.

Cleaning up

  • Remove the rules:
  1. $ kubectl delete virtualservice httpbin
  2. $ kubectl delete destinationrule httpbin
  • Shutdown the httpbin service and client:
  1. $ kubectl delete deploy httpbin-v1 httpbin-v2 sleep
  2. $ kubectl delete svc httpbin

See also

Traffic Mirroring with Istio for Testing in Production

An introduction to safer, lower-risk deployments and release to production.

Istio as a Proxy for External Services

Configure Istio ingress gateway to act as a proxy for external services.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

Secure Control of Egress Traffic in Istio, part 3

Comparison of alternative solutions to control egress traffic including performance considerations.

Secure Control of Egress Traffic in Istio, part 2

Use Istio Egress Traffic Control to prevent attacks involving egress traffic.

Secure Control of Egress Traffic in Istio, part 1

Attacks involving egress traffic and requirements for egress traffic control.