Mirroring

This task demonstrates the traffic mirroring capabilities of Istio.

Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.

In this task, you will first force all traffic to v1 of a test service. Then, you will apply a rule to mirror a portion of traffic to v2.

Istio includes beta support for the Kubernetes Gateway API and intends to make it the default API for traffic management in the future. The following instructions allow you to choose to use either the Gateway API or the Istio configuration API when configuring traffic management in the mesh. Follow instructions under either the Gateway API or Istio APIs tab, according to your preference.

Note that the Kubernetes Gateway API CRDs do not come installed by default on most Kubernetes clusters, so make sure they are installed before using the Gateway API:

  1. $ kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
  2.   { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.8.0" | kubectl apply -f -; }

Note that this document uses the Gateway API to configure internal mesh (east-west) traffic, i.e., not just ingress (north-south) traffic. Configuring internal mesh traffic is an experimental feature of the Gateway API, currently under development.

Before you begin

  • Set up Istio by following the instructions in the Installation guide.

  • Start by deploying two versions of the httpbin service that have access logging enabled:

    httpbin-v1:

    1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   name: httpbin-v1
    6. spec:
    7.   replicas: 1
    8.   selector:
    9.     matchLabels:
    10.       app: httpbin
    11.       version: v1
    12.   template:
    13.     metadata:
    14.       labels:
    15.         app: httpbin
    16.         version: v1
    17.     spec:
    18.       containers:
    19.       - image: docker.io/kennethreitz/httpbin
    20.         imagePullPolicy: IfNotPresent
    21.         name: httpbin
    22.         command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
    23.         ports:
    24.         - containerPort: 80
    25. EOF

    httpbin-v2:

    1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   name: httpbin-v2
    6. spec:
    7.   replicas: 1
    8.   selector:
    9.     matchLabels:
    10.       app: httpbin
    11.       version: v2
    12.   template:
    13.     metadata:
    14.       labels:
    15.         app: httpbin
    16.         version: v2
    17.     spec:
    18.       containers:
    19.       - image: docker.io/kennethreitz/httpbin
    20.         imagePullPolicy: IfNotPresent
    21.         name: httpbin
    22.         command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
    23.         ports:
    24.         - containerPort: 80
    25. EOF

    httpbin Kubernetes service:

    1. $ kubectl create -f - <<EOF
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: httpbin
    6.   labels:
    7.     app: httpbin
    8. spec:
    9.   ports:
    10.   - name: http
    11.     port: 8000
    12.     targetPort: 80
    13.   selector:
    14.     app: httpbin
    15. EOF

  • Start the sleep service so you can use curl to provide load:

    sleep service:

    1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   name: sleep
    6. spec:
    7.   replicas: 1
    8.   selector:
    9.     matchLabels:
    10.       app: sleep
    11.   template:
    12.     metadata:
    13.       labels:
    14.         app: sleep
    15.     spec:
    16.       containers:
    17.       - name: sleep
    18.         image: curlimages/curl
    19.         command: ["/bin/sleep","3650d"]
    20.         imagePullPolicy: IfNotPresent
    21. EOF

Creating a default routing policy

By default Kubernetes load balances across both versions of the httpbin service. In this step, you will change that behavior so that all traffic goes to v1.

  1. Create a default route rule to route all traffic to v1 of the service:
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: httpbin
  6. spec:
  7.   hosts:
  8.     - httpbin
  9.   http:
  10.   - route:
  11.     - destination:
  12.         host: httpbin
  13.         subset: v1
  14.       weight: 100
  15. ---
  16. apiVersion: networking.istio.io/v1alpha3
  17. kind: DestinationRule
  18. metadata:
  19.   name: httpbin
  20. spec:
  21.   host: httpbin
  22.   subsets:
  23.   - name: v1
  24.     labels:
  25.       version: v1
  26.   - name: v2
  27.     labels:
  28.       version: v2
  29. EOF
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: httpbin-v1
  6. spec:
  7.   ports:
  8.   - port: 80
  9.     name: http
  10.   selector:
  11.     app: httpbin
  12.     version: v1
  13. ---
  14. apiVersion: v1
  15. kind: Service
  16. metadata:
  17.   name: httpbin-v2
  18. spec:
  19.   ports:
  20.   - port: 80
  21.     name: http
  22.   selector:
  23.     app: httpbin
  24.     version: v2
  25. ---
  26. apiVersion: gateway.networking.k8s.io/v1beta1
  27. kind: HTTPRoute
  28. metadata:
  29.   name: httpbin
  30. spec:
  31.   parentRefs:
  32.   - group: ""
  33.     kind: Service
  34.     name: httpbin
  35.     port: 8000
  36.   rules:
  37.   - backendRefs:
  38.     - name: httpbin-v1
  39.       port: 80
  40. EOF

  1. Now, with all traffic directed to httpbin:v1, send a request to the service:

    1. $ export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
    2. $ kubectl exec "${SLEEP_POD}" -c sleep -- curl -sS http://httpbin:8000/headers
    3. {
    4.   "headers": {
    5.     "Accept": "*/*",
    6.     "Content-Length": "0",
    7.     "Host": "httpbin:8000",
    8.     "User-Agent": "curl/7.35.0",
    9.     "X-B3-Parentspanid": "57784f8bff90ae0b",
    10.     "X-B3-Sampled": "1",
    11.     "X-B3-Spanid": "3289ae7257c3f159",
    12.     "X-B3-Traceid": "b56eebd279a76f0b57784f8bff90ae0b",
    13.     "X-Envoy-Attempt-Count": "1",
    14.     "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/default;Hash=20afebed6da091c850264cc751b8c9306abac02993f80bdb76282237422bd098;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/default"
    15.   }
    16. }

  2. Check the logs for v1 and v2 of the httpbin pods. You should see access log entries for v1 and none for v2:

    1. $ export V1_POD=$(kubectl get pod -l app=httpbin,version=v1 -o jsonpath={.items..metadata.name})
    2. $ kubectl logs "$V1_POD" -c httpbin
    3. 127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
    1. $ export V2_POD=$(kubectl get pod -l app=httpbin,version=v2 -o jsonpath={.items..metadata.name})
    2. $ kubectl logs "$V2_POD" -c httpbin
    3. <none>

Mirroring traffic to v2

  1. Change the route rule to mirror traffic to v2:
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: httpbin
  6. spec:
  7.   hosts:
  8.     - httpbin
  9.   http:
  10.   - route:
  11.     - destination:
  12.         host: httpbin
  13.         subset: v1
  14.       weight: 100
  15.     mirror:
  16.       host: httpbin
  17.       subset: v2
  18.     mirrorPercentage:
  19.       value: 100.0
  20. EOF

This route rule sends 100% of the traffic to v1. The last stanza specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2 service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow. For example, cluster-1 becomes cluster-1-shadow.

Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.

You can use the value field under the mirrorPercentage field to mirror a fraction of the traffic, instead of mirroring all requests. If this field is absent, all traffic will be mirrored.

  1. $ kubectl apply -f - <<EOF
  2. apiVersion: gateway.networking.k8s.io/v1beta1
  3. kind: HTTPRoute
  4. metadata:
  5.   name: httpbin
  6. spec:
  7.   parentRefs:
  8.   - group: ""
  9.     kind: Service
  10.     name: httpbin
  11.     port: 8000
  12.   rules:
  13.   - filters:
  14.     - type: RequestMirror
  15.       requestMirror:
  16.         backendRef:
  17.           name: httpbin-v2
  18.           port: 80
  19.     backendRefs:
  20.     - name: httpbin-v1
  21.       port: 80
  22. EOF

This route rule sends 100% of the traffic to v1. The RequestMirror filter specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2 service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow. For example, cluster-1 becomes cluster-1-shadow.

Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.

  1. Send in traffic:

    1. $ kubectl exec "${SLEEP_POD}" -c sleep -- curl -sS http://httpbin:8000/headers

    Now, you should see access logging for both v1 and v2. The access logs created in v2 are the mirrored requests that are actually going to v1.

    1. $ kubectl logs "$V1_POD" -c httpbin
    2. 127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
    3. 127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
    1. $ kubectl logs "$V2_POD" -c httpbin
    2. 127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"

Cleaning up

  1. Remove the rules:
  1. $ kubectl delete virtualservice httpbin
  2. $ kubectl delete destinationrule httpbin
  1. $ kubectl delete httproute httpbin
  2. $ kubectl delete svc httpbin-v1 httpbin-v2

  1. Shutdown the httpbin service and client:

    1. $ kubectl delete deploy httpbin-v1 httpbin-v2 sleep
    2. $ kubectl delete svc httpbin