Mirroring
This task demonstrates the traffic mirroring capabilities of Istio.
Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.
In this task, you will first force all traffic to v1
of a test service. Then, you will apply a rule to mirror a portion of traffic to v2
.
Istio includes beta support for the Kubernetes Gateway API and intends to make it the default API for traffic management in the future. The following instructions allow you to choose to use either the Gateway API or the Istio configuration API when configuring traffic management in the mesh. Follow instructions under either the Gateway API
or Istio classic
tab, according to your preference.
Note that this document uses the Gateway API to configure internal mesh (east-west) traffic, i.e., not just ingress (north-south) traffic. Configuring internal mesh traffic is an experimental feature of the Gateway API, currently under development and pending upstream agreement. Make sure to install the experimental CRDs before using the Gateway API:
$ kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v0.6.2" | kubectl apply -f -
Before you begin
Set up Istio by following the instructions in the Installation guide.
Start by deploying two versions of the httpbin service that have access logging enabled:
httpbin-v1:
$ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-v1
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
version: v1
template:
metadata:
labels:
app: httpbin
version: v1
spec:
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin
command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
ports:
- containerPort: 80
EOF
httpbin-v2:
$ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-v2
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
version: v2
template:
metadata:
labels:
app: httpbin
version: v2
spec:
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin
command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
ports:
- containerPort: 80
EOF
httpbin Kubernetes service:
$ kubectl create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: httpbin
labels:
app: httpbin
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: httpbin
EOF
Start the
sleep
service so you can usecurl
to provide load:sleep service:
$ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: sleep
spec:
replicas: 1
selector:
matchLabels:
app: sleep
template:
metadata:
labels:
app: sleep
spec:
containers:
- name: sleep
image: curlimages/curl
command: ["/bin/sleep","3650d"]
imagePullPolicy: IfNotPresent
EOF
Creating a default routing policy
By default Kubernetes load balances across both versions of the httpbin
service. In this step, you will change that behavior so that all traffic goes to v1
.
- Create a default route rule to route all traffic to
v1
of the service:
$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- httpbin
http:
- route:
- destination:
host: httpbin
subset: v1
weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpbin
spec:
host: httpbin
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
EOF
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: httpbin-v1
spec:
ports:
- port: 80
name: http
selector:
app: httpbin
version: v1
---
apiVersion: v1
kind: Service
metadata:
name: httpbin-v2
spec:
ports:
- port: 80
name: http
selector:
app: httpbin
version: v2
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: httpbin
spec:
parentRefs:
- kind: Service
name: httpbin
port: 8000
rules:
- backendRefs:
- name: httpbin-v1
port: 80
EOF
Now, with all traffic directed to
httpbin:v1
, send a request to the service:$ export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
$ kubectl exec "${SLEEP_POD}" -c sleep -- curl -sS http://httpbin:8000/headers
{
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "httpbin:8000",
"User-Agent": "curl/7.35.0",
"X-B3-Parentspanid": "57784f8bff90ae0b",
"X-B3-Sampled": "1",
"X-B3-Spanid": "3289ae7257c3f159",
"X-B3-Traceid": "b56eebd279a76f0b57784f8bff90ae0b",
"X-Envoy-Attempt-Count": "1",
"X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/default;Hash=20afebed6da091c850264cc751b8c9306abac02993f80bdb76282237422bd098;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/default"
}
}
Check the logs for
v1
andv2
of thehttpbin
pods. You should see access log entries forv1
and none forv2
:$ export V1_POD=$(kubectl get pod -l app=httpbin,version=v1 -o jsonpath={.items..metadata.name})
$ kubectl logs "$V1_POD" -c httpbin
127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
$ export V2_POD=$(kubectl get pod -l app=httpbin,version=v2 -o jsonpath={.items..metadata.name})
$ kubectl logs "$V2_POD" -c httpbin
<none>
Mirroring traffic to v2
- Change the route rule to mirror traffic to v2:
$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- httpbin
http:
- route:
- destination:
host: httpbin
subset: v1
weight: 100
mirror:
host: httpbin
subset: v2
mirrorPercentage:
value: 100.0
EOF
This route rule sends 100% of the traffic to v1
. The last stanza specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2
service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow
. For example, cluster-1
becomes cluster-1-shadow
.
Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.
You can use the value
field under the mirrorPercentage
field to mirror a fraction of the traffic, instead of mirroring all requests. If this field is absent, all traffic will be mirrored.
$ kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: httpbin
spec:
parentRefs:
- kind: Service
name: httpbin
port: 8000
rules:
- filters:
- type: RequestMirror
requestMirror:
backendRef:
name: httpbin-v2
port: 80
backendRefs:
- name: httpbin-v1
port: 80
EOF
This route rule sends 100% of the traffic to v1
. The RequestMirror
filter specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2
service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow
. For example, cluster-1
becomes cluster-1-shadow
.
Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.
Send in traffic:
$ kubectl exec "${SLEEP_POD}" -c sleep -- curl -sS http://httpbin:8000/headers
Now, you should see access logging for both
v1
andv2
. The access logs created inv2
are the mirrored requests that are actually going tov1
.$ kubectl logs "$V1_POD" -c httpbin
127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
$ kubectl logs "$V2_POD" -c httpbin
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"
Cleaning up
- Remove the rules:
$ kubectl delete virtualservice httpbin
$ kubectl delete destinationrule httpbin
$ kubectl delete httproute httpbin
$ kubectl delete svc httpbin-v1 httpbin-v2
Shutdown the httpbin service and client:
$ kubectl delete deploy httpbin-v1 httpbin-v2 sleep
$ kubectl delete svc httpbin