Automatic mutual TLS

This tasks shows a simplified workflow for mutual TLS adoption.

With Istio auto mutual TLS feature, you can adopt mutual TLS by only configuring authentication policy without worrying about destination rule.

Istio tracks the server workloads migrated to Istio sidecar, and configures client sidecar to send mutual TLS traffic to those workloads automatically, and send plain text traffic to workloadswithout sidecars. This allows you to adopt Istio mutual TLS incrementally with minimal manual configuration.

Before you begin

  • Understand Istio authentication policy and relatedmutual TLS authentication concepts.

  • Install Istio with the global.mtls.enabled option set to false and global.mtls.auto set to true.For example, using the demo configuration profile:

  1. $ istioctl manifest apply --set profile=demo \
  2.   --set values.global.mtls.auto=true \
  3.   --set values.global.mtls.enabled=false

Instructions

Setup

Our examples deploy httpbin service into three namespaces, full, partial, and legacy.Each represents different phase of Istio migration.

full namespace contains all server workloads finishing the Istio migration. All deployments havesidecar injected.

ZipZip

  1. $ kubectl create ns full
  2. $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@) -n full
  3. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@) -n full

partial namespace contains server workloads partially migrated to Istio. Only migrated one hassidecar injected, able to serve mutual TLS traffic.

Zip

  1. $ kubectl create ns partial
  2. $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@) -n partial
  3. $ cat <<EOF | kubectl apply -n partial -f -
  4. apiVersion: apps/v1
  5. kind: Deployment
  6. metadata:
  7.   name: httpbin-nosidecar
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: httpbin
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: httpbin
  17.         version: nosidecar
  18.     spec:
  19.       containers:
  20.       - image: docker.io/kennethreitz/httpbin
  21.         imagePullPolicy: IfNotPresent
  22.         name: httpbin
  23.         ports:
  24.         - containerPort: 80
  25. EOF

legacy namespace contains the workloads and none of them have Envoy sidecar.

ZipZip

  1. $ kubectl create ns legacy
  2. $ kubectl apply -f @samples/httpbin/httpbin.yaml@ -n legacy
  3. $ kubectl apply -f @samples/sleep/sleep.yaml@ -n legacy

Last we deploy two sleep workloads, one has sidecar and one does not.

ZipZip

  1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@) -n full
  2. $ kubectl apply -f @samples/sleep/sleep.yaml@ -n legacy

You can confirm the deployments in all namespaces.

  1. $ kubectl get pods -n full
  2. $ kubectl get pods -n partial
  3. $ kubectl get pods -n legacy
  4. NAME                      READY   STATUS    RESTARTS   AGE
  5. httpbin-dcd949489-5cndk   2/2     Running   0          39s
  6. sleep-58d6644d44-gb55j    2/2     Running   0          38s
  7. NAME                       READY   STATUS    RESTARTS   AGE
  8. httpbin-6f6fc94fb6-8d62h   1/1     Running   0          10s
  9. httpbin-dcd949489-5fsbs    2/2     Running   0          12s
  10. NAME                       READY   STATUS    RESTARTS   AGE
  11. httpbin-54f5bb4957-lzxlg   1/1     Running   0          6s
  12. sleep-74564b477b-vb6h4     1/1     Running   0          4s

You should also verify that there is a default mesh authentication policy in the system, which you can do as follows:

  1. $ kubectl get policies.authentication.istio.io --all-namespaces
  2. $ kubectl get meshpolicies -o yaml | grep ' mode'
  3. NAMESPACE      NAME                          AGE
  4. istio-system   grafana-ports-mtls-disabled   2h
  5.         mode: PERMISSIVE

Last but not least, verify that there are no destination rules that apply on the example services. You can do this by checking the host: value ofexisting destination rules and make sure they do not match. For example:

  1. $ kubectl get destinationrules.networking.istio.io --all-namespaces -o yaml | grep "host:"
  2.     host: istio-policy.istio-system.svc.cluster.local
  3.     host: istio-telemetry.istio-system.svc.cluster.local

You can verify setup by sending an HTTP request with curl from any sleep pod in the namespace full, partial or legacy to either httpbin.full, httpbin.partial or httpbin.legacy. All requests should succeed with HTTP code 200.

For example, here is a command to check sleep.full to httpbin.full reachability:

  1. $ kubectl exec $(kubectl get pod -l app=sleep -n full -o jsonpath={.items..metadata.name}) -c sleep -n full -- curl http://httpbin.full:8000/headers  -s  -w "response %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$'
  2. URI=spiffe://cluster.local/ns/full/sa/sleep
  3. response 200

The SPIFFE URI shows the client identity from X509 certificate, whichindicates the traffic is sent in mutual TLS. If the traffic is in plain text, no client certificatewill be displayed.

Start from PERMISSIVE mode

In the setup, we start with PERMISSIVE for all services in the mesh.

  • All httpbin.full workloads and the workload with sidecar for httpbin.partial are able to serveboth mutual TLS traffic and plain text traffic.
  • The workload without sidecar for httpbin.partial and workloads of httpbin.legacy can only serveplain text traffic.Automatic mutual TLS configures the client, sleep.full, to send mutual TLS to the first type ofworkloads and plain text to the second type.

You can verify the reachability as:

  1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
  2. sleep.full to httpbin.full
  3. URI=spiffe://cluster.local/ns/full/sa/sleep
  4. response code: 200
  6. sleep.full to httpbin.partial
  7. URI=spiffe://cluster.local/ns/full/sa/sleep
  8. response code: 200
  10. sleep.full to httpbin.legacy
  11. response code: 200
  13. sleep.legacy to httpbin.full
  14. response code: 200
  16. sleep.legacy to httpbin.partial
  17. response code: 200
  19. sleep.legacy to httpbin.legacy
  20. response code: 200

Working with Sidecar Migration

The request to httpbin.partial can reach to server workloads with or without sidecar. Istioautomatically configures the sleep.full client to initiates mutual TLS connection to workloadwith sidecar.

  1. $ for i in `seq 1 10`; do kubectl exec $(kubectl get pod -l app=sleep -n full -o jsonpath={.items..metadata.name}) -c sleep -nfull  -- curl http://httpbin.partial:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done
  2. URI=spiffe://cluster.local/ns/full/sa/sleep
  3. response code: 200
  5. response code: 200
  7. URI=spiffe://cluster.local/ns/full/sa/sleep
  8. response code: 200
  10. response code: 200
  12. URI=spiffe://cluster.local/ns/full/sa/sleep
  13. response code: 200
  15. URI=spiffe://cluster.local/ns/full/sa/sleep
  16. response code: 200
  18. response code: 200
  20. URI=spiffe://cluster.local/ns/full/sa/sleep
  21. response code: 200
  23. response code: 200
  25. response code: 200

Without automatic mutual TLS feature, you have to track the sidecar migration finishes, and thenexplicitly configure the destination rule to make client send mutual TLS traffic to httpbin.full.

Lock down mutual TLS to STRICT

Imagine now you need to lock down the httpbin.full service to only accept mutual TLS traffic. Youcan configure authentication policy to STRICT.

  1. $ cat <<EOF | kubectl apply -n full -f -
  2. apiVersion: "authentication.istio.io/v1alpha1"
  3. kind: "Policy"
  4. metadata:
  5.   name: "httpbin"
  6. spec:
  7.   targets:
  8.   - name: httpbin
  9.   peers:
  10.   - mtls: {}
  11. EOF

All httpbin.full workloads and the workload with sidecar for httpbin.partial can only servemutual TLS traffic.

Now the requests from the sleep.legacy starts to fail, since it can’t send mutual TLS traffic.But the client sleep.full is automatically configured with auto mutual TLS, to send mutual TLSrequest, returning 200.

  1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
  2. sleep.full to httpbin.full
  3. URI=spiffe://cluster.local/ns/full/sa/sleep
  4. response code: 200
  6. sleep.full to httpbin.partial
  7. URI=spiffe://cluster.local/ns/full/sa/sleep
  8. response code: 200
  10. sleep.full to httpbin.legacy
  11. response code: 200
  13. sleep.legacy to httpbin.full
  14. response code: 000
  15. command terminated with exit code 56
  17. sleep.legacy to httpbin.partial
  18. response code: 200
  20. sleep.legacy to httpbin.legacy
  21. response code: 200

Disable mutual TLS to plain text

If for some reason, you want service to be in plain text mode explicitly, we can configure authentication policy as plain text.

  1. $ cat <<EOF | kubectl apply -n full -f -
  2. apiVersion: "authentication.istio.io/v1alpha1"
  3. kind: "Policy"
  4. metadata:
  5.   name: "httpbin"
  6. spec:
  7.   targets:
  8.   - name: httpbin
  9. EOF

In this case, since the service is in plain text mode. Istio automatically configures client sidecarsto send plain text traffic to avoid breakage.

  1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
  2. sleep.full to httpbin.full
  3. response code: 200
  5. sleep.full to httpbin.partial
  6. response code: 200
  8. sleep.full to httpbin.legacy
  9. response code: 200
  11. sleep.legacy to httpbin.full
  12. response code: 200
  14. sleep.legacy to httpbin.partial
  15. response code: 200
  17. sleep.legacy to httpbin.legacy
  18. response code: 200

All traffic are now in plain text.

Destination rule overrides

For backward compatibility, you can still use destination rule to override the TLS configuration asbefore. When destination rule has an explicit TLS configuration, that overrides the client sidecars’TLS configuration.

For example, you can explicitly configure destination rule for httpbin.full to enable ordisable mutual TLS explicitly.

  1. $ cat <<EOF | kubectl apply -n full -f -
  2. apiVersion: "networking.istio.io/v1alpha3"
  3. kind: "DestinationRule"
  4. metadata:
  5.   name: "httpbin-full-mtls"
  6. spec:
  7.   host: httpbin.full.svc.cluster.local
  8.   trafficPolicy:
  9.     tls:
  10.       mode: ISTIO_MUTUAL
  11. EOF

Since in previous steps, we already disable the authentication policy for httpbin.full to disablemutual TLS, we should see the traffic from sleep.full starting to fail.

  1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
  2. sleep.full to httpbin.full
  3. response code: 503
  5. sleep.full to httpbin.partial
  6. URI=spiffe://cluster.local/ns/full/sa/sleep
  7. response code: 200
  9. sleep.full to httpbin.legacy
  10. response code: 200
  12. sleep.legacy to httpbin.full
  13. response code: 200
  15. sleep.legacy to httpbin.partial
  16. response code: 200
  18. sleep.legacy to httpbin.legacy
  19. response code: 200

Cleanup

  1. $ kubectl delete ns full partial legacy

Summary

Automatic mutual TLS configures the client sidecar to send TLS traffic by default between sidecars.You only need to configure authentication policy.

As aforementioned, automatic mutual TLS is a mesh wide Helm installation option. You have tore-deploy Istio to enable or disable the feature. When disabling the feature, if you already relyon it to automatically encrypt the traffic, then traffic can fall back to plain text, whichcan affect your security posture or break the traffic, if the service is already configured asSTRICT to only accept mutual TLS traffic.

Currently, automatic mutual TLS is an Alpha stage feature, please be aware of the risk, and theadditional CPU cost for TLS encryption.

We’re considering to make this feature the default enabled. Please consider to send your feedbackor encountered issues when trying auto mutual TLS via Git Hub.

See also

DNS Certificate Management

Provision and manage DNS certificates in Istio.

Introducing the Istio v1beta1 Authorization Policy

Introduction, motivation and design principles for the Istio v1beta1 Authorization Policy.

Secure Webhook Management

A more secure way to manage Istio webhooks.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

App Identity and Access Adapter

Using Istio to secure multi-cloud Kubernetes applications with zero code changes.

Change in Secret Discovery Service in Istio 1.3

Taking advantage of Kubernetes trustworthy JWTs to issue certificates for workload instances more securely.