Ingress Sidecar TLS Termination

In a regular Istio mesh deployment, the TLS termination for downstream requests is performed at the Ingress Gateway. Although this satisfies most use cases, for some (like an API Gateway in the mesh) the Ingress Gateway is not necessarily needed. This task shows how to eliminate the additional hop introduced by the Istio Ingress Gateway and let the Envoy sidecar, running alongside the application, perform TLS termination for requests coming from outside of the service mesh.

The example HTTPS service used for this task is a simple httpbin service. In the following steps you will deploy the httpbin service inside your service mesh and configure it.

The following information describes an experimental feature, which is intended for evaluation purposes only.

Before you begin

  • Setup Istio by following the instructions in the Installation guide, enabling the experimental feature ENABLE_TLS_ON_SIDECAR_INGRESS.

    1. $ istioctl install --set profile=default --set values.pilot.env.ENABLE_TLS_ON_SIDECAR_INGRESS=true

  • Create the test namespace where the target httpbin service will be deployed. Make sure to enable sidecar injection for the namespace.

    1. $ kubectl create ns test
    2. $ kubectl label namespace test istio-injection=enabled

Enable global mTLS

Apply the following PeerAuthentication policy to require mTLS traffic for all workloads in the mesh.

  1. $ kubectl -n test apply -f - <<EOF
  2. apiVersion: security.istio.io/v1beta1
  3. kind: PeerAuthentication
  4. metadata:
  5.   name: default
  6. spec:
  7.   mtls:
  8.     mode: STRICT
  9. EOF

Disable PeerAuthentication for the externally exposed httpbin port

Disable PeerAuthentication for the port of the httpbin service which will perform ingress TLS termination at the sidecar. Note that this is the targetPort of the httpbin service which should be used exclusively for external communication.

  1. $ kubectl -n test apply -f - <<EOF
  2. apiVersion: security.istio.io/v1beta1
  3. kind: PeerAuthentication
  4. metadata:
  5.   name: disable-peer-auth-for-external-mtls-port
  6.   namespace: test
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app: httpbin
  11.   mtls:
  12.     mode: STRICT
  13.   portLevelMtls:
  14.     9080:
  15.       mode: DISABLE
  16. EOF

Generate CA cert, Server cert/key and Client cert/key

For this task you can use your favorite tool to generate certificates and keys. The commands below use openssl:

  1. $ #CA is example.com
  2. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
  3. $ #Server is httpbin.test.svc.cluster.local
  4. $ openssl req -out httpbin.test.svc.cluster.local.csr -newkey rsa:2048 -nodes -keyout httpbin.test.svc.cluster.local.key -subj "/CN=httpbin.test.svc.cluster.local/O=httpbin organization"
  5. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in httpbin.test.svc.cluster.local.csr -out httpbin.test.svc.cluster.local.crt
  6. $ #client is client.test.svc.cluster.local
  7. $ openssl req -out client.test.svc.cluster.local.csr -newkey rsa:2048 -nodes -keyout client.test.svc.cluster.local.key -subj "/CN=client.test.svc.cluster.local/O=client organization"
  8. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in client.test.svc.cluster.local.csr -out client.test.svc.cluster.local.crt

Create k8s secrets for the certificates and keys

  1. $ kubectl -n test create secret generic httpbin-mtls-termination-cacert --from-file=ca.crt=./example.com.crt
  2. $ kubectl -n test create secret tls httpbin-mtls-termination --cert ./httpbin.test.svc.cluster.local.crt --key ./httpbin.test.svc.cluster.local.key

Deploy the httpbin test service

When the httpbin deployment is created, we need to use userVolumeMount annotations in the deployment to mount the certificates for the istio-proxy sidecar. Note that this step is only needed because Istio does not currently support credentialName in a sidecar configuration.

  1. sidecar.istio.io/userVolume: '{"tls-secret":{"secret":{"secretName":"httpbin-mtls-termination","optional":true}},"tls-ca-secret":{"secret":{"secretName":"httpbin-mtls-termination-cacert"}}}'
  2. sidecar.istio.io/userVolumeMount: '{"tls-secret":{"mountPath":"/etc/istio/tls-certs/","readOnly":true},"tls-ca-secret":{"mountPath":"/etc/istio/tls-ca-certs/","readOnly":true}}'

Use the following command to deploy the httpbin service with the required userVolumeMount configuration:

  1. $ kubectl -n test apply -f - <<EOF
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   name: httpbin
  6. ---
  7. apiVersion: v1
  8. kind: Service
  9. metadata:
  10.   name: httpbin
  11.   labels:
  12.     app: httpbin
  13.     service: httpbin
  14. spec:
  15.   ports:
  16.   - port: 8443
  17.     name: https
  18.     targetPort: 9080
  19.   - port: 8080
  20.     name: http
  21.     targetPort: 9081
  22.   selector:
  23.     app: httpbin
  24. ---
  25. apiVersion: apps/v1
  26. kind: Deployment
  27. metadata:
  28.   name: httpbin
  29. spec:
  30.   replicas: 1
  31.   selector:
  32.     matchLabels:
  33.       app: httpbin
  34.       version: v1
  35.   template:
  36.     metadata:
  37.       labels:
  38.         app: httpbin
  39.         version: v1
  40.       annotations:
  41.         sidecar.istio.io/userVolume: '{"tls-secret":{"secret":{"secretName":"httpbin-mtls-termination","optional":true}},"tls-ca-secret":{"secret":{"secretName":"httpbin-mtls-termination-cacert"}}}'
  42.         sidecar.istio.io/userVolumeMount: '{"tls-secret":{"mountPath":"/etc/istio/tls-certs/","readOnly":true},"tls-ca-secret":{"mountPath":"/etc/istio/tls-ca-certs/","readOnly":true}}'
  43.     spec:
  44.       serviceAccountName: httpbin
  45.       containers:
  46.       - image: docker.io/kennethreitz/httpbin
  47.         imagePullPolicy: IfNotPresent
  48.         name: httpbin
  49.         ports:
  50.         - containerPort: 80
  51. EOF

Configure httpbin to enable external mTLS

This is the core step for this feature. Using the Sidecar API, configure the ingress TLS settings. The TLS mode can be SIMPLE or MUTUAL. This example uses MUTUAL.

  1. $ kubectl -n test apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Sidecar
  4. metadata:
  5.   name: ingress-sidecar
  6.   namespace: test
  7. spec:
  8.   workloadSelector:
  9.     labels:
  10.       app: httpbin
  11.       version: v1
  12.   ingress:
  13.   - port:
  14.       number: 9080
  15.       protocol: HTTPS
  16.       name: external
  17.     defaultEndpoint: 0.0.0.0:80
  18.     tls:
  19.       mode: MUTUAL
  20.       privateKey: "/etc/istio/tls-certs/tls.key"
  21.       serverCertificate: "/etc/istio/tls-certs/tls.crt"
  22.       caCertificates: "/etc/istio/tls-ca-certs/ca.crt"
  23.   - port:
  24.       number: 9081
  25.       protocol: HTTP
  26.       name: internal
  27.     defaultEndpoint: 0.0.0.0:80
  28. EOF

Verification

Now that the httpbin server is deployed and configured, bring up two clients to test the end to end connectivity from both inside and outside of the mesh:

  1. An internal client (sleep) in the same namespace (test) as the httpbin service, with sidecar injected.
  2. An external client (sleep) in the default namespace (i.e., outside of the service mesh).
  1. $ kubectl apply -f samples/sleep/sleep.yaml
  2. $ kubectl -n test apply -f samples/sleep/sleep.yaml

Run the following commands to verify that everything is up and running, and configured correctly.

  1. $ kubectl get pods
  2. NAME                     READY   STATUS    RESTARTS   AGE
  3. sleep-557747455f-xx88g   1/1     Running   0          4m14s
  1. $ kubectl get pods -n test
  2. NAME                       READY   STATUS    RESTARTS   AGE
  3. httpbin-5bbdbd6588-z9vbs   2/2     Running   0          8m44s
  4. sleep-557747455f-brzf6     2/2     Running   0          6m57s
  1. $ kubectl get svc -n test
  2. NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
  3. httpbin   ClusterIP   10.100.78.113   <none>        8443/TCP,8080/TCP   10m
  4. sleep     ClusterIP   10.110.35.153   <none>        80/TCP              8m49s

In the following command, replace httpbin-5bbdbd6588-z9vbs with the name of your httpbin pod.

  1. $ istioctl proxy-config secret httpbin-5bbdbd6588-z9vbs.test
  2. RESOURCE NAME                                                           TYPE           STATUS     VALID CERT     SERIAL NUMBER                               NOT AFTER                NOT BEFORE
  3. file-cert:/etc/istio/tls-certs/tls.crt~/etc/istio/tls-certs/tls.key     Cert Chain     ACTIVE     true           1                                           2023-02-14T09:51:56Z     2022-02-14T09:51:56Z
  4. default                                                                 Cert Chain     ACTIVE     true           329492464719328863283539045344215802956     2022-02-15T09:55:46Z     2022-02-14T09:53:46Z
  5. ROOTCA                                                                  CA             ACTIVE     true           204427760222438623495455009380743891800     2032-02-07T16:58:00Z     2022-02-09T16:58:00Z
  6. file-root:/etc/istio/tls-ca-certs/ca.crt                                Cert Chain     ACTIVE     true           14033888812979945197                        2023-02-14T09:51:56Z     2022-02-14T09:51:56Z

Verify internal mesh connectivity on port 8080

  1. $ export INTERNAL_CLIENT=$(kubectl -n test get pod -l app=sleep -o jsonpath={.items..metadata.name})
  2. $ kubectl -n test exec "${INTERNAL_CLIENT}" -c sleep -- curl -IsS "http://httpbin:8080/status/200"
  3. HTTP/1.1 200 OK
  4. server: envoy
  5. date: Mon, 24 Oct 2022 09:04:52 GMT
  6. content-type: text/html; charset=utf-8
  7. access-control-allow-origin: *
  8. access-control-allow-credentials: true
  9. content-length: 0
  10. x-envoy-upstream-service-time: 5

Verify external to internal mesh connectivity on port 8443

To verify mTLS traffic from an external client, first copy the CA certificate and client certificate/key to the sleep client running in the default namespace.

  1. $ export EXTERNAL_CLIENT=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
  2. $ kubectl cp client.test.svc.cluster.local.key default/"${EXTERNAL_CLIENT}":/tmp/
  3. $ kubectl cp client.test.svc.cluster.local.crt default/"${EXTERNAL_CLIENT}":/tmp/
  4. $ kubectl cp example.com.crt default/"${EXTERNAL_CLIENT}":/tmp/ca.crt

Now that the certificates are available for the external sleep client, you can verify connectivity from it to the internal httpbin service using the following command.

  1. $ kubectl exec "${EXTERNAL_CLIENT}" -c sleep -- curl -IsS --cacert /tmp/ca.crt --key /tmp/client.test.svc.cluster.local.key --cert /tmp/client.test.svc.cluster.local.crt -HHost:httpbin.test.svc.cluster.local "https://httpbin.test.svc.cluster.local:8443/status/200"
  2. server: istio-envoy
  3. date: Mon, 24 Oct 2022 09:05:31 GMT
  4. content-type: text/html; charset=utf-8
  5. access-control-allow-origin: *
  6. access-control-allow-credentials: true
  7. content-length: 0
  8. x-envoy-upstream-service-time: 4
  9. x-envoy-decorator-operation: ingress-sidecar.test:9080/*

In addition to verifying external mTLS connectivity via the ingress port 8443, it is also important to verify that port 8080 does not accept any external mTLS traffic.

  1. $ kubectl exec "${EXTERNAL_CLIENT}" -c sleep -- curl -IsS --cacert /tmp/ca.crt --key /tmp/client.test.svc.cluster.local.key --cert /tmp/client.test.svc.cluster.local.crt -HHost:httpbin.test.svc.cluster.local "http://httpbin.test.svc.cluster.local:8080/status/200"
  2. curl: (56) Recv failure: Connection reset by peer
  3. command terminated with exit code 56

Cleanup the mutual TLS termination example

  1. Remove created Kubernetes resources:

    1. $ kubectl delete secret httpbin-mtls-termination httpbin-mtls-termination-cacert -n test
    2. $ kubectl delete service httpbin sleep -n test
    3. $ kubectl delete deployment httpbin sleep -n test
    4. $ kubectl delete namespace test
    5. $ kubectl delete service sleep
    6. $ kubectl delete deployment sleep

  2. Delete the certificates and private keys:

    1. $ rm example.com.crt example.com.key httpbin.test.svc.cluster.local.crt httpbin.test.svc.cluster.local.key httpbin.test.svc.cluster.local.csr \
    2.     client.test.svc.cluster.local.crt client.test.svc.cluster.local.key client.test.svc.cluster.local.csr

  3. Uninstall Istio from your cluster:

    1. $ istioctl uninstall --purge -y