Egress Gateways with TLS Origination (File Mount)

The TLS Origination for Egress Traffic example shows how to configure Istio to perform TLS origination for traffic to an external service. The Configure an Egress Gateway example shows how to configure Istio to direct egress traffic through a dedicated egress gateway service. This example combines the previous two by describing how to configure an egress gateway to perform TLS origination for traffic to external services.

Before you begin

  • Setup Istio by following the instructions in the Installation guide.

  • Start the sleep sample which will be used as a test source for external calls.

    If you have enabled automatic sidecar injection, do

    Zip

    1. $ kubectl apply -f @samples/sleep/sleep.yaml@

    otherwise, you have to manually inject the sidecar before deploying the sleep application:

    Zip

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

    Note that any pod that you can exec and curl from would do.

  • Create a shell variable to hold the name of the source pod for sending requests to external services. If you used the sleep sample, run:

    1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

  • For macOS users, verify that you are using openssl version 1.1 or later:

    1. $ openssl version -a | grep OpenSSL
    2. OpenSSL 1.1.1g  21 Apr 2020

    If the previous command outputs a version 1.1 or later, as shown, your openssl command should work correctly with the instructions in this task. Otherwise, upgrade your openssl or try a different implementation of openssl, for example on a Linux machine.

  • Deploy Istio egress gateway.

  • Enable Envoy’s access logging

Perform TLS origination with an egress gateway

This section describes how to perform the same TLS origination as in the TLS Origination for Egress Traffic example, only this time using an egress gateway. Note that in this case the TLS origination will be done by the egress gateway, as opposed to by the sidecar in the previous example.

  1. Define a ServiceEntry for edition.cnn.com:

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: cnn
    6. spec:
    7.   hosts:
    8.   - edition.cnn.com
    9.   ports:
    10.   - number: 80
    11.     name: http
    12.     protocol: HTTP
    13.   - number: 443
    14.     name: https
    15.     protocol: HTTPS
    16.   resolution: DNS
    17. EOF

  2. Verify that your ServiceEntry was applied correctly by sending a request to http://edition.cnn.com/politics.

    1. $ kubectl exec "${SOURCE_POD}" -c sleep -- curl -sSL -o /dev/null -D - http://edition.cnn.com/politics
    2. HTTP/1.1 301 Moved Permanently
    3. ...
    4. location: https://edition.cnn.com/politics
    5. ...

    Your ServiceEntry was configured correctly if you see 301 Moved Permanently in the output.

  3. Create an egress Gateway for edition.cnn.com, port 80, and a destination rule for sidecar requests that will be directed to the egress gateway.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: istio-egressgateway
    6. spec:
    7.   selector:
    8.     istio: egressgateway
    9.   servers:
    10.   - port:
    11.       number: 80
    12.       name: https-port-for-tls-origination
    13.       protocol: HTTPS
    14.     hosts:
    15.     - edition.cnn.com
    16.     tls:
    17.       mode: ISTIO_MUTUAL
    18. ---
    19. apiVersion: networking.istio.io/v1alpha3
    20. kind: DestinationRule
    21. metadata:
    22.   name: egressgateway-for-cnn
    23. spec:
    24.   host: istio-egressgateway.istio-system.svc.cluster.local
    25.   subsets:
    26.   - name: cnn
    27.     trafficPolicy:
    28.       loadBalancer:
    29.         simple: ROUND_ROBIN
    30.       portLevelSettings:
    31.       - port:
    32.           number: 80
    33.         tls:
    34.           mode: ISTIO_MUTUAL
    35.           sni: edition.cnn.com
    36. EOF

  4. Define a VirtualService to direct the traffic through the egress gateway, and a DestinationRule to perform TLS origination for requests to edition.cnn.com:

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: direct-cnn-through-egress-gateway
    6. spec:
    7.   hosts:
    8.   - edition.cnn.com
    9.   gateways:
    10.   - istio-egressgateway
    11.   - mesh
    12.   http:
    13.   - match:
    14.     - gateways:
    15.       - mesh
    16.       port: 80
    17.     route:
    18.     - destination:
    19.         host: istio-egressgateway.istio-system.svc.cluster.local
    20.         subset: cnn
    21.         port:
    22.           number: 80
    23.       weight: 100
    24.   - match:
    25.     - gateways:
    26.       - istio-egressgateway
    27.       port: 80
    28.     route:
    29.     - destination:
    30.         host: edition.cnn.com
    31.         port:
    32.           number: 443
    33.       weight: 100
    34. ---
    35. apiVersion: networking.istio.io/v1alpha3
    36. kind: DestinationRule
    37. metadata:
    38.   name: originate-tls-for-edition-cnn-com
    39. spec:
    40.   host: edition.cnn.com
    41.   trafficPolicy:
    42.     loadBalancer:
    43.       simple: ROUND_ROBIN
    44.     portLevelSettings:
    45.     - port:
    46.         number: 443
    47.       tls:
    48.         mode: SIMPLE # initiates HTTPS for connections to edition.cnn.com
    49. EOF

  5. Send an HTTP request to http://edition.cnn.com/politics.

    1. $ kubectl exec "${SOURCE_POD}" -c sleep -- curl -sSL -o /dev/null -D - http://edition.cnn.com/politics
    2. HTTP/1.1 200 OK
    3. ...

    The output should be the same as in the TLS Origination for Egress Traffic example, with TLS origination: without the 301 Moved Permanently message.

  6. Check the log of the istio-egressgateway pod and you should see a line corresponding to our request. If Istio is deployed in the istio-system namespace, the command to print the log is:

    1. $ kubectl logs -l istio=egressgateway -c istio-proxy -n istio-system | tail

    You should see a line similar to the following:

    1. [2020-06-30T16:17:56.763Z] "GET /politics HTTP/2" 200 - "-" "-" 0 1295938 529 89 "10.244.0.171" "curl/7.64.0" "cf76518d-3209-9ab7-a1d0-e6002728ef5b" "edition.cnn.com" "151.101.129.67:443" outbound|443||edition.cnn.com 10.244.0.170:54280 10.244.0.170:8080 10.244.0.171:35628 - -

Cleanup the TLS origination example

Remove the Istio configuration items you created:

  1. $ kubectl delete gateway istio-egressgateway
  2. $ kubectl delete serviceentry cnn
  3. $ kubectl delete virtualservice direct-cnn-through-egress-gateway
  4. $ kubectl delete destinationrule originate-tls-for-edition-cnn-com
  5. $ kubectl delete destinationrule egressgateway-for-cnn

Perform mutual TLS origination with an egress gateway

Similar to the previous section, this section describes how to configure an egress gateway to perform TLS origination for an external service, only this time using a service that requires mutual TLS.

This example is considerably more involved because you need to first:

  1. generate client and server certificates
  2. deploy an external service that supports the mutual TLS protocol
  3. redeploy the egress gateway with the needed mutual TLS certs

Only then can you configure the external traffic to go through the egress gateway which will perform TLS origination.

Generate client and server certificates and keys

For this task you can use your favorite tool to generate certificates and keys. The commands below use openssl

  1. Create a root certificate and private key to sign the certificate for your services:

    1. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt

  2. Create a certificate and a private key for my-nginx.mesh-external.svc.cluster.local:

    1. $ openssl req -out my-nginx.mesh-external.svc.cluster.local.csr -newkey rsa:2048 -nodes -keyout my-nginx.mesh-external.svc.cluster.local.key -subj "/CN=my-nginx.mesh-external.svc.cluster.local/O=some organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in my-nginx.mesh-external.svc.cluster.local.csr -out my-nginx.mesh-external.svc.cluster.local.crt

  3. Generate client certificate and private key:

    1. $ openssl req -out client.example.com.csr -newkey rsa:2048 -nodes -keyout client.example.com.key -subj "/CN=client.example.com/O=client organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in client.example.com.csr -out client.example.com.crt

Deploy a mutual TLS server

To simulate an actual external service that supports the mutual TLS protocol, deploy an NGINX server in your Kubernetes cluster, but running outside of the Istio service mesh, i.e., in a namespace without Istio sidecar proxy injection enabled.

  1. Create a namespace to represent services outside the Istio mesh, namely mesh-external. Note that the sidecar proxy will not be automatically injected into the pods in this namespace since the automatic sidecar injection was not enabled on it.

    1. $ kubectl create namespace mesh-external

  2. Create Kubernetes Secrets to hold the server’s and CA certificates.

    1. $ kubectl create -n mesh-external secret tls nginx-server-certs --key my-nginx.mesh-external.svc.cluster.local.key --cert my-nginx.mesh-external.svc.cluster.local.crt
    2. $ kubectl create -n mesh-external secret generic nginx-ca-certs --from-file=example.com.crt

  3. Create a configuration file for the NGINX server:

    1. $ cat <<\EOF > ./nginx.conf
    2. events {
    3. }
    5. http {
    6.   log_format main '$remote_addr - $remote_user [$time_local]  $status '
    7.   '"$request" $body_bytes_sent "$http_referer" '
    8.   '"$http_user_agent" "$http_x_forwarded_for"';
    9.   access_log /var/log/nginx/access.log main;
    10.   error_log  /var/log/nginx/error.log;
    12.   server {
    13.     listen 443 ssl;
    15.     root /usr/share/nginx/html;
    16.     index index.html;
    18.     server_name my-nginx.mesh-external.svc.cluster.local;
    19.     ssl_certificate /etc/nginx-server-certs/tls.crt;
    20.     ssl_certificate_key /etc/nginx-server-certs/tls.key;
    21.     ssl_client_certificate /etc/nginx-ca-certs/example.com.crt;
    22.     ssl_verify_client on;
    23.   }
    24. }
    25. EOF

  4. Create a Kubernetes ConfigMap to hold the configuration of the NGINX server:

    1. $ kubectl create configmap nginx-configmap -n mesh-external --from-file=nginx.conf=./nginx.conf

  5. Deploy the NGINX server:

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: my-nginx
    6.   namespace: mesh-external
    7.   labels:
    8.     run: my-nginx
    9. spec:
    10.   ports:
    11.   - port: 443
    12.     protocol: TCP
    13.   selector:
    14.     run: my-nginx
    15. ---
    16. apiVersion: apps/v1
    17. kind: Deployment
    18. metadata:
    19.   name: my-nginx
    20.   namespace: mesh-external
    21. spec:
    22.   selector:
    23.     matchLabels:
    24.       run: my-nginx
    25.   replicas: 1
    26.   template:
    27.     metadata:
    28.       labels:
    29.         run: my-nginx
    30.     spec:
    31.       containers:
    32.       - name: my-nginx
    33.         image: nginx
    34.         ports:
    35.         - containerPort: 443
    36.         volumeMounts:
    37.         - name: nginx-config
    38.           mountPath: /etc/nginx
    39.           readOnly: true
    40.         - name: nginx-server-certs
    41.           mountPath: /etc/nginx-server-certs
    42.           readOnly: true
    43.         - name: nginx-ca-certs
    44.           mountPath: /etc/nginx-ca-certs
    45.           readOnly: true
    46.       volumes:
    47.       - name: nginx-config
    48.         configMap:
    49.           name: nginx-configmap
    50.       - name: nginx-server-certs
    51.         secret:
    52.           secretName: nginx-server-certs
    53.       - name: nginx-ca-certs
    54.         secret:
    55.           secretName: nginx-ca-certs
    56. EOF

Redeploy the egress gateway with the client certificates

  1. Create Kubernetes Secrets to hold the client’s and CA certificates.

    1. $ kubectl create -n istio-system secret tls nginx-client-certs --key client.example.com.key --cert client.example.com.crt
    2. $ kubectl create -n istio-system secret generic nginx-ca-certs --from-file=example.com.crt

  2. To include a volume mounted from the new created secret, update the istio-egressgateway deployment. To patch the istio-egressgateway deployment, create the following gateway-patch.json file:

    1. $ cat > gateway-patch.json <<EOF
    2. [{
    3.   "op": "add",
    4.   "path": "/spec/template/spec/containers/0/volumeMounts/0",
    5.   "value": {
    6.     "mountPath": "/etc/istio/nginx-client-certs",
    7.     "name": "nginx-client-certs",
    8.     "readOnly": true
    9.   }
    10. },
    11. {
    12.   "op": "add",
    13.   "path": "/spec/template/spec/volumes/0",
    14.   "value": {
    15.   "name": "nginx-client-certs",
    16.     "secret": {
    17.       "secretName": "nginx-client-certs",
    18.       "optional": true
    19.     }
    20.   }
    21. },
    22. {
    23.   "op": "add",
    24.   "path": "/spec/template/spec/containers/0/volumeMounts/1",
    25.   "value": {
    26.     "mountPath": "/etc/istio/nginx-ca-certs",
    27.     "name": "nginx-ca-certs",
    28.     "readOnly": true
    29.   }
    30. },
    31. {
    32.   "op": "add",
    33.   "path": "/spec/template/spec/volumes/1",
    34.   "value": {
    35.   "name": "nginx-ca-certs",
    36.     "secret": {
    37.       "secretName": "nginx-ca-certs",
    38.       "optional": true
    39.     }
    40.   }
    41. }]
    42. EOF

  3. Apply istio-egressgateway deployment patch with the following command:

    1. $ kubectl -n istio-system patch --type=json deploy istio-egressgateway -p "$(cat gateway-patch.json)"

  4. Verify that the key and the certificate are successfully loaded in the istio-egressgateway pod:

    1. $ kubectl exec -n istio-system "$(kubectl -n istio-system get pods -l istio=egressgateway -o jsonpath='{.items[0].metadata.name}')" -- ls -al /etc/istio/nginx-client-certs /etc/istio/nginx-ca-certs

    tls.crt and tls.key should exist in /etc/istio/nginx-client-certs, while ca-chain.cert.pem in /etc/istio/nginx-ca-certs.

Configure mutual TLS origination for egress traffic

  1. Create an egress Gateway for my-nginx.mesh-external.svc.cluster.local, port 443, and destination rules and virtual services to direct the traffic through the egress gateway and from the egress gateway to the external service.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: istio-egressgateway
    6. spec:
    7.   selector:
    8.     istio: egressgateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https
    13.       protocol: HTTPS
    14.     hosts:
    15.     - my-nginx.mesh-external.svc.cluster.local
    16.     tls:
    17.       mode: ISTIO_MUTUAL
    18. ---
    19. apiVersion: networking.istio.io/v1alpha3
    20. kind: DestinationRule
    21. metadata:
    22.   name: egressgateway-for-nginx
    23. spec:
    24.   host: istio-egressgateway.istio-system.svc.cluster.local
    25.   subsets:
    26.   - name: nginx
    27.     trafficPolicy:
    28.       loadBalancer:
    29.         simple: ROUND_ROBIN
    30.       portLevelSettings:
    31.       - port:
    32.           number: 443
    33.         tls:
    34.           mode: ISTIO_MUTUAL
    35.           sni: my-nginx.mesh-external.svc.cluster.local
    36. EOF

  2. Define a VirtualService to direct the traffic through the egress gateway:

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: direct-nginx-through-egress-gateway
    6. spec:
    7.   hosts:
    8.   - my-nginx.mesh-external.svc.cluster.local
    9.   gateways:
    10.   - istio-egressgateway
    11.   - mesh
    12.   http:
    13.   - match:
    14.     - gateways:
    15.       - mesh
    16.       port: 80
    17.     route:
    18.     - destination:
    19.         host: istio-egressgateway.istio-system.svc.cluster.local
    20.         subset: nginx
    21.         port:
    22.           number: 443
    23.       weight: 100
    24.   - match:
    25.     - gateways:
    26.       - istio-egressgateway
    27.       port: 443
    28.     route:
    29.     - destination:
    30.         host: my-nginx.mesh-external.svc.cluster.local
    31.         port:
    32.           number: 443
    33.       weight: 100
    34. EOF

  3. Add a DestinationRule to perform mutual TLS origination

    1. $ kubectl apply -n istio-system -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: DestinationRule
    4. metadata:
    5.   name: originate-mtls-for-nginx
    6. spec:
    7.   host: my-nginx.mesh-external.svc.cluster.local
    8.   trafficPolicy:
    9.     loadBalancer:
    10.       simple: ROUND_ROBIN
    11.     portLevelSettings:
    12.     - port:
    13.         number: 443
    14.       tls:
    15.         mode: MUTUAL
    16.         clientCertificate: /etc/istio/nginx-client-certs/tls.crt
    17.         privateKey: /etc/istio/nginx-client-certs/tls.key
    18.         caCertificates: /etc/istio/nginx-ca-certs/example.com.crt
    19.         sni: my-nginx.mesh-external.svc.cluster.local
    20. EOF

  4. Send an HTTP request to http://my-nginx.mesh-external.svc.cluster.local:

    1. $ kubectl exec "$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c sleep -- curl -sS http://my-nginx.mesh-external.svc.cluster.local
    2. <!DOCTYPE html>
    3. <html>
    4. <head>
    5. <title>Welcome to nginx!</title>
    6. ...

  5. Check the log of the istio-egressgateway pod for a line corresponding to our request. If Istio is deployed in the istio-system namespace, the command to print the log is:

    1. $ kubectl logs -l istio=egressgateway -n istio-system | grep 'my-nginx.mesh-external.svc.cluster.local' | grep HTTP

    You should see a line similar to the following:

    1. [2018-08-19T18:20:40.096Z] "GET / HTTP/1.1" 200 - 0 612 7 5 "172.30.146.114" "curl/7.35.0" "b942b587-fac2-9756-8ec6-303561356204" "my-nginx.mesh-external.svc.cluster.local" "172.21.72.197:443"

Cleanup the mutual TLS origination example

  1. Remove created Kubernetes resources:

    1. $ kubectl delete secret nginx-server-certs nginx-ca-certs -n mesh-external
    2. $ kubectl delete secret istio-egressgateway-certs istio-egressgateway-ca-certs nginx-client-certs nginx-ca-certs -n istio-system
    3. $ kubectl delete configmap nginx-configmap -n mesh-external
    4. $ kubectl delete service my-nginx -n mesh-external
    5. $ kubectl delete deployment my-nginx -n mesh-external
    6. $ kubectl delete namespace mesh-external
    7. $ kubectl delete gateway istio-egressgateway
    8. $ kubectl delete virtualservice direct-nginx-through-egress-gateway
    9. $ kubectl delete destinationrule -n istio-system originate-mtls-for-nginx
    10. $ kubectl delete destinationrule egressgateway-for-nginx

  2. Delete the certificates and private keys:

    1. $ rm example.com.crt example.com.key my-nginx.mesh-external.svc.cluster.local.crt my-nginx.mesh-external.svc.cluster.local.key my-nginx.mesh-external.svc.cluster.local.csr client.example.com.crt client.example.com.csr client.example.com.key

  3. Delete the generated configuration files used in this example:

    1. $ rm ./nginx.conf
    2. $ rm ./gateway-patch.json

Cleanup

Delete the sleep service and deployment:

  1. $ kubectl delete service sleep
  2. $ kubectl delete deployment sleep

See also

Secure Control of Egress Traffic in Istio, part 3

Comparison of alternative solutions to control egress traffic including performance considerations.

Secure Control of Egress Traffic in Istio, part 2

Use Istio Egress Traffic Control to prevent attacks involving egress traffic.

Secure Control of Egress Traffic in Istio, part 1

Attacks involving egress traffic and requirements for egress traffic control.

Egress Gateway Performance Investigation

Verifies the performance impact of adding an egress gateway.

Consuming External MongoDB Services

Describes a simple scenario based on Istio’s Bookinfo example.

Monitoring and Access Policies for HTTP Egress Traffic

Describes how to configure Istio for monitoring and access policies of HTTP egress traffic.