Egress Gateway 的双向 TLS。

配置出口网关示例描述了如何配置 Istio 以通过名为 egress gateway 的专用服务引导出口流量。此示例展示如何配置出口网关以启用到外部服务的流量的双向 TLS。

要模拟支持 mutual TLS 协议的实际外部服务，首先在 Kubernetes 集群中部署 NGINX 服务器，但在 Istio 服务网格之外运行，即在命名空间中运行没有启用 Istio 的代理注入 sidecar 。接下来，配置出口网关以与外部 NGINX 服务器执行双向 TLS。最后，通过出口网关将流量从网格内的应用程序 pod 引导到网格外的 NGINX 服务器。

生成客户端和服务器证书和密钥

  1. $ git clone https://github.com/nicholasjackson/mtls-go-example
  • 进入克隆存储库的目录：
  1. $ cd mtls-go-example
  • 为 nginx.example.com 生成证书。使用以下命令（密码任意指定）：
  1. $ ./generate.sh nginx.example.com <password>

为所有出现的提示选择 y。

  • 将证书移动到 nginx.example.com 目录：
  1. $ mkdir ../nginx.example.com && mv 1_root 2_intermediate 3_application 4_client ../nginx.example.com
  • 返回上级目录：
  1. $ cd ..

部署 NGINX 服务器

  • 创建一个命名空间来表示 Istio 网格之外的服务，即 mesh-external。请注意，由于自动注入 sidecar 没有启用，因此 sidecar 代理不会自动注入此命名空间中的 pod。
  1. $ kubectl create namespace mesh-external
  • 创建 Kubernetes Secret 用来保存服务器端证书和 CA 证书。
  1. $ kubectl create -n mesh-external secret tls nginx-server-certs --key nginx.example.com/3_application/private/nginx.example.com.key.pem --cert nginx.example.com/3_application/certs/nginx.example.com.cert.pem
  2. $ kubectl create -n mesh-external secret generic nginx-ca-certs --from-file=nginx.example.com/2_intermediate/certs/ca-chain.cert.pem
  • 为 NGINX 服务器创建配置文件：
  1. $ cat <<EOF > ./nginx.conf
  2. events {
  3. }
  5. http {
  6.   log_format main '$remote_addr - $remote_user [$time_local]  $status '
  7.   '"$request" $body_bytes_sent "$http_referer" '
  8.   '"$http_user_agent" "$http_x_forwarded_for"';
  9.   access_log /var/log/nginx/access.log main;
  10.   error_log  /var/log/nginx/error.log;
  12.   server {
  13.     listen 443 ssl;
  15.     root /usr/share/nginx/html;
  16.     index index.html;
  18.     server_name nginx.example.com;
  19.     ssl_certificate /etc/nginx-server-certs/tls.crt;
  20.     ssl_certificate_key /etc/nginx-server-certs/tls.key;
  21.     ssl_client_certificate /etc/nginx-ca-certs/ca-chain.cert.pem;
  22.     ssl_verify_client on;
  23.   }
  24. }
  25. EOF
  • 创建一个 Kubernetes ConfigMap 用来保存 NGINX 服务器的配置：
  1. $ kubectl create configmap nginx-configmap -n mesh-external --from-file=nginx.conf=./nginx.conf
  • 部署 NGINX 服务器：
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: my-nginx
  6.   namespace: mesh-external
  7.   labels:
  8.     run: my-nginx
  9. spec:
  10.   ports:
  11.   - port: 443
  12.     protocol: TCP
  13.   selector:
  14.     run: my-nginx
  15. ---
  16. apiVersion: apps/v1
  17. kind: Deployment
  18. metadata:
  19.   name: my-nginx
  20.   namespace: mesh-external
  21. spec:
  22.   selector:
  23.     matchLabels:
  24.       run: my-nginx
  25.   replicas: 1
  26.   template:
  27.     metadata:
  28.       labels:
  29.         run: my-nginx
  30.     spec:
  31.       containers:
  32.       - name: my-nginx
  33.         image: nginx
  34.         ports:
  35.         - containerPort: 443
  36.         volumeMounts:
  37.         - name: nginx-config
  38.           mountPath: /etc/nginx
  39.           readOnly: true
  40.         - name: nginx-server-certs
  41.           mountPath: /etc/nginx-server-certs
  42.           readOnly: true
  43.         - name: nginx-ca-certs
  44.           mountPath: /etc/nginx-ca-certs
  45.           readOnly: true
  46.       volumes:
  47.       - name: nginx-config
  48.         configMap:
  49.           name: nginx-configmap
  50.       - name: nginx-server-certs
  51.         secret:
  52.           secretName: nginx-server-certs
  53.       - name: nginx-ca-certs
  54.         secret:
  55.           secretName: nginx-ca-certs
  56. EOF
  • 为 nginx.example.com 定义一个 ServiceEntry 和 VirtualService，用来指示 Istio 将指向 nginx.example.com 的流量定向到你的 NGINX 服务器：
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: ServiceEntry
  4. metadata:
  5.   name: nginx
  6. spec:
  7.   hosts:
  8.   - nginx.example.com
  9.   ports:
  10.   - number: 80
  11.     name: http
  12.     protocol: HTTP
  13.   - number: 443
  14.     name: https
  15.     protocol: HTTPS
  16.   resolution: DNS
  17.   endpoints:
  18.   - address: my-nginx.mesh-external.svc.cluster.local
  19.     ports:
  20.       https: 443
  21. ---
  22. apiVersion: networking.istio.io/v1alpha3
  23. kind: VirtualService
  24. metadata:
  25.   name: nginx
  26. spec:
  27.   hosts:
  28.   - nginx.example.com
  29.   tls:
  30.   - match:
  31.     - port: 443
  32.       sni_hosts:
  33.       - nginx.example.com
  34.     route:
  35.     - destination:
  36.         host: nginx.example.com
  37.         port:
  38.           number: 443
  39.       weight: 100
  40. EOF

部署容器以测试 NGINX 部署

  • 创建 Kubernetes Secret 来保存客户端证书和 CA 证书：
  1. $ kubectl create secret tls nginx-client-certs --key nginx.example.com/4_client/private/nginx.example.com.key.pem --cert nginx.example.com/4_client/certs/nginx.example.com.cert.pem
  2. $ kubectl create secret generic nginx-ca-certs --from-file=nginx.example.com/2_intermediate/certs/ca-chain.cert.pem
  • 使用已安装的客户端和 CA 证书部署 sleep 示例，以测试向 NGINX 服务器发送请求：
  1. $ kubectl apply -f - <<EOF
  2. # Copyright 2017 Istio Authors
  3. #
  4. #   Licensed under the Apache License, Version 2.0 (the "License");
  5. #   you may not use this file except in compliance with the License.
  6. #   You may obtain a copy of the License at
  7. #
  8. #       http://www.apache.org/licenses/LICENSE-2.0
  9. #
  10. #   Unless required by applicable law or agreed to in writing, software
  11. #   distributed under the License is distributed on an "AS IS" BASIS,
  12. #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13. #   See the License for the specific language governing permissions and
  14. #   limitations under the License.
  16. ##################################################################################################
  17. # Sleep service
  18. ##################################################################################################
  19. apiVersion: v1
  20. kind: Service
  21. metadata:
  22.   name: sleep
  23.   labels:
  24.     app: sleep
  25. spec:
  26.   ports:
  27.   - port: 80
  28.     name: http
  29.   selector:
  30.     app: sleep
  31. ---
  32. apiVersion: extensions/v1beta1
  33. kind: Deployment
  34. metadata:
  35.   name: sleep
  36. spec:
  37.   replicas: 1
  38.   template:
  39.     metadata:
  40.       labels:
  41.         app: sleep
  42.     spec:
  43.       containers:
  44.       - name: sleep
  45.         image: tutum/curl
  46.         command: ["/bin/sleep","infinity"]
  47.         imagePullPolicy: IfNotPresent
  48.         volumeMounts:
  49.         - name: nginx-client-certs
  50.           mountPath: /etc/nginx-client-certs
  51.           readOnly: true
  52.         - name: nginx-ca-certs
  53.           mountPath: /etc/nginx-ca-certs
  54.           readOnly: true
  55.       volumes:
  56.       - name: nginx-client-certs
  57.         secret:
  58.           secretName: nginx-client-certs
  59.       - name: nginx-ca-certs
  60.         secret:
  61.           secretName: nginx-ca-certs
  62. EOF
  • 定义一个环境变量来保存 sleep pod 的名称：
  1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
  • 使用部署的 sleep pod 将请求发送到 NGINX 服务器。由于 nginx.example.com 实际上并不存在，因此 DNS 无法解析它，以下 curl 命令使用 —resolve 选项手动解析主机名。在 –resolve 选项（下面的1.1.1.1）中传递的 IP 值并不重要。可以使用 127.0.0.1 以外的任何值。通常，目标主机名存在 DNS 条目，您不会使用 curl 的 —resolve 选项。
  1. $ kubectl exec -it $SOURCE_POD -c sleep -- curl -v --resolve nginx.example.com:443:1.1.1.1 --cacert /etc/nginx-ca-certs/ca-chain.cert.pem --cert /etc/nginx-client-certs/tls.crt --key /etc/nginx-client-certs/tls.key https://nginx.example.com
  2. ...
  3. Server certificate:
  4.   subject: C=US; ST=Denial; L=Springfield; O=Dis; CN=nginx.example.com
  5.   start date: 2018-08-16 04:31:20 GMT
  6.   expire date: 2019-08-26 04:31:20 GMT
  7.   common name: nginx.example.com (matched)
  8.   issuer: C=US; ST=Denial; O=Dis; CN=nginx.example.com
  9.   SSL certificate verify ok.
  10. > GET / HTTP/1.1
  11. > User-Agent: curl/7.35.0
  12. > Host: nginx.example.com
  13. ...
  14. < HTTP/1.1 200 OK
  16. < Server: nginx/1.15.2
  17. ...
  18. <!DOCTYPE html>
  19. <html>
  20. <head>
  21. <title>Welcome to nginx!</title>
  22. ...
  • 验证服务器是否需要客户端证书：
  1. $ kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) -c sleep -- curl -k --resolve nginx.example.com:443:1.1.1.1 https://nginx.example.com
  2. <html>
  3. <head><title>400 No required SSL certificate was sent</title></head>
  4. <body bgcolor="white">
  5. <center><h1>400 Bad Request</h1></center>
  6. <center>No required SSL certificate was sent</center>
  7. <hr><center>nginx/1.15.2</center>
  8. </body>
  9. </html>

使用客户端证书重新部署 Egress 网关

  • 创建 Kubernetes Secret 以保存客户端证书和 CA 证书。
  1. $ kubectl create -n istio-system secret tls nginx-client-certs --key nginx.example.com/4_client/private/nginx.example.com.key.pem --cert nginx.example.com/4_client/certs/nginx.example.com.cert.pem
  2. $ kubectl create -n istio-system secret generic nginx-ca-certs --from-file=nginx.example.com/2_intermediate/certs/ca-chain.cert.pem
  • 生成 istio-egressgateway deployment，其中包含要从新 Secret 安装的 volume。用生成 istio.yaml 相同的选项：
  1. $ helm template install/kubernetes/helm/istio/ --name istio-egressgateway --namespace istio-system -x charts/gateways/templates/deployment.yaml --set gateways.istio-ingressgateway.enabled=false \
  2. --set gateways.istio-egressgateway.secretVolumes[0].name=egressgateway-certs \
  3. --set gateways.istio-egressgateway.secretVolumes[0].secretName=istio-egressgateway-certs \
  4. --set gateways.istio-egressgateway.secretVolumes[0].mountPath=/etc/istio/egressgateway-certs \
  5. --set gateways.istio-egressgateway.secretVolumes[1].name=egressgateway-ca-certs \
  6. --set gateways.istio-egressgateway.secretVolumes[1].secretName=istio-egressgateway-ca-certs \
  7. --set gateways.istio-egressgateway.secretVolumes[1].mountPath=/etc/istio/egressgateway-ca-certs \
  8. --set gateways.istio-egressgateway.secretVolumes[2].name=nginx-client-certs \
  9. --set gateways.istio-egressgateway.secretVolumes[2].secretName=nginx-client-certs \
  10. --set gateways.istio-egressgateway.secretVolumes[2].mountPath=/etc/nginx-client-certs \
  11. --set gateways.istio-egressgateway.secretVolumes[3].name=nginx-ca-certs \
  12. --set gateways.istio-egressgateway.secretVolumes[3].secretName=nginx-ca-certs \
  13. --set gateways.istio-egressgateway.secretVolumes[3].mountPath=/etc/nginx-ca-certs > \
  14. ./istio-egressgateway.yaml
  • 重新部署 istio-egressgateway：
  1. $ kubectl apply -f ./istio-egressgateway.yaml
  2. deployment "istio-egressgateway" configured
  • 验证密钥和证书是否已成功加载到 istio-egressgateway pod 中：
  1. $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=egressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/nginx-client-certs /etc/nginx-ca-certs

/etc/istio/nginx-client-certs 中应该存在 tls.crt 和 tls.key，而 /etc/istio/nginx-ca-certs 中应该存在 ca-chain.cert.pem 。

为出口流量配置双向 TLS

  • 给 nginx.example.com 在 443 端口上创建出口 Gateway，以及目的地规则和虚拟服务，以通过出口网关和出口网关将流量引导到外部服务。
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: istio-egressgateway
  6. spec:
  7.   selector:
  8.     istio: egressgateway
  9.   servers:
  10.   - port:
  11.       number: 443
  12.       name: https
  13.       protocol: HTTPS
  14.     hosts:
  15.     - nginx.example.com
  16.     tls:
  17.       mode: MUTUAL
  18.       serverCertificate: /etc/certs/cert-chain.pem
  19.       privateKey: /etc/certs/key.pem
  20.       caCertificates: /etc/certs/root-cert.pem
  21. ---
  22. apiVersion: networking.istio.io/v1alpha3
  23. kind: DestinationRule
  24. metadata:
  25.   name: egressgateway-for-nginx
  26. spec:
  27.   host: istio-egressgateway.istio-system.svc.cluster.local
  28.   subsets:
  29.   - name: nginx
  30.     trafficPolicy:
  31.       loadBalancer:
  32.         simple: ROUND_ROBIN
  33.       portLevelSettings:
  34.       - port:
  35.           number: 443
  36.         tls:
  37.           mode: ISTIO_MUTUAL
  38.           sni: nginx.example.com
  39. EOF
  • 定义一个 VirtualService 来引导流量通过出口网关，一个 DestinationRule 来执行双向的 TLS 的发起：
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: direct-nginx-through-egress-gateway
  6. spec:
  7.   hosts:
  8.   - nginx.example.com
  9.   gateways:
  10.   - istio-egressgateway
  11.   - mesh
  12.   http:
  13.   - match:
  14.     - gateways:
  15.       - mesh
  16.       port: 80
  17.     route:
  18.     - destination:
  19.         host: istio-egressgateway.istio-system.svc.cluster.local
  20.         subset: nginx
  21.         port:
  22.           number: 443
  23.       weight: 100
  24.   - match:
  25.     - gateways:
  26.       - istio-egressgateway
  27.       port: 443
  28.     route:
  29.     - destination:
  30.         host: nginx.example.com
  31.         port:
  32.           number: 443
  33.       weight: 100
  34. ---
  35. apiVersion: networking.istio.io/v1alpha3
  36. kind: DestinationRule
  37. metadata:
  38.   name: originate-mtls-for-nginx
  39. spec:
  40.   host: nginx.example.com
  41.   trafficPolicy:
  42.     loadBalancer:
  43.       simple: ROUND_ROBIN
  44.     portLevelSettings:
  45.     - port:
  46.         number: 443
  47.       tls:
  48.         mode: MUTUAL
  49.         clientCertificate: /etc/nginx-client-certs/tls.crt
  50.         privateKey: /etc/nginx-client-certs/tls.key
  51.         caCertificates: /etc/nginx-ca-certs/ca-chain.cert.pem
  52.         sni: nginx.example.com
  53. EOF
  1. $ kubectl exec -it $SOURCE_POD -c sleep -- curl -s --resolve nginx.example.com:80:1.1.1.1 http://nginx.example.com
  2. <!DOCTYPE html>
  3. <html>
  4. <head>
  5. <title>Welcome to nginx!</title>
  6. ...
  • 检查 istio-egressgateway pod 的日志，看看与我们的请求相对应的行。如果 Istio 部署在 istio-system 命名空间中，则打印日志的命令是：
  1. $ kubectl logs -l istio=egressgateway -n istio-system | grep 'nginx.example.com' | grep HTTP

您应该看到与您的请求相关的行，类似于以下内容：

  1. [2018-08-19T18:20:40.096Z] "GET / HTTP/1.1" 200 - 0 612 7 5 "172.30.146.114" "curl/7.35.0" "b942b587-fac2-9756-8ec6-303561356204" "nginx.example.com" "172.21.72.197:443"

清理

  1. $ kubectl delete secret nginx-server-certs nginx-ca-certs -n mesh-external
  2. $ kubectl delete secret nginx-client-certs nginx-ca-certs
  3. $ kubectl delete secret nginx-client-certs nginx-ca-certs -n istio-system
  4. $ kubectl delete configmap nginx-configmap -n mesh-external
  5. $ kubectl delete service my-nginx -n mesh-external
  6. $ kubectl delete deployment my-nginx -n mesh-external
  7. $ kubectl delete namespace mesh-external
  8. $ kubectl delete gateway istio-egressgateway
  9. $ kubectl delete serviceentry nginx
  10. $ kubectl delete virtualservice direct-nginx-through-egress-gateway
  11. $ kubectl delete destinationrule originate-mtls-for-nginx
  12. $ kubectl delete destinationrule egressgateway-for-nginx
  • 删除证书的目录和用于生成它们的存储库：
  1. $ rm -rf nginx.example.com mtls-go-example
  • 删除此示例中使用的生成的配置文件：
  1. $ rm -f ./nginx.conf ./istio-egressgateway.yaml
  • 删除 sleep service 和 deployment：
  1. $ kubectl delete service sleep
  2. $ kubectl delete deployment sleep