Secure Gateways

The Control Ingress Traffic task describes how to configure an ingress gateway to expose an HTTP service to external traffic. This task shows how to expose a secure HTTPS service using either simple or mutual TLS.

Istio includes beta support for the Kubernetes Gateway API and intends to make it the default API for traffic management in the future. The following instructions allow you to choose to use either the Gateway API or the Istio configuration API when configuring traffic management in the mesh. Follow instructions under either the Gateway API or Istio APIs tab, according to your preference.

Note that the Kubernetes Gateway API CRDs do not come installed by default on most Kubernetes clusters, so make sure they are installed before using the Gateway API:

  1. $ kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
  2.   { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.8.0" | kubectl apply -f -; }

Before you begin

  • Setup Istio by following the instructions in the Installation guide.

  • Start the httpbin sample:

    Zip

    1. $ kubectl apply -f @samples/httpbin/httpbin.yaml@

  • For macOS users, verify that you use curl compiled with the LibreSSL library:

    1. $ curl --version | grep LibreSSL
    2. curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0

    If the previous command outputs a version of LibreSSL as shown, your curl command should work correctly with the instructions in this task. Otherwise, try a different implementation of curl, for example on a Linux machine.

Generate client and server certificates and keys

This task requires several sets of certificates and keys which are used in the following examples. You can use your favorite tool to create them or use the commands below to generate them using openssl.

  1. Create a root certificate and private key to sign the certificates for your services:

    1. $ mkdir example_certs1
    2. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example_certs1/example.com.key -out example_certs1/example.com.crt

  2. Generate a certificate and a private key for httpbin.example.com:

    1. $ openssl req -out example_certs1/httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs1/httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
    2. $ openssl x509 -req -sha256 -days 365 -CA example_certs1/example.com.crt -CAkey example_certs1/example.com.key -set_serial 0 -in example_certs1/httpbin.example.com.csr -out example_certs1/httpbin.example.com.crt

  3. Create a second set of the same kind of certificates and keys:

    1. $ mkdir example_certs2
    2. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example_certs2/example.com.key -out example_certs2/example.com.crt
    3. $ openssl req -out example_certs2/httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs2/httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
    4. $ openssl x509 -req -sha256 -days 365 -CA example_certs2/example.com.crt -CAkey example_certs2/example.com.key -set_serial 0 -in example_certs2/httpbin.example.com.csr -out example_certs2/httpbin.example.com.crt

  4. Generate a certificate and a private key for helloworld.example.com:

    1. $ openssl req -out example_certs1/helloworld.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs1/helloworld.example.com.key -subj "/CN=helloworld.example.com/O=helloworld organization"
    2. $ openssl x509 -req -sha256 -days 365 -CA example_certs1/example.com.crt -CAkey example_certs1/example.com.key -set_serial 1 -in example_certs1/helloworld.example.com.csr -out example_certs1/helloworld.example.com.crt

  5. Generate a client certificate and private key:

    1. $ openssl req -out example_certs1/client.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs1/client.example.com.key -subj "/CN=client.example.com/O=client organization"
    2. $ openssl x509 -req -sha256 -days 365 -CA example_certs1/example.com.crt -CAkey example_certs1/example.com.key -set_serial 1 -in example_certs1/client.example.com.csr -out example_certs1/client.example.com.crt

You can confirm that you have all of the needed files by running the following command:

  1. $ ls example_cert*
  2. example_certs1:
  3. client.example.com.crt          example.com.key                 httpbin.example.com.crt
  4. client.example.com.csr          helloworld.example.com.crt      httpbin.example.com.csr
  5. client.example.com.key          helloworld.example.com.csr      httpbin.example.com.key
  6. example.com.crt                 helloworld.example.com.key
  8. example_certs2:
  9. example.com.crt         httpbin.example.com.crt httpbin.example.com.key
  10. example.com.key         httpbin.example.com.csr

Configure a TLS ingress gateway for a single host

  1. Create a secret for the ingress gateway:

    1. $ kubectl create -n istio-system secret tls httpbin-credential \
    2.   --key=example_certs1/httpbin.example.com.key \
    3.   --cert=example_certs1/httpbin.example.com.crt

  2. Configure the ingress gateway:

First, define a gateway with a servers: section for port 443, and specify values for credentialName to be httpbin-credential. The values are the same as the secret’s name. The TLS mode should have the value of SIMPLE.

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: mygateway
  6. spec:
  7.   selector:
  8.     istio: ingressgateway # use istio default ingress gateway
  9.   servers:
  10.   - port:
  11.       number: 443
  12.       name: https
  13.       protocol: HTTPS
  14.     tls:
  15.       mode: SIMPLE
  16.       credentialName: httpbin-credential # must be the same as secret
  17.     hosts:
  18.     - httpbin.example.com
  19. EOF

Next, configure the gateway’s ingress traffic routes by defining a corresponding virtual service:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: httpbin
  6. spec:
  7.   hosts:
  8.   - "httpbin.example.com"
  9.   gateways:
  10.   - mygateway
  11.   http:
  12.   - match:
  13.     - uri:
  14.         prefix: /status
  15.     - uri:
  16.         prefix: /delay
  17.     route:
  18.     - destination:
  19.         port:
  20.           number: 8000
  21.         host: httpbin
  22. EOF

Finally, follow these instructions to set the INGRESS_HOST and SECURE_INGRESS_PORT variables for accessing the gateway.

First, create a Kubernetes Gateway:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: gateway.networking.k8s.io/v1beta1
  3. kind: Gateway
  4. metadata:
  5.   name: mygateway
  6.   namespace: istio-system
  7. spec:
  8.   gatewayClassName: istio
  9.   listeners:
  10.   - name: https
  11.     hostname: "httpbin.example.com"
  12.     port: 443
  13.     protocol: HTTPS
  14.     tls:
  15.       mode: Terminate
  16.       certificateRefs:
  17.       - name: httpbin-credential
  18.     allowedRoutes:
  19.       namespaces:
  20.         from: Selector
  21.         selector:
  22.           matchLabels:
  23.             kubernetes.io/metadata.name: default
  24. EOF

Next, configure the gateway’s ingress traffic routes by defining a corresponding HTTPRoute:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: gateway.networking.k8s.io/v1beta1
  3. kind: HTTPRoute
  4. metadata:
  5.   name: httpbin
  6. spec:
  7.   parentRefs:
  8.   - name: mygateway
  9.     namespace: istio-system
  10.   hostnames: ["httpbin.example.com"]
  11.   rules:
  12.   - matches:
  13.     - path:
  14.         type: PathPrefix
  15.         value: /status
  16.     - path:
  17.         type: PathPrefix
  18.         value: /delay
  19.     backendRefs:
  20.     - name: httpbin
  21.       port: 8000
  22. EOF

Finally, get the gateway address and port from the Gateway resource:

  1. $ kubectl wait --for=condition=programmed gtw mygateway -n istio-system
  2. $ export INGRESS_HOST=$(kubectl get gtw mygateway -n istio-system -o jsonpath='{.status.addresses[0].value}')
  3. $ export SECURE_INGRESS_PORT=$(kubectl get gtw mygateway -n istio-system -o jsonpath='{.spec.listeners[?(@.name=="https")].port}')

  1. Send an HTTPS request to access the httpbin service through HTTPS:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2.   --cacert example_certs1/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. ...
    4. HTTP/2 418
    5. ...
    6.     -=[ teapot ]=-
    8.        _...._
    9.      .'  _ _ `.
    10.     | ."` ^ `". _,
    11.     \_;`"---"`|//
    12.       |       ;/
    13.       \_     _/
    14.         `"""`

    The httpbin service will return the 418 I’m a Teapot code.

  2. Change the gateway’s credentials by deleting the gateway’s secret and then recreating it using different certificates and keys:

    1. $ kubectl -n istio-system delete secret httpbin-credential
    2. $ kubectl create -n istio-system secret tls httpbin-credential \
    3.   --key=example_certs2/httpbin.example.com.key \
    4.   --cert=example_certs2/httpbin.example.com.crt

  3. Access the httpbin service with curl using the new certificate chain:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2.   --cacert example_certs2/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. ...
    4. HTTP/2 418
    5. ...
    6.     -=[ teapot ]=-
    8.        _...._
    9.      .'  _ _ `.
    10.     | ."` ^ `". _,
    11.     \_;`"---"`|//
    12.       |       ;/
    13.       \_     _/
    14.         `"""`

  4. If you try to access httpbin using the previous certificate chain, the attempt now fails:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2.   --cacert example_certs1/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. ...
    4. * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    5. * TLSv1.2 (IN), TLS handshake, Server hello (2):
    6. * TLSv1.2 (IN), TLS handshake, Certificate (11):
    7. * TLSv1.2 (OUT), TLS alert, Server hello (2):
    8. * curl: (35) error:04FFF06A:rsa routines:CRYPTO_internal:block type is not 01

Configure a TLS ingress gateway for multiple hosts

You can configure an ingress gateway for multiple hosts, httpbin.example.com and helloworld.example.com, for example. The ingress gateway is configured with unique credentials corresponding to each host.

  1. Restore the httpbin credentials from the previous example by deleting and recreating the secret with the original certificates and keys:

    1. $ kubectl -n istio-system delete secret httpbin-credential
    2. $ kubectl create -n istio-system secret tls httpbin-credential \
    3.   --key=example_certs1/httpbin.example.com.key \
    4.   --cert=example_certs1/httpbin.example.com.crt

  2. Start the helloworld-v1 sample:

    ZipZip

    1. $ kubectl apply -f @samples/helloworld/helloworld.yaml@ -l service=helloworld
    2. $ kubectl apply -f @samples/helloworld/helloworld.yaml@ -l version=v1

  3. Create a helloworld-credential secret:

    1. $ kubectl create -n istio-system secret tls helloworld-credential \
    2.   --key=example_certs1/helloworld.example.com.key \
    3.   --cert=example_certs1/helloworld.example.com.crt

  4. Configure the ingress gateway with hosts httpbin.example.com and helloworld.example.com:

Define a gateway with two server sections for port 443. Set the value of credentialName on each port to httpbin-credential and helloworld-credential respectively. Set TLS mode to SIMPLE.

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: mygateway
  6. spec:
  7.   selector:
  8.     istio: ingressgateway # use istio default ingress gateway
  9.   servers:
  10.   - port:
  11.       number: 443
  12.       name: https-httpbin
  13.       protocol: HTTPS
  14.     tls:
  15.       mode: SIMPLE
  16.       credentialName: httpbin-credential
  17.     hosts:
  18.     - httpbin.example.com
  19.   - port:
  20.       number: 443
  21.       name: https-helloworld
  22.       protocol: HTTPS
  23.     tls:
  24.       mode: SIMPLE
  25.       credentialName: helloworld-credential
  26.     hosts:
  27.     - helloworld.example.com
  28. EOF

Configure the gateway’s traffic routes by defining a corresponding virtual service.

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: helloworld
  6. spec:
  7.   hosts:
  8.   - helloworld.example.com
  9.   gateways:
  10.   - mygateway
  11.   http:
  12.   - match:
  13.     - uri:
  14.         exact: /hello
  15.     route:
  16.     - destination:
  17.         host: helloworld
  18.         port:
  19.           number: 5000
  20. EOF

Configure a Gateway with two listeners for port 443. Set the value of certificateRefs on each listener to httpbin-credential and helloworld-credential respectively.

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: gateway.networking.k8s.io/v1beta1
  3. kind: Gateway
  4. metadata:
  5.   name: mygateway
  6.   namespace: istio-system
  7. spec:
  8.   gatewayClassName: istio
  9.   listeners:
  10.   - name: https-httpbin
  11.     hostname: "httpbin.example.com"
  12.     port: 443
  13.     protocol: HTTPS
  14.     tls:
  15.       mode: Terminate
  16.       certificateRefs:
  17.       - name: httpbin-credential
  18.     allowedRoutes:
  19.       namespaces:
  20.         from: Selector
  21.         selector:
  22.           matchLabels:
  23.             kubernetes.io/metadata.name: default
  24.   - name: https-helloworld
  25.     hostname: "helloworld.example.com"
  26.     port: 443
  27.     protocol: HTTPS
  28.     tls:
  29.       mode: Terminate
  30.       certificateRefs:
  31.       - name: helloworld-credential
  32.     allowedRoutes:
  33.       namespaces:
  34.         from: Selector
  35.         selector:
  36.           matchLabels:
  37.             kubernetes.io/metadata.name: default
  38. EOF

Configure the gateway’s traffic routes for the helloworld service:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: gateway.networking.k8s.io/v1beta1
  3. kind: HTTPRoute
  4. metadata:
  5.   name: helloworld
  6. spec:
  7.   parentRefs:
  8.   - name: mygateway
  9.     namespace: istio-system
  10.   hostnames: ["helloworld.example.com"]
  11.   rules:
  12.   - matches:
  13.     - path:
  14.         type: Exact
  15.         value: /hello
  16.     backendRefs:
  17.     - name: helloworld
  18.       port: 5000
  19. EOF

  1. Send an HTTPS request to helloworld.example.com:

    1. $ curl -v -HHost:helloworld.example.com --resolve "helloworld.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2.   --cacert example_certs1/example.com.crt "https://helloworld.example.com:$SECURE_INGRESS_PORT/hello"
    3. ...
    4. HTTP/2 200
    5. ...

  2. Send an HTTPS request to httpbin.example.com and still get a teapot in return:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2.   --cacert example_certs1/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. ...
    4.     -=[ teapot ]=-
    6.        _...._
    7.      .'  _ _ `.
    8.     | ."` ^ `". _,
    9.     \_;`"---"`|//
    10.       |       ;/
    11.       \_     _/
    12.         `"""`

Configure a mutual TLS ingress gateway

You can extend your gateway’s definition to support mutual TLS.

  1. Change the credentials of the ingress gateway by deleting its secret and creating a new one. The server uses the CA certificate to verify its clients, and we must use the key ca.crt to hold the CA certificate.

    1. $ kubectl -n istio-system delete secret httpbin-credential
    2. $ kubectl create -n istio-system secret generic httpbin-credential \
    3.   --from-file=tls.key=example_certs1/httpbin.example.com.key \
    4.   --from-file=tls.crt=example_certs1/httpbin.example.com.crt \
    5.   --from-file=ca.crt=example_certs1/example.com.crt

    Optionally, the credential may include a certificate revocation list (CRL) using the key ca.crl. If so, add another argument to the above example to provide the CRL: –from-file=ca.crl=/some/path/to/your-crl.pem.

    The credential may also include an OCSP Staple using the key tls.ocsp-staple which can be specified by an additional argument: --from-file=tls.ocsp-staple=/some/path/to/your-ocsp-staple.pem.

  2. Configure the ingress gateway:

Change the gateway’s definition to set the TLS mode to MUTUAL.

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: mygateway
  6. spec:
  7.   selector:
  8.     istio: ingressgateway # use istio default ingress gateway
  9.   servers:
  10.   - port:
  11.       number: 443
  12.       name: https
  13.       protocol: HTTPS
  14.     tls:
  15.       mode: MUTUAL
  16.       credentialName: httpbin-credential # must be the same as secret
  17.     hosts:
  18.     - httpbin.example.com
  19. EOF

Because the Kubernetes Gateway API does not currently support mutual TLS termination in a Gateway, we use an Istio-specific option, gateway.istio.io/tls-terminate-mode: MUTUAL, to configure it:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: gateway.networking.k8s.io/v1beta1
  3. kind: Gateway
  4. metadata:
  5.   name: mygateway
  6.   namespace: istio-system
  7. spec:
  8.   gatewayClassName: istio
  9.   listeners:
  10.   - name: https
  11.     hostname: "httpbin.example.com"
  12.     port: 443
  13.     protocol: HTTPS
  14.     tls:
  15.       mode: Terminate
  16.       certificateRefs:
  17.       - name: httpbin-credential
  18.       options:
  19.         gateway.istio.io/tls-terminate-mode: MUTUAL
  20.     allowedRoutes:
  21.       namespaces:
  22.         from: Selector
  23.         selector:
  24.           matchLabels:
  25.             kubernetes.io/metadata.name: default
  26. EOF

  1. Attempt to send an HTTPS request using the prior approach and see how it fails:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert example_certs1/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    4. * TLSv1.3 (IN), TLS handshake, Server hello (2):
    5. * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
    6. * TLSv1.3 (IN), TLS handshake, Request CERT (13):
    7. * TLSv1.3 (IN), TLS handshake, Certificate (11):
    8. * TLSv1.3 (IN), TLS handshake, CERT verify (15):
    9. * TLSv1.3 (IN), TLS handshake, Finished (20):
    10. * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
    11. * TLSv1.3 (OUT), TLS handshake, Certificate (11):
    12. * TLSv1.3 (OUT), TLS handshake, Finished (20):
    13. * TLSv1.3 (IN), TLS alert, unknown (628):
    14. * OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0

  2. Pass a client certificate and private key to curl and resend the request. Pass your client’s certificate with the --cert flag and your private key with the --key flag to curl:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2.   --cacert example_certs1/example.com.crt --cert example_certs1/client.example.com.crt --key example_certs1/client.example.com.key \
    3.   "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    4. ...
    5.     -=[ teapot ]=-
    7.        _...._
    8.      .'  _ _ `.
    9.     | ."` ^ `". _,
    10.     \_;`"---"`|//
    11.       |       ;/
    12.       \_     _/
    13.         `"""`

More info

Key formats

Istio supports reading a few different Secret formats, to support integration with various tools such as cert-manager:

  • A TLS Secret with keys tls.key and tls.crt, as described above. For mutual TLS, a ca.crt key can be used.
  • A generic Secret with keys key and cert. For mutual TLS, a cacert key can be used.
  • A generic Secret with keys key and cert. For mutual TLS, a separate generic Secret named <secret>-cacert, with a cacert key. For example, httpbin-credential has key and cert, and httpbin-credential-cacert has cacert.
  • The cacert key value can be a CA bundle consisting of concatenated individual CA certificates.

SNI Routing

An HTTPS Gateway will perform SNI matching against its configured host(s) before forwarding a request, which may cause some requests to fail. See configuring SNI routing for details.

Troubleshooting

  • Inspect the values of the INGRESS_HOST and SECURE_INGRESS_PORT environment variables. Make sure they have valid values, according to the output of the following commands:

    1. $ kubectl get svc -n istio-system
    2. $ echo "INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT"

  • Make sure the value of INGRESS_HOST is an IP address. In some cloud platforms, e.g., AWS, you may get a domain name, instead. This task expects an IP address, so you will need to convert it with commands similar to the following:

    1. $ nslookup ab52747ba608744d8afd530ffd975cbf-330887905.us-east-1.elb.amazonaws.com
    2. $ export INGRESS_HOST=3.225.207.109

  • Check the log of the gateway controller for error messages:

    1. $ kubectl logs -n istio-system <gateway-service-pod>

  • If using macOS, verify you are using curl compiled with the LibreSSL library, as described in the Before you begin section.

  • Verify that the secrets are successfully created in the istio-system namespace:

    1. $ kubectl -n istio-system get secrets

    httpbin-credential and helloworld-credential should show in the secrets list.

  • Check the logs to verify that the ingress gateway agent has pushed the key/certificate pair to the ingress gateway:

    1. $ kubectl logs -n istio-system <gateway-service-pod>

    The log should show that the httpbin-credential secret was added. If using mutual TLS, then the httpbin-credential-cacert secret should also appear. Verify the log shows that the gateway agent receives SDS requests from the ingress gateway, that the resource’s name is httpbin-credential, and that the ingress gateway obtained the key/certificate pair. If using mutual TLS, the log should show key/certificate was sent to the ingress gateway, that the gateway agent received the SDS request with the httpbin-credential-cacert resource name, and that the ingress gateway obtained the root certificate.

Cleanup

  1. Delete the gateway configuration and routes:
  1. $ kubectl delete gateway mygateway
  2. $ kubectl delete virtualservice httpbin helloworld
  1. $ kubectl delete -n istio-system gtw mygateway
  2. $ kubectl delete httproute httpbin helloworld

  1. Delete the secrets, certificates and keys:

    1. $ kubectl delete -n istio-system secret httpbin-credential helloworld-credential
    2. $ rm -rf ./example_certs1 ./example_certs2

  2. Shutdown the httpbin and helloworld services:

    1. $ kubectl delete -f samples/httpbin/httpbin.yaml
    2. $ kubectl delete deployment helloworld-v1
    3. $ kubectl delete service helloworld