Secure Gateways

The Control Ingress Traffic task describes how to configure an ingress gateway to expose an HTTP service to external traffic. This task shows how to expose a secure HTTPS service using either simple or mutual TLS.

Before you begin

  1. Perform the steps in the Before you begin. and Determining the ingress IP and ports sections of the Control Ingress Traffic task. After performing those steps you should have Istio and the httpbin service deployed, and the environment variables INGRESS_HOST and SECURE_INGRESS_PORT set.

  2. For macOS users, verify that you use curl compiled with the LibreSSL library:

    1. $ curl --version | grep LibreSSL
    2. curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0

    If the previous command outputs a version of LibreSSL as shown, your curl command should work correctly with the instructions in this task. Otherwise, try a different implementation of curl, for example on a Linux machine.

Generate client and server certificates and keys

For this task you can use your favorite tool to generate certificates and keys. The commands below use openssl

  1. Create a root certificate and private key to sign the certificates for your services:

    1. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt

  2. Create a certificate and a private key for httpbin.example.com:

    1. $ openssl req -out httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in httpbin.example.com.csr -out httpbin.example.com.crt

Configure a TLS ingress gateway for a single host

  1. Ensure you have deployed the httpbin service from Before you begin.

  2. Create a secret for the ingress gateway:

    1. $ kubectl create -n istio-system secret tls httpbin-credential --key=httpbin.example.com.key --cert=httpbin.example.com.crt

    The secret name should not begin with istio or prometheus, and the secret should not contain a token field.

  3. Define a gateway with a servers: section for port 443, and specify values for credentialName to be httpbin-credential. The values are the same as the secret’s name. The TLS mode should have the value of SIMPLE.

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: mygateway
    6. spec:
    7.   selector:
    8.     istio: ingressgateway # use istio default ingress gateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https
    13.       protocol: HTTPS
    14.     tls:
    15.       mode: SIMPLE
    16.       credentialName: httpbin-credential # must be the same as secret
    17.     hosts:
    18.     - httpbin.example.com
    19. EOF

  4. Configure the gateway’s ingress traffic routes. Define the corresponding virtual service.

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: httpbin
    6. spec:
    7.   hosts:
    8.   - "httpbin.example.com"
    9.   gateways:
    10.   - mygateway
    11.   http:
    12.   - match:
    13.     - uri:
    14.         prefix: /status
    15.     - uri:
    16.         prefix: /delay
    17.     route:
    18.     - destination:
    19.         port:
    20.           number: 8000
    21.         host: httpbin
    22. EOF

  5. Send an HTTPS request to access the httpbin service through HTTPS:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

    The httpbin service will return the 418 I’m a Teapot code.

  6. Delete the gateway’s secret and create a new one to change the ingress gateway’s credentials.

    1. $ kubectl -n istio-system delete secret httpbin-credential
    1. $ mkdir new_certificates
    2. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout new_certificates/example.com.key -out new_certificates/example.com.crt
    3. $ openssl req -out new_certificates/httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout new_certificates/httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
    4. $ openssl x509 -req -days 365 -CA new_certificates/example.com.crt -CAkey new_certificates/example.com.key -set_serial 0 -in new_certificates/httpbin.example.com.csr -out new_certificates/httpbin.example.com.crt
    5. $ kubectl create -n istio-system secret tls httpbin-credential \
    6. --key=new_certificates/httpbin.example.com.key \
    7. --cert=new_certificates/httpbin.example.com.crt

  7. Access the httpbin service using curl using the new certificate chain:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert new_certificates/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. ...
    4. HTTP/2 418
    5. ...
    6.     -=[ teapot ]=-
    8.        _...._
    9.      .'  _ _ `.
    10.     | ."` ^ `". _,
    11.     \_;`"---"`|//
    12.       |       ;/
    13.       \_     _/
    14.         `"""`

  8. If you try to access httpbin with the previous certificate chain, the attempt now fails.

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. ...
    4. * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    5. * TLSv1.2 (IN), TLS handshake, Server hello (2):
    6. * TLSv1.2 (IN), TLS handshake, Certificate (11):
    7. * TLSv1.2 (OUT), TLS alert, Server hello (2):
    8. * curl: (35) error:04FFF06A:rsa routines:CRYPTO_internal:block type is not 01

Configure a TLS ingress gateway for multiple hosts

You can configure an ingress gateway for multiple hosts, httpbin.example.com and helloworld-v1.example.com, for example. The ingress gateway retrieves unique credentials corresponding to a specific credentialName.

  1. To restore the credentials for httpbin, delete its secret and create it again.

    1. $ kubectl -n istio-system delete secret httpbin-credential
    2. $ kubectl create -n istio-system secret tls httpbin-credential \
    3. --key=httpbin.example.com.key \
    4. --cert=httpbin.example.com.crt

  2. Start the helloworld-v1 sample

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: helloworld-v1
    6.   labels:
    7.     app: helloworld-v1
    8. spec:
    9.   ports:
    10.   - name: http
    11.     port: 5000
    12.   selector:
    13.     app: helloworld-v1
    14. ---
    15. apiVersion: apps/v1
    16. kind: Deployment
    17. metadata:
    18.   name: helloworld-v1
    19. spec:
    20.   replicas: 1
    21.   selector:
    22.     matchLabels:
    23.       app: helloworld-v1
    24.       version: v1
    25.   template:
    26.     metadata:
    27.       labels:
    28.         app: helloworld-v1
    29.         version: v1
    30.     spec:
    31.       containers:
    32.       - name: helloworld
    33.         image: istio/examples-helloworld-v1
    34.         resources:
    35.           requests:
    36.             cpu: "100m"
    37.         imagePullPolicy: IfNotPresent #Always
    38.         ports:
    39.         - containerPort: 5000
    40. EOF

  3. Generate a certificate and a private key for helloworld-v1.example.com:

    1. $ openssl req -out helloworld-v1.example.com.csr -newkey rsa:2048 -nodes -keyout helloworld-v1.example.com.key -subj "/CN=helloworld-v1.example.com/O=helloworld organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in helloworld-v1.example.com.csr -out helloworld-v1.example.com.crt

  4. Create the helloworld-credential secret:

    1. $ kubectl create -n istio-system secret tls helloworld-credential --key=helloworld-v1.example.com.key --cert=helloworld-v1.example.com.crt

  5. Define a gateway with two server sections for port 443. Set the value of credentialName on each port to httpbin-credential and helloworld-credential respectively. Set TLS mode to SIMPLE.

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: mygateway
    6. spec:
    7.   selector:
    8.     istio: ingressgateway # use istio default ingress gateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https-httpbin
    13.       protocol: HTTPS
    14.     tls:
    15.       mode: SIMPLE
    16.       credentialName: httpbin-credential
    17.     hosts:
    18.     - httpbin.example.com
    19.   - port:
    20.       number: 443
    21.       name: https-helloworld
    22.       protocol: HTTPS
    23.     tls:
    24.       mode: SIMPLE
    25.       credentialName: helloworld-credential
    26.     hosts:
    27.     - helloworld-v1.example.com
    28. EOF

  6. Configure the gateway’s traffic routes. Define the corresponding virtual service.

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: helloworld-v1
    6. spec:
    7.   hosts:
    8.   - helloworld-v1.example.com
    9.   gateways:
    10.   - mygateway
    11.   http:
    12.   - match:
    13.     - uri:
    14.         exact: /hello
    15.     route:
    16.     - destination:
    17.         host: helloworld-v1
    18.         port:
    19.           number: 5000
    20. EOF

  7. Send an HTTPS request to helloworld-v1.example.com:

    1. $ curl -v -HHost:helloworld-v1.example.com --resolve "helloworld-v1.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert example.com.crt "https://helloworld-v1.example.com:$SECURE_INGRESS_PORT/hello"
    3. HTTP/2 200

  8. Send an HTTPS request to httpbin.example.com and still get a teapot in return:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. ...
    4.     -=[ teapot ]=-
    6.        _...._
    7.      .'  _ _ `.
    8.     | ."` ^ `". _,
    9.     \_;`"---"`|//
    10.       |       ;/
    11.       \_     _/
    12.         `"""`

Configure a mutual TLS ingress gateway

You can extend your gateway’s definition to support mutual TLS. Change the credentials of the ingress gateway by deleting its secret and creating a new one. The server uses the CA certificate to verify its clients, and we must use the name cacert to hold the CA certificate.

  1. $ kubectl -n istio-system delete secret httpbin-credential
  2. $ kubectl create -n istio-system secret generic httpbin-credential --from-file=tls.key=httpbin.example.com.key \
  3. --from-file=tls.crt=httpbin.example.com.crt --from-file=ca.crt=example.com.crt

  1. Change the gateway’s definition to set the TLS mode to MUTUAL.

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.  name: mygateway
    6. spec:
    7.  selector:
    8.    istio: ingressgateway # use istio default ingress gateway
    9.  servers:
    10.  - port:
    11.      number: 443
    12.      name: https
    13.      protocol: HTTPS
    14.    tls:
    15.      mode: MUTUAL
    16.      credentialName: httpbin-credential # must be the same as secret
    17.    hosts:
    18.    - httpbin.example.com
    19. EOF

  2. Attempt to send an HTTPS request using the prior approach and see how it fails:

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    3. * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    4. * TLSv1.3 (IN), TLS handshake, Server hello (2):
    5. * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
    6. * TLSv1.3 (IN), TLS handshake, Request CERT (13):
    7. * TLSv1.3 (IN), TLS handshake, Certificate (11):
    8. * TLSv1.3 (IN), TLS handshake, CERT verify (15):
    9. * TLSv1.3 (IN), TLS handshake, Finished (20):
    10. * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
    11. * TLSv1.3 (OUT), TLS handshake, Certificate (11):
    12. * TLSv1.3 (OUT), TLS handshake, Finished (20):
    13. * TLSv1.3 (IN), TLS alert, unknown (628):
    14. * OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0

  3. Generate client certificate and private key:

    1. $ openssl req -out client.example.com.csr -newkey rsa:2048 -nodes -keyout client.example.com.key -subj "/CN=client.example.com/O=client organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in client.example.com.csr -out client.example.com.crt

  4. Pass a client certificate and private key to curl and resend the request. Pass your client’s certificate with the --cert flag and your private key with the --key flag to curl.

    1. $ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
    2. --cacert example.com.crt --cert client.example.com.crt --key client.example.com.key \
    3. "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
    4. ...
    5.     -=[ teapot ]=-
    7.        _...._
    8.      .'  _ _ `.
    9.     | ."` ^ `". _,
    10.     \_;`"---"`|//
    11.       |       ;/
    12.       \_     _/
    13.         `"""`

Istio supports reading a few different Secret formats, to support integration with various tools such as cert-manager:

  • A TLS Secret with keys tls.key and tls.crt, as described above. For mutual TLS, a ca.crt key can be used.
  • A generic Secret with keys key and cert. For mutual TLS, a cacert key can be used.
  • A generic Secret with keys key and cert. For mutual TLS, a separate generic Secret named <secret>-cacert, with a cacert key. For example, httpbin-credential has key and cert, and httpbin-credential-cacert has cacert.

Troubleshooting

  • Inspect the values of the INGRESS_HOST and SECURE_INGRESS_PORT environment variables. Make sure they have valid values, according to the output of the following commands:

    1. $ kubectl get svc -n istio-system
    2. $ echo "INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT"

  • Check the log of the istio-ingressgateway controller for error messages:

    1. $ kubectl logs -n istio-system "$(kubectl get pod -l istio=ingressgateway \
    2. -n istio-system -o jsonpath='{.items[0].metadata.name}')"

  • If using macOS, verify you are using curl compiled with the LibreSSL library, as described in the Before you begin section.

  • Verify that the secrets are successfully created in the istio-system namespace:

    1. $ kubectl -n istio-system get secrets

    httpbin-credential and helloworld-credential should show in the secrets list.

  • Check the logs to verify that the ingress gateway agent has pushed the key/certificate pair to the ingress gateway.

    1. $ kubectl logs -n istio-system "$(kubectl get pod -l istio=ingressgateway \
    2. -n istio-system -o jsonpath='{.items[0].metadata.name}')"

    The log should show that the httpbin-credential secret was added. If using mutual TLS, then the httpbin-credential-cacert secret should also appear. Verify the log shows that the gateway agent receives SDS requests from the ingress gateway, that the resource’s name is httpbin-credential, and that the ingress gateway obtained the key/certificate pair. If using mutual TLS, the log should show key/certificate was sent to the ingress gateway, that the gateway agent received the SDS request with the httpbin-credential-cacert resource name, and that the ingress gateway obtained the root certificate.

Cleanup

  1. Delete the gateway configuration, the virtual service definition, and the secrets:

    1. $ kubectl delete gateway mygateway
    2. $ kubectl delete virtualservice httpbin
    3. $ kubectl delete --ignore-not-found=true -n istio-system secret httpbin-credential \
    4. helloworld-credential
    5. $ kubectl delete --ignore-not-found=true virtualservice helloworld-v1

  2. Delete the certificates and keys:

    1. $ rm -rf example.com.crt example.com.key httpbin.example.com.crt httpbin.example.com.key httpbin.example.com.csr helloworld-v1.example.com.crt helloworld-v1.example.com.key helloworld-v1.example.com.csr client.example.com.crt client.example.com.csr client.example.com.key ./new_certificates

  3. Shutdown the httpbin and helloworld-v1 services:

    1. $ kubectl delete deployment --ignore-not-found=true httpbin helloworld-v1
    2. $ kubectl delete service --ignore-not-found=true httpbin helloworld-v1

See also

Direct encrypted traffic from IBM Cloud Kubernetes Service Ingress to Istio Ingress Gateway

Configure the IBM Cloud Kubernetes Service Application Load Balancer to direct traffic to the Istio Ingress gateway with mutual TLS.

Istio as a Proxy for External Services

Configure Istio ingress gateway to act as a proxy for external services.

Deploy a Custom Ingress Gateway Using Cert-Manager

Describes how to deploy a custom ingress gateway using cert-manager manually.

Configuring Istio Ingress with AWS NLB

Describes how to configure Istio ingress with a network load balancer on AWS.

Configuring Gateway Network Topology [experimental]

How to configure gateway network topology.

Ingress (Kubernetes)

Describes how to configure a Kubernetes Ingress object to expose a service outside of the service mesh.