Accessing External Services

Because all outbound traffic from an Istio-enabled pod is redirected to its sidecar proxy by default,accessibility of URLs outside of the cluster depends on the configuration of the proxy.By default, Istio configures the Envoy proxy to passthrough requests for unknown services.Although this provides a convenient way to get started with Istio, configuringstricter control is usually preferable.

This task shows you how to access external services in three different ways:

  • Allow the Envoy proxy to pass requests through to services that are not configured inside the mesh.
  • Configure service entries to provide controlled access to external services.
  • Completely bypass the Envoy proxy for a specific range of IPs.

Before you begin

Zip

  1. $ kubectl apply -f @samples/sleep/sleep.yaml@

Otherwise, manually inject the sidecar before deploying the sleep application with the following command:

Zip

  1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

You can use any pod with curl installed as a test source.

  • Set the SOURCE_POD environment variable to the name of your source pod:
  1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

Envoy passthrough to external services

Istio has an installation option,global.outboundTrafficPolicy.mode, that configures the sidecar handlingof external services, that is, those services that are not defined in Istio’s internal service registry.If this option is set to ALLOW_ANY, the Istio proxy lets calls to unknown services pass through.If the option is set to REGISTRY_ONLY, then the Istio proxy blocks any host without an HTTP service orservice entry defined within the mesh.ALLOW_ANY is the default value, allowing you to start evaluating Istio quickly,without controlling access to external services.You can then decide to configure access to external services later.

  • To see this approach in action you need to ensure that your Istio installation is configuredwith the global.outboundTrafficPolicy.mode option set to ALLOW_ANY. Unless you explicitlyset it to REGISTRY_ONLY mode when you installed Istio, it is probably enabled by default.

Run the following command to confirm it is configured correctly:

  1. $ kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: ALLOW_ANY"
  2. mode: ALLOW_ANY

The string mode: ALLOW_ANY should appear in the output if it is enabled.

If you have explicitly configured REGISTRY_ONLY mode, you can run the following command to change it:

  1. $ kubectl get configmap istio -n istio-system -o yaml | sed 's/mode: REGISTRY_ONLY/mode: ALLOW_ANY/g' | kubectl replace -n istio-system -f -
  2. configmap "istio" replaced
  • Make a couple of requests to external HTTPS services from the SOURCE_POD to confirmsuccessful 200 responses:
  1. $ kubectl exec -it $SOURCE_POD -c sleep -- curl -I https://www.google.com | grep  "HTTP/"; kubectl exec -it $SOURCE_POD -c sleep -- curl -I https://edition.cnn.com | grep "HTTP/"
  2. HTTP/2 200
  3. HTTP/2 200

Congratulations! You successfully sent egress traffic from your mesh.

This simple approach to access external services, has the drawback that you lose Istio monitoring and controlfor traffic to external services; calls to external services will not appear in the Mixer log, for example.The next section shows you how to monitor and control your mesh’s access to external services.

Controlled access to external services

Using Istio ServiceEntry configurations, you can access any publicly accessible servicefrom within your Istio cluster. This section shows you how to configure access to an external HTTP service,httpbin.org, as well as an external HTTPS service,www.google.com without losing Istio’s traffic monitoring and control features.

Change to the blocking-by-default policy

To demonstrate the controlled way of enabling access to external services, you need to change theglobal.outboundTrafficPolicy.mode option from the ALLOW_ANY mode to the REGISTRY_ONLY mode.

You can add controlled access to services that are already accessible in ALLOW_ANY mode.This way, you can start using Istio features on some external services without blocking any others.Once you’ve configured all of your services, you can then switch the mode to REGISTRY_ONLY to blockany other unintentional accesses.

  • Run the following command to change the global.outboundTrafficPolicy.mode option to REGISTRY_ONLY:
  1. $ kubectl get configmap istio -n istio-system -o yaml | sed 's/mode: ALLOW_ANY/mode: REGISTRY_ONLY/g' | kubectl replace -n istio-system -f -
  2. configmap "istio" replaced
  • Make a couple of requests to external HTTPS services from SOURCE_POD to verify that they are now blocked:
  1. $ kubectl exec -it $SOURCE_POD -c sleep -- curl -I https://www.google.com | grep  "HTTP/"; kubectl exec -it $SOURCE_POD -c sleep -- curl -I https://edition.cnn.com | grep "HTTP/"
  2. command terminated with exit code 35
  3. command terminated with exit code 35

It may take a while for the configuration change to propagate, so you might still get successful connections.Wait for several seconds and then retry the last command.

Access an external HTTP service

  • Create a ServiceEntry to allow access to an external HTTP service:
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: ServiceEntry
  4. metadata:
  5.   name: httpbin-ext
  6. spec:
  7.   hosts:
  8.   - httpbin.org
  9.   ports:
  10.   - number: 80
  11.     name: http
  12.     protocol: HTTP
  13.   resolution: DNS
  14.   location: MESH_EXTERNAL
  15. EOF
  • Make a request to the external HTTP service from SOURCE_POD:
  1. $  kubectl exec -it $SOURCE_POD -c sleep -- curl http://httpbin.org/headers
  2. {
  3.   "headers": {
  4.   "Accept": "*/*",
  5.   "Connection": "close",
  6.   "Host": "httpbin.org",
  7.   "User-Agent": "curl/7.60.0",
  8.   ...
  9.   "X-Envoy-Decorator-Operation": "httpbin.org:80/*",
  10.   }
  11. }

Note the headers added by the Istio sidecar proxy: X-Envoy-Decorator-Operation.

  • Check the log of the sidecar proxy of SOURCE_POD:
  1. $  kubectl logs $SOURCE_POD -c istio-proxy | tail
  2. [2019-01-24T12:17:11.640Z] "GET /headers HTTP/1.1" 200 - 0 599 214 214 "-" "curl/7.60.0" "17fde8f7-fa62-9b39-8999-302324e6def2" "httpbin.org" "35.173.6.94:80" outbound|80||httpbin.org - 35.173.6.94:80 172.30.109.82:55314 -

Note the entry related to your HTTP request to httpbin.org/headers.

  • Check the Mixer log. If Istio is deployed in the istio-system namespace, the command to print the log is:
  1. $ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep 'httpbin.org'
  2. {"level":"info","time":"2019-01-24T12:17:11.855496Z","instance":"accesslog.logentry.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"unknown","destinationApp":"","destinationIp":"I60GXg==","destinationName":"unknown","destinationNamespace":"default","destinationOwner":"unknown","destinationPrincipal":"","destinationServiceHost":"httpbin.org","destinationWorkload":"unknown","grpcMessage":"","grpcStatus":"","httpAuthority":"httpbin.org","latency":"214.661667ms","method":"GET","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":270,"referer":"","reporter":"source","requestId":"17fde8f7-fa62-9b39-8999-302324e6def2","requestSize":0,"requestedServerName":"","responseCode":200,"responseSize":599,"responseTimestamp":"2019-01-24T12:17:11.855521Z","sentBytes":806,"sourceApp":"sleep","sourceIp":"AAAAAAAAAAAAAP//rB5tUg==","sourceName":"sleep-88ddbcfdd-rgk77","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/deployments/sleep","sourcePrincipal":"","sourceWorkload":"sleep","url":"/headers","userAgent":"curl/7.60.0","xForwardedFor":"0.0.0.0"}

Note that the destinationServiceHost attribute is equal to httpbin.org. Also notice the HTTP-related attributes:method, url, responseCode and others. Using Istio egress traffic control, you can monitor access to externalHTTP services, including the HTTP-related information of each access.

Access an external HTTPS service

  • Create a ServiceEntry to allow access to an external HTTPS service.
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: ServiceEntry
  4. metadata:
  5.   name: google
  6. spec:
  7.   hosts:
  8.   - www.google.com
  9.   ports:
  10.   - number: 443
  11.     name: https
  12.     protocol: HTTPS
  13.   resolution: DNS
  14.   location: MESH_EXTERNAL
  15. EOF
  • Make a request to the external HTTPS service from SOURCE_POD:
  1. $ kubectl exec -it $SOURCE_POD -c sleep -- curl -I https://www.google.com | grep  "HTTP/"
  2. HTTP/2 200
  • Check the log of the sidecar proxy of SOURCE_POD:
  1. $ kubectl logs $SOURCE_POD -c istio-proxy | tail
  2. [2019-01-24T12:48:54.977Z] "- - -" 0 - 601 17766 1289 - "-" "-" "-" "-" "172.217.161.36:443" outbound|443||www.google.com 172.30.109.82:59480 172.217.161.36:443 172.30.109.82:59478 www.google.com

Note the entry related to your HTTPS request to www.google.com.

  • Check the Mixer log. If Istio is deployed in the istio-system namespace, the command to print the log is:
  1. $ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep 'www.google.com'
  2. {"level":"info","time":"2019-01-24T12:48:56.266553Z","instance":"tcpaccesslog.logentry.istio-system","connectionDuration":"1.289085134s","connectionEvent":"close","connection_security_policy":"unknown","destinationApp":"","destinationIp":"rNmhJA==","destinationName":"unknown","destinationNamespace":"default","destinationOwner":"unknown","destinationPrincipal":"","destinationServiceHost":"www.google.com","destinationWorkload":"unknown","protocol":"tcp","receivedBytes":601,"reporter":"source","requestedServerName":"www.google.com","sentBytes":17766,"sourceApp":"sleep","sourceIp":"rB5tUg==","sourceName":"sleep-88ddbcfdd-rgk77","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/deployments/sleep","sourcePrincipal":"","sourceWorkload":"sleep","totalReceivedBytes":601,"totalSentBytes":17766}

Note that the requestedServerName attribute is equal to www.google.com. Using Istio egress traffic control, youcan monitor access to external HTTPS services, in particular theSNI and the number of sent and received bytes. Note that inHTTPS all the HTTP-related information like method, URL path, response code, is encrypted so Istio cannot see andcannot monitor that information for HTTPS. If you need to monitor HTTP-related information in access to externalHTTPS services, you may want to let your applications issue HTTP requests andconfigure Istio to perform TLS origination.

Manage traffic to external services

Similar to inter-cluster requests, Istiorouting rulescan also be set for external services that are accessed using ServiceEntry configurations.In this example, you set a timeout rule on calls to the httpbin.org service.

  • From inside the pod being used as the test source, make a curl request to the /delay endpoint of thehttpbin.org external service:
  1. $ kubectl exec -it $SOURCE_POD -c sleep sh
  2. $ time curl -o /dev/null -s -w "%{http_code}\n" http://httpbin.org/delay/5
  3. 200
  5. real    0m5.024s
  6. user    0m0.003s
  7. sys     0m0.003s

The request should return 200 (OK) in approximately 5 seconds.

  • Exit the source pod and use kubectl to set a 3s timeout on calls to the httpbin.org external service:
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5.   name: httpbin-ext
  6. spec:
  7.   hosts:
  8.     - httpbin.org
  9.   http:
  10.   - timeout: 3s
  11.     route:
  12.       - destination:
  13.           host: httpbin.org
  14.         weight: 100
  15. EOF
  • Wait a few seconds, then make the curl request again:
  1. $ kubectl exec -it $SOURCE_POD -c sleep sh
  2. $ time curl -o /dev/null -s -w "%{http_code}\n" http://httpbin.org/delay/5
  3. 504
  5. real    0m3.149s
  6. user    0m0.004s
  7. sys     0m0.004s

This time a 504 (Gateway Timeout) appears after 3 seconds.Although httpbin.org was waiting 5 seconds, Istio cut off the request at 3 seconds.

Cleanup the controlled access to external services

  1. $ kubectl delete serviceentry httpbin-ext google
  2. $ kubectl delete virtualservice httpbin-ext --ignore-not-found=true

Direct access to external services

If you want to completely bypass Istio for a specific IP range,you can configure the Envoy sidecars to prevent them frominterceptingexternal requests. To set up the bypass, change either the global.proxy.includeIPRangesor the global.proxy.excludeIPRanges configuration option andupdate the istio-sidecar-injector configuration map using the kubectl apply command. This can alsobe configured on a pod by setting corresponding annotations such astraffic.sidecar.istio.io/includeOutboundIPRanges.After updating the istio-sidecar-injector configuration, it affects allfuture application pod deployments.

Unlike Envoy passthrough to external services,which uses the ALLOW_ANY traffic policy to instruct the Istio sidecar proxy topassthrough calls to unknown services,this approach completely bypasses the sidecar, essentially disabling all of Istio’s featuresfor the specified IPs. You cannot incrementally add service entries for specificdestinations, as you can with the ALLOW_ANY approach.Therefore, this configuration approach is only recommended as a last resortwhen, for performance or other reasons, external access cannot be configured using the sidecar.

A simple way to exclude all external IPs from being redirected to the sidecar proxy isto set the global.proxy.includeIPRanges configuration option to the IP range or rangesused for internal cluster services.These IP range values depend on the platform where your cluster runs.

Determine the internal IP ranges for your platform

Set the value of global.proxy.includeIPRanges according to your cluster provider.

IBM Cloud Private

  • Get your service_cluster_ip_range from IBM Cloud Private configuration file under cluster/config.yaml:
  1. $ cat cluster/config.yaml | grep service_cluster_ip_range

The following is a sample output:

  1. service_cluster_ip_range: 10.0.0.1/24
  • Use —set global.proxy.includeIPRanges="10.0.0.1/24"

IBM Cloud Kubernetes Service

Use —set global.proxy.includeIPRanges="172.30.0.0/16\,172.21.0.0/16\,10.10.10.0/24"

Google Container Engine (GKE)

The ranges are not fixed, so you will need to run the gcloud container clusters describe command to determine theranges to use. For example:

  1. $ gcloud container clusters describe XXXXXXX --zone=XXXXXX | grep -e clusterIpv4Cidr -e servicesIpv4Cidr
  2. clusterIpv4Cidr: 10.4.0.0/14
  3. servicesIpv4Cidr: 10.7.240.0/20

Use —set global.proxy.includeIPRanges="10.4.0.0/14\,10.7.240.0/20"

Azure Container Service(ACS)

Use —set global.proxy.includeIPRanges="10.244.0.0/16\,10.240.0.0/16

Minikube, Docker For Desktop, Bare Metal

The default value is 10.96.0.0/12, but it’s not fixed. Use the following command to determine your actual value:

  1. $ kubectl describe pod kube-apiserver -n kube-system | grep 'service-cluster-ip-range'
  2.       --service-cluster-ip-range=10.96.0.0/12

Use —set global.proxy.includeIPRanges="10.96.0.0/12"

Configuring the proxy bypass

Remove the service entry and virtual service previously deployed in this guide.

Update your istio-sidecar-injector configuration map using the IP ranges specific to your platform.For example, if the range is 10.0.0.1/24, use the following command:

  1. $ istioctl manifest apply <the flags you used to install Istio> --set values.global.proxy.includeIPRanges="10.0.0.1/24"

Use the same command that you used to install Istio andadd —set values.global.proxy.includeIPRanges="10.0.0.1/24".

Access the external services

Because the bypass configuration only affects new deployments, you need to redeploy the sleepapplication as described in the Before you begin section.

After updating the istio-sidecar-injector configmap and redeploying the sleep application,the Istio sidecar will only intercept and manage internal requestswithin the cluster. Any external request bypasses the sidecar and goes straight to its intended destination.For example:

  1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
  2. $ kubectl exec -it $SOURCE_POD -c sleep curl http://httpbin.org/headers
  3. {
  4.   "headers": {
  5.     "Accept": "*/*",
  6.     "Connection": "close",
  7.     "Host": "httpbin.org",
  8.     "User-Agent": "curl/7.60.0"
  9.   }
  10. }

Unlike accessing external services through HTTP or HTTPS, you don’t see any headers related to the Istio sidecar and therequests sent to external services appear neither in the log of the sidecar nor in the Mixer log.Bypassing the Istio sidecars means you can no longer monitor the access to external services.

Cleanup the direct access to external services

Update the istio-sidecar-injector.configmap.yaml configuration map to redirect all outbound traffic to the sidecarproxies:

  1. $ istioctl manifest apply <the flags you used to install Istio>

Understanding what happened

In this task you looked at three ways to call external services from an Istio mesh:

  • Configuring Envoy to allow access to any external service.

  • Use a service entry to register an accessible external service inside the mesh. This is therecommended approach.

  • Configuring the Istio sidecar to exclude external IPs from its remapped IP table.

The first approach directs traffic through the Istio sidecar proxy, including calls to servicesthat are unknown inside the mesh. When using this approach,you can’t monitor access to external services or take advantage of Istio’s traffic control features for them.To easily switch to the second approach for specific services, simply create service entries for those external services.This process allows you to initially access any external service and then laterdecide whether or not to control access, enable traffic monitoring, and use traffic control features as needed.

The second approach lets you use all of the same Istio service mesh features for calls to services inside oroutside of the cluster. In this task, you learned how to monitor access to external services and set a timeoutrule for calls to an external service.

The third approach bypasses the Istio sidecar proxy, giving your services direct access to any external server.However, configuring the proxy this way does require cluster-provider specific knowledge and configuration.Similar to the first approach, you also lose monitoring of access to external services and you can’t applyIstio features on traffic to external services.

Security note

Note that configuration examples in this task do not enable secure egress traffic control in Istio.A malicious application can bypass the Istio sidecar proxy and access any external service without Istio control.

To implement egress traffic control in a more secure way, you mustdirect egress traffic through an egress gatewayand review the security concerns described in theadditional security considerationssection.

Cleanup

Shutdown the sleep service:

Zip

  1. $ kubectl delete -f @samples/sleep/sleep.yaml@

Set the outbound traffic policy mode to your desired value

  • Check the current value:
  1. $ kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: ALLOW_ANY" | uniq
  2. $ kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: REGISTRY_ONLY" | uniq
  3. mode: ALLOW_ANY

The output will be either mode: ALLOW_ANY or mode: REGISTRY_ONLY.

  • If you want to change the mode, perform the following commands:
  1. $ kubectl get configmap istio -n istio-system -o yaml | sed 's/mode: ALLOW_ANY/mode: REGISTRY_ONLY/g' | kubectl replace -n istio-system -f -
  2. configmap/istio replaced
  1. $ kubectl get configmap istio -n istio-system -o yaml | sed 's/mode: REGISTRY_ONLY/mode: ALLOW_ANY/g' | kubectl replace -n istio-system -f -
  2. configmap/istio replaced

See also

Secure Control of Egress Traffic in Istio, part 3

Comparison of alternative solutions to control egress traffic including performance considerations.

Secure Control of Egress Traffic in Istio, part 2

Use Istio Egress Traffic Control to prevent attacks involving egress traffic.

Secure Control of Egress Traffic in Istio, part 1

Attacks involving egress traffic and requirements for egress traffic control.

Egress Gateway Performance Investigation

Verifies the performance impact of adding an egress gateway.

Consuming External MongoDB Services

Describes a simple scenario based on Istio's Bookinfo example.

Monitoring and Access Policies for HTTP Egress Traffic

Describes how to configure Istio for monitoring and access policies of HTTP egress traffic.