Avoiding Traffic Management Issues

This section provides specific deployment or configuration guidelines to avoid networking or traffic management issues.

Cross-namespace configuration sharing

You can define virtual services, destination rules, or service entriesin one namespace and then reuse them in other namespaces, if they are exportedto those namespaces.Istio exports all traffic management resources to all namespaces by default,but you can override the visibility with the exportTo field.For example, only clients in the same namespace can use the following virtual service:

  1. apiVersion: networking.istio.io/v1alpha3
  2. kind: VirtualService
  3. metadata:
  4.   name: myservice
  5. spec:
  6.   hosts:
  7.   - myservice.com
  8.   exportTo:
  9.   - "."
  10.   http:
  11.   - route:
  12.     - destination:
  13.         host: myservice

You can similarly control the visibility of a Kubernetes Service using the networking.istio.io/exportTo annotation.

Setting the visibility of destination rules in a particular namespace doesn’tguarantee the rule is used. Exporting a destination rule to other namespaces enables you to use itin those namespaces, but to actually be applied during a request the namespace also needs to beon the destination rule lookup path:

  • client namespace
  • service namespace
  • Istio configuration root (istio-system by default)For example, consider the following destination rule:
  1. apiVersion: networking.istio.io/v1alpha3
  2. kind: DestinationRule
  3. metadata:
  4.   name: myservice
  5. spec:
  6.   host: myservice.default.svc.cluster.local
  7.   trafficPolicy:
  8.     connectionPool:
  9.       tcp:
  10.         maxConnections: 100

Let’s assume you create this destination rule in namespace ns1.

If you send a request to the myservice service from a client in ns1, the destinationrule would be applied, because it is in the first namespace on the lookup path, that is,in the client namespace.

If you now send the request from a different namespace, for example ns2,the client is no longer in the same namespace as the destination rule, ns1.Because the corresponding service, myservice.default.svc.cluster.local, is also not in ns1,but rather in the default namespace, the destination rule will also not be found inthe second namespace of the lookup path, the service namespace.

Even if the myservice service is exported to all namespaces and therefore visiblein ns2 and the destination rule is also exported to all namespaces, including ns2,it will not be applied during the request from ns2 because it’s not in anyof the namespaces on the lookup path.

You can avoid this problem by creating the destination rule in the same namespace asthe corresponding service, default in this example. It would then get applied to requestsfrom clients in any namespace.You can also move the destination rule to the istio-system namespace, the third namespace onthe lookup path, although this isn’t recommended unless the destination rule is really a globalconfiguration that is applicable in all namespaces, and it would require administrator authority.

Istio uses this restricted destination rule lookup path for two reasons:

  • Prevent destination rules from being defined that can override the behavior of servicesin completely unrelated namespaces.
  • Have a clear lookup order in case there is more than one destination rule forthe same host.

Configuring multiple TLS hosts in a gateway

If you apply a Gateway configuration that has the same selector labels as anotherexisting Gateway, then if they both expose the same HTTPS port you must ensure that they haveunique port names. Otherwise, the configuration will be applied without an immediate error indicationbut it will be ignored in the runtime gateway configuration. For example:

  1. apiVersion: networking.istio.io/v1alpha3
  2. kind: Gateway
  3. metadata:
  4.   name: mygateway
  5. spec:
  6.   selector:
  7.     istio: ingressgateway # use istio default ingress gateway
  8.   servers:
  9.   - port:
  10.       number: 443
  11.       name: https
  12.       protocol: HTTPS
  13.     tls:
  14.       mode: SIMPLE
  15.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
  16.       privateKey: /etc/istio/ingressgateway-certs/tls.key
  17.     hosts:
  18.     - "myhost.com"
  19. ---
  20. apiVersion: networking.istio.io/v1alpha3
  21. kind: Gateway
  22. metadata:
  23.   name: mygateway2
  24. spec:
  25.   selector:
  26.     istio: ingressgateway # use istio default ingress gateway
  27.   servers:
  28.   - port:
  29.       number: 443
  30.       name: https
  31.       protocol: HTTPS
  32.     tls:
  33.       mode: SIMPLE
  34.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
  35.       privateKey: /etc/istio/ingressgateway-certs/tls.key
  36.     hosts:
  37.     - "myhost2.com"

With this configuration, requests to the second host, myhost2.com, will fail becauseboth gateway ports have name: https.A curl request, for example, will produce an error message something like this:

  1. curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhost2.com:443

You can confirm that this has happened by checking Pilot’s logs for a message similar to the following:

  1. $ kubectl logs -n istio-system $(kubectl get pod -l istio=pilot -n istio-system -o jsonpath={.items..metadata.name}) -c discovery | grep "non unique port"
  2. 2018-09-14T19:02:31.916960Z info    model   skipping server on gateway mygateway2 port https.443.HTTPS: non unique port name for HTTPS port

To avoid this problem, ensure that multiple uses of the same protocol: HTTPS port are uniquely named.For example, change the second one to https2:

  1. apiVersion: networking.istio.io/v1alpha3
  2. kind: Gateway
  3. metadata:
  4.   name: mygateway2
  5. spec:
  6.   selector:
  7.     istio: ingressgateway # use istio default ingress gateway
  8.   servers:
  9.   - port:
  10.       number: 443
  11.       name: https2
  12.       protocol: HTTPS
  13.     tls:
  14.       mode: SIMPLE
  15.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
  16.       privateKey: /etc/istio/ingressgateway-certs/tls.key
  17.     hosts:
  18.     - "myhost2.com"

Multiple virtual services and destination rules for the same host

In situations where it is inconvenient to define the complete set of route rules or policies for a particularhost in a single VirtualService or DestinationRule resource, it may be preferable to incrementally specifythe configuration for the host in multiple resources.Starting with Istio 1.0.1, Pilot will merge such destination rulesand merge such virtual services if they are bound to a gateway.

Consider the case of a VirtualService bound to an ingress gateway exposing an application host which usespath-based delegation to several implementation services, something like this:

  1. apiVersion: networking.istio.io/v1alpha3
  2. kind: VirtualService
  3. metadata:
  4.   name: myapp
  5. spec:
  6.   hosts:
  7.   - myapp.com
  8.   gateways:
  9.   - myapp-gateway
  10.   http:
  11.   - match:
  12.     - uri:
  13.         prefix: /service1
  14.     route:
  15.     - destination:
  16.         host: service1.default.svc.cluster.local
  17.   - match:
  18.     - uri:
  19.         prefix: /service2
  20.     route:
  21.     - destination:
  22.         host: service2.default.svc.cluster.local
  23.   - match:
  24.     ...

The downside of this kind of configuration is that other configuration (e.g., route rules) for any of theunderlying microservices, will need to also be included in this single configuration file, instead ofin separate resources associated with, and potentially owned by, the individual service teams.See Route rules have no effect on ingress gateway requestsfor details.

To avoid this problem, it may be preferable to break up the configuration of myapp.com into severalVirtualService fragments, one per backend service. For example:

  1. apiVersion: networking.istio.io/v1alpha3
  2. kind: VirtualService
  3. metadata:
  4.   name: myapp-service1
  5. spec:
  6.   hosts:
  7.   - myapp.com
  8.   gateways:
  9.   - myapp-gateway
  10.   http:
  11.   - match:
  12.     - uri:
  13.         prefix: /service1
  14.     route:
  15.     - destination:
  16.         host: service1.default.svc.cluster.local
  17. ---
  18. apiVersion: networking.istio.io/v1alpha3
  19. kind: VirtualService
  20. metadata:
  21.   name: myapp-service2
  22. spec:
  23.   hosts:
  24.   - myapp.com
  25.   gateways:
  26.   - myapp-gateway
  27.   http:
  28.   - match:
  29.     - uri:
  30.         prefix: /service2
  31.     route:
  32.     - destination:
  33.         host: service2.default.svc.cluster.local
  34. ---
  35. apiVersion: networking.istio.io/v1alpha3
  36. kind: VirtualService
  37. metadata:
  38.   name: myapp-...

When a second and subsequent VirtualService for an existing host is applied, istio-pilot will mergethe additional route rules into the existing configuration of the host. There are, however, severalcaveats with this feature that must be considered carefully when using it.

  • Although the order of evaluation for rules in any given source VirtualService will be retained,the cross-resource order is UNDEFINED. In other words, there is no guaranteed order of evaluationfor rules across the fragment configurations, so it will only have predictable behavior if thereare no conflicting rules or order dependency between rules across fragments.
  • There should only be one “catch-all” rule (i.e., a rule that matches any request path or header) in the fragments.All such “catch-all” rules will be moved to the end of the list in the merged configuration, butsince they catch all requests, whichever is applied first will essentially override and disable any others.

  • A VirtualService can only be fragmented this way if it is bound to a gateway.Host merging is not supported in sidecars.A DestinationRule can also be fragmented with similar merge semantic and restrictions.

  • There should only be one definition of any given subset across multiple destination rules for the same host.If there is more than one with the same name, the first definition is used and any following duplicates are discarded.No merging of subset content is supported.

  • There should only be one top-level trafficPolicy for the same host.When top-level traffic policies are defined in multiple destination rules, the first one will be used.Any following top-level trafficPolicy configuration is discarded.
  • Unlike virtual service merging, destination rule merging works in both sidecars and gateways.

Avoid 503 errors while reconfiguring service routes

When setting route rules to direct traffic to specific versions (subsets) of a service, care must be taken to ensurethat the subsets are available before they are used in the routes. Otherwise, calls to the service may return503 errors during a reconfiguration period.

Creating both the VirtualServices and DestinationRules that define the corresponding subsets using a single kubectlcall (e.g., kubectl apply -f myVirtualServiceAndDestinationRule.yaml is not sufficient because theresources propagate (from the configuration server, i.e., Kubernetes API server) to the Pilot instances in an eventually consistent manner. If theVirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.

To make sure services will have zero down-time when configuring routes with subsets, follow a “make-before-break” process as described below:

  • When adding new subsets:

    • Update DestinationRules to add a new subset first, before updating any VirtualServices that use it. Apply the rule using kubectl or any platform-specific tooling.

    • Wait a few seconds for the DestinationRule configuration to propagate to the Envoy sidecars

    • Update the VirtualService to refer to the newly added subsets.

  • When removing subsets:

    • Update VirtualServices to remove any references to a subset, before removing the subset from a DestinationRule.

    • Wait a few seconds for the VirtualService configuration to propagate to the Envoy sidecars.

    • Update the DestinationRule to remove the unused subsets.