Configurability (Beta/Development)

Istio provides the ability to configure advanced tracing options, such as sampling rate and adding custom tags to reported spans. Sampling is a beta feature, but adding custom tags and tracing tag length are considered in-development for this release.

Before you begin

  1. Ensure that your applications propagate tracing headers as described here.

  2. Follow the tracing installation guide located under Integrations based on your preferred tracing backend to install the appropriate addon and configure your Istio proxies to send traces to the tracing deployment.

Available tracing configurations

You can configure the following tracing options in Istio:

  1. Random sampling rate for percentage of requests that will be selected for trace generation.

  2. Maximum length of the request path after which the path will be truncated for reporting. This can be useful in limiting trace data storage specially if you’re collecting traces at ingress gateways.

  3. Adding custom tags in spans. These tags can be added based on static literal values, environment values or fields from request headers. This can be used to inject additional information in spans specific to your environment.

There are two ways you can configure tracing options:

  1. Globally via MeshConfig options.

  2. Per pod annotations for workload specific customization.

In order for the new tracing configuration to take effect for either of these options you need to restart pods injected with Istio proxies.

Note that any pod annotations added for tracing configuration overrides global settings. In order to preserve any global settings you should copy them from global mesh config to pod annotations along with workload specific customization. In particular, make sure that the tracing backend address is always provided in the annotations to ensure that the traces are reported correctly for the workload.

Using MeshConfig for trace settings

All tracing options can be configured globally via MeshConfig. To simplify configuration, it is recommended to create a single YAML file which you can pass to the istioctl install -f command.

  1. cat <<'EOF' > tracing.yaml
  2. apiVersion: install.istio.io/v1alpha1
  3. kind: IstioOperator
  4. spec:
  5.   meshConfig:
  6.     defaultConfig:
  7.       tracing:
  8.         sampling: 10
  9.         custom_tags:
  10.           my_tag_header:
  11.             header:
  12.               name: host
  13. EOF

Using proxy.istio.io/config annotation for trace settings

You can add the proxy.istio.io/config annotation to your Pod metadata specification to override any mesh-wide tracing settings. For instance, to modify the sleep deployment shipped with Istio you would add the following to samples/sleep/sleep.yaml:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: sleep
  5. spec:
  6.   ...
  7.   template:
  8.     metadata:
  9.       ...
  10.       proxy.istio.io/config: |
  11.         tracing:
  12.           sampling: 10
  13.           custom_tags:
  14.             my_tag_header:
  15.               header:
  16.                 name: host
  17.     spec:
  18.       ...

Customizing Trace sampling

The sampling rate option can be used to control what percentage of requests get reported to your tracing system. This should be configured depending upon your traffic in the mesh and the amount of tracing data you want to collect. The default rate is 1%.

Previously, the recommended method was to change the values.pilot.traceSampling setting during the mesh setup or to change the PILOT_TRACE_SAMPLE environment variable in the pilot or istiod deployment. While this method to alter sampling continues to work, the following method is strongly recommended instead.

In the event that both are specified, the value specified in the MeshConfig will override any other setting.

To modify the default random sampling to 50, add the following option to your tracing.yaml file.

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       tracing:
  7.         sampling: 50

The sampling rate should be in the range of 0.0 to 100.0 with a precision of 0.01. For example, to trace 5 requests out of every 10000, use 0.05 as the value here.

Customizing tracing tags

Custom tags can be added to spans based on literals, environmental variables and client request headers in order to provide additional information in spans specific to your environment.

There is no limit on the number of custom tags that you can add, but tag names must be unique.

You can customize the tags using any of the three supported options below.

  1. Literal represents a static value that gets added to each span.

    1. apiVersion: install.istio.io/v1alpha1
    2. kind: IstioOperator
    3. spec:
    4.   meshConfig:
    5.     defaultConfig:
    6.       tracing:
    7.         custom_tags:
    8.           my_tag_literal:
    9.             literal:
    10.               value: <VALUE>

  2. Environmental variables can be used where the value of the custom tag is populated from a workload proxy environment variable.

    1. apiVersion: install.istio.io/v1alpha1
    2. kind: IstioOperator
    3. spec:
    4.   meshConfig:
    5.     defaultConfig:
    6.       tracing:
    7.         custom_tags:
    8.           my_tag_env:
    9.             environment:
    10.               name: <ENV_VARIABLE_NAME>
    11.               defaultValue: <VALUE>      # optional

    In order to add custom tags based on environmental variables, you must modify the istio-sidecar-injector ConfigMap in your root Istio system namespace.

  3. Client request header option can be used to populate tag value from an incoming client request header.

    1. apiVersion: install.istio.io/v1alpha1
    2. kind: IstioOperator
    3. spec:
    4.   meshConfig:
    5.     defaultConfig:
    6.       tracing:
    7.         custom_tags:
    8.           my_tag_header:
    9.             header:
    10.               name: <CLIENT-HEADER>
    11.               defaultValue: <VALUE>      # optional

Customizing tracing tag length

By default, the maximum length for the request path included as part of the HttpUrl span tag is 256. To modify this maximum length, add the following to your tracing.yaml file.

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       tracing:
  7.         max_path_tag_length: <VALUE>

See also

Jaeger

Learn how to configure the proxies to send tracing requests to Jaeger.

Lightstep

How to configure the proxies to send tracing requests to Lightstep.

Overview

Overview of distributed tracing in Istio.

Remotely Accessing Telemetry Addons

This task shows you how to configure external access to the set of Istio telemetry addons.

Zipkin

Learn how to configure the proxies to send tracing requests to Zipkin.

Reworking our Addon Integrations

A new way to manage installation of telemetry addons.