Configure tracing with Telemetry API

Istio provides the ability to configure advanced tracing options, such as sampling rate and adding custom tags to reported spans. This task shows you how to customize the tracing options with Telemetry API.

Before you begin

  1. Ensure that your applications propagate tracing headers as described here.

  2. Follow the tracing installation guide located under Integrations based on your preferred tracing backend to install the appropriate addon and configure your Istio proxies to send traces to the tracing deployment.

Installation

In this example, we will send tracing to zipkin so make sure it is installed:

  1. $ cat <<EOF > ./tracing.yaml
  2. apiVersion: install.istio.io/v1alpha1
  3. kind: IstioOperator
  4. spec:
  5.   meshConfig:
  6.     enableTracing: true
  7.     defaultConfig:
  8.       tracing: {} # disabled MeshConfig tracing options
  9.     extensionProviders:
  10.     # add zipkin provider
  11.     - name: zipkin
  12.       zipkin:
  13.         service: zipkin.istio-system.svc.cluster.local
  14.         port: 9411
  15. EOF
  16. $ istioctl install -f ./tracing.yaml --skip-confirmation

Enable tracing for mesh

Enable tracing by applying the following configuration:

  1. $ kubectl apply -f - <<EOF
  2. apiVersion: telemetry.istio.io/v1
  3. kind: Telemetry
  4. metadata:
  5.   name: mesh-default
  6.   namespace: istio-system
  7. spec:
  8.   tracing:
  9.     - providers:
  10.         - name: "zipkin"
  11. EOF

Customizing Trace sampling

The sampling rate option can be used to control what percentage of requests get reported to your tracing system. This should be configured based upon your traffic in the mesh and the amount of tracing data you want to collect. The default rate is 1%.

  1. $ kubectl apply -f - <<EOF
  2. apiVersion: telemetry.istio.io/v1
  3. kind: Telemetry
  4. metadata:
  5.   name: mesh-default
  6.   namespace: istio-system
  7. spec:
  8.   tracing:
  9.     - providers:
  10.         - name: "zipkin"
  11.       randomSamplingPercentage: 100.00
  12. EOF

Customizing tracing tags

Custom tags can be added to spans based on literals, environmental variables and client request headers in order to provide additional information in spans specific to your environment.

There is no limit on the number of custom tags that you can add, but tag names must be unique.

You can customize the tags using any of the three supported options below.

  1. Literal represents a static value that gets added to each span.

    1. apiVersion: telemetry.istio.io/v1
    2. kind: Telemetry
    3. metadata:
    4. name: mesh-default
    5. namespace: istio-system
    6. spec:
    7. tracing:
    8.     - providers:
    9.         - name: "zipkin"
    10.     randomSamplingPercentage: 100.00
    11.     customTags:
    12.       "provider":
    13.         literal:
    14.           value: "zipkin"

  2. Environmental variables can be used where the value of the custom tag is populated from a workload proxy environment variable.

    1. apiVersion: telemetry.istio.io/v1
    2. kind: Telemetry
    3. metadata:
    4.   name: mesh-default
    5.   namespace: istio-system
    6. spec:
    7.   tracing:
    8.     - providers:
    9.         - name: "zipkin"
    10.       randomSamplingPercentage: 100.00
    11.       customTags:
    12.         "cluster_id":
    13.           environment:
    14.             name: ISTIO_META_CLUSTER_ID
    15.             defaultValue: Kubernetes # optional

    In order to add custom tags based on environmental variables, you must modify the istio-sidecar-injector ConfigMap in your root Istio system namespace.

  3. Client request header option can be used to populate tag value from an incoming client request header.

    1. apiVersion: telemetry.istio.io/v1
    2. kind: Telemetry
    3. metadata:
    4.   name: mesh-default
    5.   namespace: istio-system
    6. spec:
    7.   tracing:
    8.     - providers:
    9.         - name: "zipkin"
    10.       randomSamplingPercentage: 100.00
    11.       customTags:
    12.         my_tag_header:
    13.           header:
    14.             name: <CLIENT-HEADER>
    15.             defaultValue: <VALUE>      # optional

Customizing tracing tag length

By default, the maximum length for the request path included as part of the HttpUrl span tag is 256. To modify this maximum length, add the following to your tracing.yaml file.

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     enableTracing: true
  6.     defaultConfig:
  7.       tracing: {} # disabled tracing options via `MeshConfig`
  8.     extensionProviders:
  9.     # add zipkin provider
  10.     - name: zipkin
  11.       zipkin:
  12.         service: zipkin.istio-system.svc.cluster.local
  13.         port: 9411
  14.         maxTagLength: <VALUE>

Verify the results

You can verify the results with Zipkin UI.