Overview

Distributed tracing enables users to track a request through mesh that is distributed across multiple services. This allows a deeper understanding about request latency, serialization and parallelism via visualization.

Istio leverages Envoy’s distributed tracing feature to provide tracing integration out of the box. Specifically, Istio provides options to install various tracing backends and configure proxies to send trace spans to them automatically. See Zipkin, Jaeger, and Lightstep task docs about how Istio works with those tracing systems.

Trace context propagation

Although Istio proxies can automatically send spans, extra information is needed to join those spans into a single trace. Applications must propagate this information in HTTP headers, so that when proxies send spans, the backend can join them together into a single trace.

To do this, each application must collect headers from each incoming request and forward the headers to all outgoing requests triggered by that incoming request. The choice of headers to forward depends on the configured trace backend. The set of headers to forward are described in each trace backend-specific task page. The following is a summary:

All applications should forward the following header:

  • x-request-id: this is an envoy-specific header that is used to consistently sample logs and traces.

For Zipkin, Jaeger, and Stackdriver the B3 multi-header format should be forwarded:

  • x-b3-traceid
  • x-b3-spanid
  • x-b3-parentspanid
  • x-b3-sampled
  • x-b3-flags

These are supported by Zipkin, Jaeger, and many other tools.

For Datadog, the following headers should be forwarded. Forwarding these is handled automatically by Datadog client libraries for many languages and frameworks.

  • x-datadog-trace-id.
  • x-datadog-parent-id.
  • x-datadog-sampling-priority.

For Lightstep, the OpenTracing span context header should be forwarded:

  • x-ot-span-context

For Stackdriver, you can choose to use any one of the following headers instead of the B3 multi-header format.

  • grpc-trace-bin: Standard grpc trace header.
  • traceparent: W3C Trace Context standard for tracing. Supported by OpenTelemetry and an increasing number of Jaeger client libraries.
  • x-cloud-trace-context: used by Google Cloud product APIs.

If you look at the sample Python productpage service, for example, you see that the application extracts the required headers for all tracers from an HTTP request using OpenTracing libraries:

  1. def getForwardHeaders(request):
  2.     headers = {}
  4.     # x-b3-*** headers can be populated using the opentracing span
  5.     span = get_current_span()
  6.     carrier = {}
  7.     tracer.inject(
  8.         span_context=span.context,
  9.         format=Format.HTTP_HEADERS,
  10.         carrier=carrier)
  12.     headers.update(carrier)
  14.     # ...
  16.         incoming_headers = ['x-request-id',
  17.         'x-ot-span-context',
  18.         'x-datadog-trace-id',
  19.         'x-datadog-parent-id',
  20.         'x-datadog-sampling-priority',
  21.         'traceparent',
  22.         'tracestate',
  23.         'x-cloud-trace-context',
  24.         'grpc-trace-bin',
  25.         'user-agent',
  26.         'cookie',
  27.         'authorization',
  28.         'jwt',
  29.     ]
  31.     # ...
  33.     for ihdr in incoming_headers:
  34.         val = request.headers.get(ihdr)
  35.         if val is not None:
  36.             headers[ihdr] = val
  38.     return headers

The reviews application (Java) does something similar using requestHeaders:

  1. @GET
  2. @Path("/reviews/{productId}")
  3. public Response bookReviewsById(@PathParam("productId") int productId, @Context HttpHeaders requestHeaders) {
  5.   // ...
  7.   if (ratings_enabled) {
  8.     JsonObject ratingsResponse = getRatings(Integer.toString(productId), requestHeaders);

When you make downstream calls in your applications, make sure to include these headers.