How-To: Use W3C trace context with Dapr

Using W3C tracing standard with Dapr

How to use trace context

Dapr uses W3C trace context for distributed tracing for both service invocation and pub/sub messaging. Dapr does all the heavy lifting of generating and propagating the trace context information and there are very few cases where you need to either propagate or create a trace context. First read scenarios in the W3C distributed tracing article to understand whether you need to propagate or create a trace context.

To view traces, read the how to diagnose with tracing article.

How to retrieve trace context from a response

Note: There are no helper methods exposed in Dapr SDKs to propagate and retrieve trace context. You need to use http/gRPC clients to propagate and retrieve trace headers through http headers and gRPC metadata.

Retrieve trace context in Go

For HTTP calls

OpenCensus Go SDK provides ochttp package that provides methods to retrieve trace context from http response.

To retrieve the trace context from HTTP response, you can use :

  1. f := tracecontext.HTTPFormat{}
  2. sc, ok := f.SpanContextFromRequest(req)

For gRPC calls

To retrieve the trace context header when the gRPC call is returned, you can pass the response header reference as gRPC call option which contains response headers:

  1. var responseHeader metadata.MD
  3. // Call the InvokeService with call option
  4. // grpc.Header(&responseHeader)
  6. client.InvokeService(ctx, &pb.InvokeServiceRequest{
  7.         Id: "client",
  8.         Message: &commonv1pb.InvokeRequest{
  9.             Method:      "MyMethod",
  10.             ContentType: "text/plain; charset=UTF-8",
  11.             Data:        &any.Any{Value: []byte("Hello")},
  12.         },
  13.     },
  14.     grpc.Header(&responseHeader))

Retrieve trace context in C

For HTTP calls

To retrieve the trace context from HTTP response, you can use .NET API :

  1. // client is HttpClient. req is HttpRequestMessage
  2. HttpResponseMessage response = await client.SendAsync(req);
  3. IEnumerable<string> values1, values2;
  4. string traceparentValue = "";
  5. string tracestateValue = "";
  6. if (response.Headers.TryGetValues("traceparent", out values1))
  7. {
  8.     traceparentValue = values1.FirstOrDefault();
  9. }
  10. if (response.Headers.TryGetValues("tracestate", out values2))
  11. {
  12.     tracestateValue = values2.FirstOrDefault();
  13. }

For gRPC calls

To retrieve the trace context from gRPC response, you can use Grpc.Net.Client ResponseHeadersAsync method.

  1. // client is Dapr proto client
  2. using var call = client.InvokeServiceAsync(req);
  3. var response = await call.ResponseAsync;
  4. var headers = await call.ResponseHeadersAsync();
  5. var tracecontext = headers.First(e => e.Key == "grpc-trace-bin");

Additional general details on calling gRPC services with .NET client here.

How to propagate trace context in a request

Note: There are no helper methods exposed in Dapr SDKs to propagate and retrieve trace context. You need to use http/gRPC clients to propagate and retrieve trace headers through http headers and gRPC metadata.

Pass trace context in Go

For HTTP calls

OpenCensus Go SDK provides ochttp package that provides methods to attach trace context in http request.

  1. f := tracecontext.HTTPFormat{}
  2. req, _ := http.NewRequest("GET", "http://localhost:3500/v1.0/invoke/mathService/method/api/v1/add", nil)
  4. traceContext := span.SpanContext()
  5. f.SpanContextToRequest(traceContext, req)

For gRPC calls

  1. traceContext := span.SpanContext()
  2. traceContextBinary := propagation.Binary(traceContext)

You can then pass the trace context through gRPC metadata through grpc-trace-bin header.

  1. ctx = metadata.AppendToOutgoingContext(ctx, "grpc-trace-bin", string(traceContextBinary))

You can then continuing passing this go context ctx in subsequent Dapr gRPC calls as first parameter. For example InvokeService, context is passed in first parameter.

Pass trace context in C

For HTTP calls

To pass trace context in HTTP request, you can use .NET API :

  1. // client is HttpClient. req is HttpRequestMessage
  2. req.Headers.Add("traceparent", traceparentValue);
  3. req.Headers.Add("tracestate", tracestateValue);
  4. HttpResponseMessage response = await client.SendAsync(req);

For gRPC calls

To pass the trace context in gRPC call metadata, you can use Grpc.Net.Client ResponseHeadersAsync method.

  1. // client is Dapr.Client.Autogen.Grpc.v1
  2. var headers = new Metadata();
  3. headers.Add("grpc-trace-bin", tracecontext);
  4. using var call = client.InvokeServiceAsync(req, headers);

Additional general details on calling gRPC services with .NET client here.

How to create trace context

You can create a trace context using the recommended OpenCensus SDKs. OpenCensus supports several different programming languages.

LanguageSDK
GoLink
JavaLink
C#Link
C++Link
Node.jsLink
PythonLink

Create trace context in Go

1. Get the OpenCensus Go SDK

Prerequisites: OpenCensus Go libraries require Go 1.8 or later. For details on installation go here.

2. Import the package “go.opencensus.io/trace”

$ go get -u go.opencensus.io

3. Create trace context

  1. ctx, span := trace.StartSpan(ctx, "cache.Get")
  2. defer span.End()
  4. // Do work to get from cache.

Create trace context in Java

  1. try (Scope ss = TRACER.spanBuilder("cache.Get").startScopedSpan()) {
  2. }

Create trace context in Python

  1. with tracer.span(name="cache.get") as span:
  2.     pass

Create trace context in NodeJS

  1. tracer.startRootSpan({name: 'cache.Get'}, rootSpan => {
  2. });

Create trace context in C++

  1. opencensus::trace::Span span = opencensus::trace::Span::StartSpan(
  2.                                             "cache.Get", nullptr, {&sampler});

Create trace context in C

  1. var span = tracer.SpanBuilder("cache.Get").StartScopedSpan();

Putting it all together with a Go Sample

Configure tracing in Dapr

First you need to enable tracing configuration in Dapr. This step is mentioned for completeness from enabling tracing to invoking Dapr with trace context. Create a deployment config yaml e.g. appconfig.yaml with following configuration.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: appconfig
  5. spec:
  6.   tracing:
  7.     samplingRate: "1"

In Kubernetes, you can apply the configuration as below :

  1. kubectl apply -f appconfig.yaml

You then set the following tracing annotation in your deployment YAML. You can add the following annotaion in sample grpc app deployment yaml.

  1. dapr.io/config: "appconfig"

Invoking Dapr with trace context

Dapr covers generating trace context and you do not need to explicitly create trace context.

However if you choose to pass the trace context explicitly, then Dapr will use the passed trace context and propagate all across the HTTP/gRPC call.

Using the grpc app in the example and putting this all together, the following steps show you how to create a Dapr client and call the InvokeService method passing the trace context:

The Rest code snippet and details, refer to the grpc app.

1. Import the package

  1. package main
  3. import (
  4.     pb "github.com/dapr/go-sdk/dapr"
  5.     "go.opencensus.io/trace"
  6.       "go.opencensus.io/trace/propagation"
  7.       "google.golang.org/grpc"
  8.       "google.golang.org/grpc/metadata"
  9. )

2. Create the client

  1.   // Get the Dapr port and create a connection
  2.   daprPort := os.Getenv("DAPR_GRPC_PORT")
  3.   daprAddress := fmt.Sprintf("localhost:%s", daprPort)
  4.   conn, err := grpc.Dial(daprAddress, grpc.WithInsecure())
  5.   if err != nil {
  6.     fmt.Println(err)
  7.   }
  8.   defer conn.Close()
  10.   // Create the client
  11.   client := pb.NewDaprClient(conn)

3. Invoke the InvokeService method With Trace Context

  1.   // Create the Trace Context
  2.   ctx , span := trace.StartSpan(context.Background(), "InvokeService")
  4.   // The returned context can be used to keep propagating the newly created span in the current context.
  5.   // In the same process, context.Context is used to propagate trace context.
  7.   // Across the process, use the propagation format of Trace Context to propagate trace context.
  8.   traceContext := propagation.Binary(span.SpanContext())
  9.   ctx = metadata.NewOutgoingContext(ctx, string(traceContext))
  11.   // Pass the trace context
  12.   resp, err := client.InvokeService(ctx, &pb.InvokeServiceRequest{
  13.         Id: "client",
  14.         Message: &commonv1pb.InvokeRequest{
  15.             Method:      "MyMethod",
  16.             ContentType: "text/plain; charset=UTF-8",
  17.             Data:        &any.Any{Value: []byte("Hello")},
  18.         },
  19.     })

You can now correlate the calls in your app and across services with Dapr using the same trace context.

Last modified September 17, 2021 : Merge pull request #1757 from georgestevens99/1440SecretKeyRefExplanation (620a5f8)