Using OpenTelemetry Collector to collect traces to send to Jaeger

How to push trace events to Jaeger distributed tracing platform, using the OpenTelemetry Collector.

While Dapr supports writing traces using OpenTelemetry (OTLP) and Zipkin protocols, Zipkin support for Jaeger has been deprecated in favor of OTLP. Although Jaeger supports OTLP directly, the recommended approach for production is to use the OpenTelemetry Collector to collect traces from Dapr and send them to Jaeger, allowing your application to quickly offload data and take advantage of features like retries, batching, and encryption. For more information, read the Open Telemetry Collector documentation.

Configure Jaeger in self-hosted mode

Local setup

The simplest way to start Jaeger is to run the pre-built, all-in-one Jaeger image published to DockerHub and expose the OTLP port:

  1. docker run -d --name jaeger \
  2.   -p 4317:4317  \
  3.   -p 16686:16686 \
  4.   jaegertracing/all-in-one:1.49

Next, create the following config.yaml file locally:

Note: Because you are using the Open Telemetry protocol to talk to Jaeger, you need to fill out the otel section of the tracing configuration and set the endpointAddress to the address of the Jaeger container.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: tracing
  5.   namespace: default
  6. spec:
  7.   tracing:
  8.     samplingRate: "1"
  9.     stdout: true
  10.     otel:
  11.       endpointAddress: "localhost:4317"
  12.       isSecure: false
  13.       protocol: grpc

To launch the application referring to the new YAML configuration file, use the --config option. For example:

  1. dapr run --app-id myapp --app-port 3000 node app.js --config config.yaml

View traces

To view traces in your browser, go to http://localhost:16686 to see the Jaeger UI.

Configure Jaeger on Kubernetes with the OpenTelemetry Collector

The following steps show you how to configure Dapr to send distributed tracing data to the OpenTelemetry Collector which, in turn, sends the traces to Jaeger.

Prerequisites

Set up OpenTelemetry Collector to push to Jaeger

To push traces to your Jaeger instance, install the OpenTelemetry Collector on your Kubernetes cluster.

  1. Download and inspect the open-telemetry-collector-jaeger.yaml file.

  2. In the data section of the otel-collector-conf ConfigMap, update the otlp/jaeger.endpoint value to reflect the endpoint of your Jaeger collector Kubernetes service object.

  3. Deploy the OpenTelemetry Collector into the same namespace where your Dapr-enabled applications are running:

    1. kubectl apply -f open-telemetry-collector-jaeger.yaml

Set up Dapr to send traces to OpenTelemetryCollector

Create a Dapr configuration file to enable tracing and export the sidecar traces to the OpenTelemetry Collector.

  1. Use the collector-config-otel.yaml file to create your own Dapr configuration.

  2. Update the namespace and otel.endpointAddress values to align with the namespace where your Dapr-enabled applications and OpenTelemetry Collector are deployed.

  3. Apply the configuration with:

    1. kubectl apply -f collector-config.yaml

Deploy your app with tracing enabled

Apply the tracing Dapr configuration by adding a dapr.io/config annotation to the application deployment that you want to enable distributed tracing for, as shown in the following example:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   ...
  5. spec:
  6.   ...
  7.   template:
  8.     metadata:
  9.       ...
  10.       annotations:
  11.         dapr.io/enabled: "true"
  12.         dapr.io/app-id: "MyApp"
  13.         dapr.io/app-port: "8080"
  14.         dapr.io/config: "tracing"

You can register multiple tracing exporters at the same time, and the tracing logs are forwarded to all registered exporters.

That’s it! There’s no need to include the OpenTelemetry SDK or instrument your application code. Dapr automatically handles the distributed tracing for you.

View traces

To view Dapr sidecar traces, port-forward the Jaeger Service and open the UI:

  1. kubectl port-forward svc/jaeger-query 16686 -n observability

In your browser, go to http://localhost:16686 and you will see the Jaeger UI.

jaeger

References

Last modified October 12, 2023: Update config.toml (#3826) (0ffc2e7)