Jaeger data source

Grafana ships with built-in support for Jaeger, which provides open source, end-to-end distributed tracing. Just add it as a data source and you are ready to query your traces in Explore.

Add data source

To access Jaeger settings, click the Configuration (gear) icon, then click Data Sources > Jaeger.

NameDescription
NameThe data source name in panels, queries, and Explore.
DefaultThe pre-selected data source for a new panel.
URLThe URL of the Jaeger instance. For example, http://localhost:16686.
Basic AuthEnable basic authentication for the Jaeger data source.
UserSpecify a user name for basic authentication.
PasswordSpecify a password for basic authentication.

Trace to logs

Note: This feature is available in Grafana 7.4+.

This is a configuration for the trace to logs feature. Select target data source (at this moment limited to Loki and Splunk [logs] data sources) and select which tags will be used in the logs query.

  • Data source - Target data source.
  • Tags - The tags that will be used in the logs query. Default is 'cluster', 'hostname', 'namespace', 'pod'.
  • Map tag names - When enabled, allows configuring how Jaeger tag names map to logs label names. For example, map service.name to service.
  • Span start time shift - Shift in the start time for the logs query based on the span start time. In order to extend to the past, you need to use a negative value. Use time interval units like 5s, 1m, 3h. The default is 0.
  • Span end time shift - Shift in the end time for the logs query based on the span end time. Time units can be used here, for example, 5s, 1m, 3h. The default is 0.
  • Filter by Trace ID - Toggle to append the trace ID to the logs query.
  • Filter by Span ID - Toggle to append the span ID to the logs query.

Trace to metrics

Note: This feature is behind the traceToMetrics feature toggle.

To configure trace to metrics, select the target Prometheus data source and create any desired linked queries.

Data source - Target data source. – Tags - You can use tags in the linked queries. The key is the span attribute name. The optional value is the corresponding metric label name (for example, map k8s.pod to pod). You may interpolate these tags into your queries using the $__tags keyword.

Each linked query consists of:

Link Label - (Optional) Descriptive label for the linked query. – Query - Query that runs when navigating from a trace to the metrics data source. Interpolate tags using the $__tags keyword. For example, when you configure the query requests_total{$__tags}with the tags k8s.pod=pod and cluster, it results in requests_total{pod="nginx-554b9", cluster="us-east-1"}.

Node Graph

This is a configuration for the beta Node Graph visualization. The Node Graph is shown after the trace view is loaded and is disabled by default.

Enable Node Graph - Enables the Node Graph visualization.

Span bar label

You can configure the span bar label. The span bar label allows you add additional information to the span bar row.

Select one of the following four options. The default selection is Duration.

  • None - Do not show any additional information on the span bar row.
  • Duration - Show the span duration on the span bar row.
  • Tag - Show the span tag on the span bar row. Note: You will also need to specify the tag key to use to get the tag value. For example, span.kind.

Query traces

You can query and display traces from Jaeger via Explore.

Screenshot of the Jaeger query editor Screenshot of the Jaeger query editor Jaeger - 图2

You can query by trace ID or use the search form to find traces. To query by trace ID, select the TraceID from the Query type selector and insert the ID into the text input.

Screenshot of the Jaeger query editor with trace ID selected Screenshot of the Jaeger query editor with trace ID selected Jaeger - 图4

To perform a search, set the query type selector to Search, then use the following fields to find traces:

  • Service - Returns a list of services.
  • Operation - Field gets populated once you select a service. It then lists the operations related to the selected service. Select All option to query all operations.
  • Tags - Use values in the logfmt format. For example error=true db.statement="select * from User".
  • Min Duration - Filter all traces with a duration higher than the set value. Possible values are 1.2s, 100ms, 500us.
  • Max Duration - Filter all traces with a duration lower than the set value. Possible values are 1.2s, 100ms, 500us.
  • Limit - Limits the number of traces returned.

Upload JSON trace file

You can upload a JSON file that contains a single trace to visualize it. If the file has multiple traces then the first trace is used for visualization.

Screenshot of the Jaeger data source in explore with upload selected Screenshot of the Jaeger data source in explore with upload selected Jaeger - 图6

Here is an example JSON:

  1. {
  2. "data": [
  3. {
  4. "traceID": "2ee9739529395e31",
  5. "spans": [
  6. {
  7. "traceID": "2ee9739529395e31",
  8. "spanID": "2ee9739529395e31",
  9. "flags": 1,
  10. "operationName": "CAS",
  11. "references": [],
  12. "startTime": 1616095319593196,
  13. "duration": 1004,
  14. "tags": [
  15. {
  16. "key": "sampler.type",
  17. "type": "string",
  18. "value": "const"
  19. }
  20. ],
  21. "logs": [],
  22. "processID": "p1",
  23. "warnings": null
  24. }
  25. ],
  26. "processes": {
  27. "p1": {
  28. "serviceName": "loki-all",
  29. "tags": [
  30. {
  31. "key": "jaeger.version",
  32. "type": "string",
  33. "value": "Go-2.25.0"
  34. }
  35. ]
  36. }
  37. },
  38. "warnings": null
  39. }
  40. ],
  41. "total": 0,
  42. "limit": 0,
  43. "offset": 0,
  44. "errors": null
  45. }

Linking Trace ID from logs

You can link to Jaeger trace from logs in Loki by configuring a derived field with internal link. See the Derived fields section in the Loki data source documentation for details.

Configure the data source with provisioning

You can set up the data source via configuration files with Grafana’s provisioning system. Refer to provisioning docs page for more information on configuring various settings.

Here is an example with basic auth and trace-to-logs field.

  1. apiVersion: 1
  2. datasources:
  3. - name: Jaeger
  4. type: jaeger
  5. uid: jaeger-spectra
  6. access: proxy
  7. url: http://localhost:16686/
  8. basicAuth: true
  9. basicAuthUser: my_user
  10. editable: true
  11. isDefault: false
  12. jsonData:
  13. tracesToLogs:
  14. # Field with internal link pointing to a logs data source in Grafana.
  15. # datasourceUid value must match the `datasourceUid` value of the logs data source.
  16. datasourceUid: 'loki'
  17. tags: ['job', 'instance', 'pod', 'namespace']
  18. mappedTags: [{ key: 'service.name', value: 'service' }]
  19. mapTagNamesEnabled: false
  20. spanStartTimeShift: '1h'
  21. spanEndTimeShift: '1h'
  22. filterByTraceID: false
  23. filterBySpanID: false
  24. tracesToMetrics:
  25. datasourceUid: 'prom'
  26. tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }]
  27. queries:
  28. - name: 'Sample query'
  29. query: 'sum(rate(tempo_spanmetrics_latency_bucket{$__tags}[5m]))'
  30. secureJsonData:
  31. basicAuthPassword: my_password