Tempo data source

Grafana ships with built-in support for Tempo a high volume, minimal dependency trace storage, OSS tracing solution from Grafana Labs. Add it as a data source, and you are ready to query your traces in Explore.

Add data source

To access Tempo settings, click the Configuration (gear) icon, then click Data Sources > Tempo.

NameDescription
NameThe name using which you will refer to the data source in panels, queries, and Explore.
DefaultThe default data source will be pre-selected for new panels.
URLThe URL of the Tempo instance, e.g., http://tempo
Basic AuthEnable basic authentication to the Tempo data source.
UserUser name for basic authentication.
PasswordPassword for basic authentication.

Trace to logs

Note: This feature is available in Grafana 7.4+.

This is a configuration for the trace to logs feature. Select target data source (at this moment limited to Loki data sources) and select which tags will be used in the logs query.

  • Data source - Target data source.
  • Tags - The tags that will be used in the Loki query. Default is 'cluster', 'hostname', 'namespace', 'pod'.
  • Map tag names - When enabled, allows configuring how Tempo tag names map to Loki label names. For example, map service.name to service.
  • Span start time shift - A shift in the start time for the Loki query based on the start time for the span. To extend the time to the past, use a negative value. You can use time units, for example, 5s, 1m, 3h. The default is 0.
  • Span end time shift - Shift in the end time for the Loki query based on the span end time. Time units can be used here, for example, 5s, 1m, 3h. The default is 0.
  • Filter by Trace ID - Toggle to append the trace ID to the Loki query.
  • Filter by Span ID - Toggle to append the span ID to the Loki query.

Screenshot of the trace to logs settings Screenshot of the trace to logs settings Tempo - 图2

Service Graph

This is a configuration for the Service Graph feature.

– Data source - Prometheus instance where the Service Graph data is stored.

This is a configuration for Tempo search.

– Hide search - Optionally, hide the search query option in Explore in cases where search is not configured in the Tempo instance.

Node Graph

This is a configuration for the beta Node Graph visualization. The Node Graph is shown after the trace view is loaded and is disabled by default.

– Enable Node Graph - Enables the Node Graph visualization.

Query traces

You can query and display traces from Tempo via Explore. You can search for traces if you set up the trace to logs setting in the data source configuration page. To find traces to visualize, use the Loki query editor. To get search results, you must have derived fields configured, which point to this data source.

Screenshot of the Tempo query editor showing the search tab Screenshot of the Tempo query editor showing the search tab Tempo - 图4 To query a particular trace, select the TraceID query type, and then put the ID into the Trace ID field.

Screenshot of the Tempo TraceID query type Screenshot of the Tempo TraceID query type Tempo - 图6

Upload JSON trace file

You can upload a JSON file that contains a single trace to visualize it. If the file has multiple traces then the first trace is used for visualization.

Here is an example JSON:

  1. {
  2.   "batches": [
  3.     {
  4.       "resource": {
  5.         "attributes": [
  6.           { "key": "service.name", "value": { "stringValue": "db" } },
  7.           { "key": "job", "value": { "stringValue": "tns/db" } },
  8.           { "key": "opencensus.exporterversion", "value": { "stringValue": "Jaeger-Go-2.22.1" } },
  9.           { "key": "host.name", "value": { "stringValue": "63d16772b4a2" } },
  10.           { "key": "ip", "value": { "stringValue": "0.0.0.0" } },
  11.           { "key": "client-uuid", "value": { "stringValue": "39fb01637a579639" } }
  12.         ]
  13.       },
  14.       "instrumentationLibrarySpans": [
  15.         {
  16.           "instrumentationLibrary": {},
  17.           "spans": [
  18.             {
  19.               "traceId": "AAAAAAAAAABguiq7RPE+rg==",
  20.               "spanId": "cmteMBAvwNA=",
  21.               "parentSpanId": "OY8PIaPbma4=",
  22.               "name": "HTTP GET - root",
  23.               "kind": "SPAN_KIND_SERVER",
  24.               "startTimeUnixNano": "1627471657255809000",
  25.               "endTimeUnixNano": "1627471657256268000",
  26.               "attributes": [
  27.                 { "key": "http.status_code", "value": { "intValue": "200" } },
  28.                 { "key": "http.method", "value": { "stringValue": "GET" } },
  29.                 { "key": "http.url", "value": { "stringValue": "/" } },
  30.                 { "key": "component", "value": { "stringValue": "net/http" } }
  31.               ],
  32.               "status": {}
  33.             }
  34.           ]
  35.         }
  36.       ]
  37.     }
  38.   ]
  39. }

Service Graph

A service graph is a visual representation of the relationships between services. Each node on the graph represents a service such as an API or database. With this graph, customers can easily detect performance issues, increases in error, fault, or throttle rates in any of their services, and dive deep into corresponding traces and root causes.

To display the service graph:

  • Configure the Grafana Agent to generate service graph data
  • Link a Prometheus datasource in the Tempo datasource settings.
  • Navigate to Explore
  • Select the Tempo datasource
  • Select the Service Graph query type and run the query
  • Optionally, filter by service name

You can pan and zoom the view with buttons or you mouse. For details about the visualization, refer to Node graph panel.

Each service in the graph is represented as a circle. Numbers on the inside shows average time per request and request per second.

The color of each circle represents the percentage of requests in each of the following states:

  • green = success
  • red = fault
  • yellow = errors
  • purple = throttled responses

Click on the service to see a context menu with additional links for quick navigation to other relevant information.

Linking Trace ID from logs

You can link to Tempo trace from logs in Loki or Elastic by configuring an internal link. See the Derived fields section in the Loki data source or Data links section in the Elastic data source for configuration instructions.

Provision the Tempo data source

You can modify the Grafana configuration files to provision the Tempo data source. Read more about how it works and all the settings you can set for data sources on the provisioning topic.

Here is an example config:

  1. apiVersion: 1
  3. datasources:
  4.   - name: Tempo
  5.     type: tempo
  6.     # Access mode - proxy (server in the UI) or direct (browser in the UI).
  7.     access: proxy
  8.     url: http://localhost:3200
  9.     jsonData:
  10.       httpMethod: GET
  11.       tracesToLogs:
  12.         datasourceUid: 'loki'
  13.         tags: ['job', 'instance', 'pod', 'namespace']
  14.         mappedTags: [{ key: 'service.name', value: 'service' }]
  15.         mapTagNamesEnabled: false
  16.         spanStartTimeShift: '1h'
  17.         spanEndTimeShift: '1h'
  18.         filterByTraceID: false
  19.         filterBySpanID: false
  20.         lokiSearch: true
  21.       serviceMap:
  22.         datasourceUid: 'prometheus'
  23.       search:
  24.         hide: false
  25.       nodeGraph:
  26.         enabled: true