Skip to main content

07 – Enriching Tempo Traces in the Grafana Stack

Learning Objectives​

  • Link traces with logs (Loki) and metrics (Prometheus/Mimir).
  • Query traces using TraceQL and build service graphs.
  • Enable spanmetrics and exemplars.

Linking Traces with Logs (Loki)​

  • Ensure your apps log a trace_id field or Loki pipeline extracts it.
  • In Grafana data source settings, map TraceID to the log label that carries it.
  • From a span in Tempo, jump to related logs.

TraceQL Basics​

Examples:

from traces
| where span.service.name = "orders" and status.code = "Error"

from traces
| where duration > 500ms and attributes["http.method"] = "GET"

Service Graphs​

  • Grafana builds graphs from span relationships to reveal dependencies and hot paths.

SpanMetrics Processor (Collector)​

processors:
  spanmetrics:
    metrics_exporter: prometheus

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [spanmetrics]
      exporters: [prometheus]

Hands-on Lab​

  1. Add Loki to your stack (optional from Section 5). Ensure logs carry trace_id.
  2. Enable spanmetrics and scrape its metrics in Prometheus.
  3. Build a dashboard with RED metrics and enable exemplars to link back to traces.

Deliverables​

  • Demonstrate a trace → log jump and a metrics chart with exemplars.

Quiz (Self-check)​

  • Write a TraceQL query to find slow requests to /checkout.
  • What does spanmetrics produce and why is it useful?

Resources​

  • TraceQL Reference
  • Grafana Loki LogQL Reference

Visual: Correlating Traces, Logs, and Metrics​