Skip to main content

08 – Advanced Tempo Concepts & Scaling for Production

Learning Objectives​

  • Choose deployment mode: single-binary vs microservices.
  • Configure object storage backends and retention policies.
  • Optimize performance, sharding, and handle high cardinality.

Deployment Modes​

  • Single-binary: dev/test, simplest ops.
  • Microservices: independent scaling of distributor/ingester/query/compactor.

Storage Backends​

  • S3 / GCS / Azure / MinIO. Example S3 config:
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: s3.amazonaws.com
      access_key: ${S3_ACCESS_KEY}
      secret_key: ${S3_SECRET_KEY}
      s3forcepathstyle: false

Retention & Compaction​

  • Per-tenant retention; compactor merges blocks for query efficiency.
  • Schedule compaction outside peak hours.

Performance & Sharding​

  • Tune max_block_bytes, flush intervals, and queue sizes.
  • Shuffle sharding tenants across ingesters to isolate noisy neighbors.

High Cardinality Strategies​

  • Keep high-value attributes; hash/drop low-value free-form fields.
  • Use tail-based sampling in the Collector to keep rare errors.

Hands-on Lab​

  1. Switch filesystem backend to MinIO or S3-compatible storage locally.
  2. Configure a shorter retention for dev (24–72h).

Deliverables​

  • Updated Tempo config demonstrating object storage and retention.

Quiz (Self-check)​

  • Why does index-free design help with high cardinality?
  • Explain shuffle sharding.

Resources​

  • Tempo Production Guide
  • Grafana Scaling References

Visual: Distributed Mode Topology​