Skip to main content

10 – Deploying Tempo in Production (Kubernetes + Helm)

Learning Objectives​

  • Deploy Tempo on Kubernetes using Helm.
  • Configure HA, storage, authentication, and multi-tenancy.
  • Integrate with GitOps (Argo CD/Flux) and CI/CD.

Helm Chart (values excerpt)​

Create values-tempo.yaml:

tempo:
  retention:
    traces: 168h
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: s3.amazonaws.com
        s3forcepathstyle: false
  query_frontend:
    replicas: 2
  distributor:
    replicas: 2
  ingester:
    replicas: 3
  compactor:
    replicas: 1

serviceMonitor:
  enabled: true

Commands​

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install tempo grafana/tempo-distributed -n observability -f values-tempo.yaml --create-namespace

Security & Multi-Tenancy​

  • Add mTLS at ingress; use proxy/auth to enforce tenant IDs.
  • Configure tenant quotas and per-tenant retention.

Operator vs Helm​

  • Helm gives explicit control; Operators provide declarative APIs and automation. Choose by team preference and ops model.

GitOps​

  • Store values-tempo.yaml in Git; reconcile with Argo CD.
  • Automate smoke checks via TraceQL queries in CI.

Hands-on Lab​

  1. Deploy on Kind/Minikube.
  2. Validate ingestion, queries, and retention.

Deliverables​

  • A working Tempo on Kubernetes with object storage configured.

Quiz (Self-check)​

  • Which components should run multiple replicas for HA?
  • Where is retention defined in values?

Resources​

  • Grafana Helm Charts
  • Tempo Distributed chart docs

Visual: Kubernetes Deployment​