Integrating with Tempo for Distributed Tracing

Metrics and logs are powerful, but traces add the full request story. In this lesson, you'll configure Alloy to receive OTLP traces and forward them to Tempo (Grafana Cloud or self-hosted).

Learning goals:

• Understand the role of distributed tracing in observability
• Configure Alloy to receive traces via OTLP
• Forward traces to Tempo using otelcol.exporter.otlp
• Apply sampling and batching when needed

Receive Traces via OTLP

The course includes an OTLP receiver example that forwards traces to the Grafana Cloud exporter defined in 01-global.alloy:

docs/grafanaalloy/examples/10-receiver.alloy

// 10-receiver.alloy

otelcol.receiver.otlp "default" {
  // Listen for gRPC traces from your Python app (default port 4317)
  grpc {
    endpoint = "0.0.0.0:4317"
  }


  output {
    // Forward everything to the same Grafana Cloud exporter
    // we defined in your 01-global.alloy
    traces = [otelcol.exporter.otlp.grafana_cloud.input]
  }
}

eBPF Auto-Instrumentation (Beyla)

Alloy can also ingest traces directly from eBPF instrumentation using Beyla:

docs/grafanaalloy/examples/09-ebpf.alloy

// 09-ebpf.alloy

beyla.ebpf "tg_otp" {
	discovery {
		instrument {
			open_ports = "8002"
      name       = "tg_otp"
		}
	}

	ebpf {
		heuristic_sql_detect = true
	}

	output {
		traces = [otelcol.exporter.otlp.grafana_cloud.input]
	}
}

Optional: Sampling and Batching

If trace volume is high, insert sampling and batching processors between the receiver and exporter:

sampling-and-batching.alloy

otelcol.processor.probabilistic_sampler "sample" {
  sampling_percentage = 10.0

  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  timeout = "5s"
  send_batch_size = 1000

  output {
    traces = [otelcol.exporter.otlp.grafana_cloud.input]
  }
}

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }

  output {
    traces = [otelcol.processor.probabilistic_sampler.sample.input]
  }
}

Sending Traces from Your Application

Here's how to instrument a Go application to send traces to Alloy:

main.go

package main

import (
	"context"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func setupTracing(ctx context.Context) (*sdktrace.TracerProvider, error) {
	// Create OTLP exporter to send to Alloy
	exporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint("alloy:4317"),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	// Create resource with service name
	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceName("my-service"),
		),
	)
	if err != nil {
		return nil, err
	}

	// Create tracer provider
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
	)

	// Set as global tracer provider
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	))

	return tp, nil
}

Common Pitfalls

• Missing spans: Ensure all services use the same trace context propagation.
• Port conflicts: OTLP uses ports 4317/4318; don't confuse them with Prometheus or Loki ports.
• Permission issues: eBPF instrumentation requires elevated privileges on most systems.
• Sampling too aggressively: Over-sampling can remove critical traces.

Summary

In this lesson, you've configured Alloy to receive traces, apply optional sampling, and send data to Tempo. You now have a complete observability pipeline across metrics, logs, and traces.

Quiz

Tempo Tracing with Alloy - Quick Check

What protocol does Grafana Alloy use to receive traces from applications?

Question 1/5