Skip to main content

Processing and Transforming Metrics with Pipelines

Alloy is designed to build pipelines. For metrics, the primary processing step is relabeling: filtering series and standardizing labels before data reaches your backend.

Learning Goals​

By the end of this lesson, you'll be able to:

  • Understand where metric processing happens in Alloy
  • Use prometheus.relabel to add, drop, and transform labels
  • Chain scrapes into relabeling pipelines
  • Decide which transformations belong in Alloy vs. your metrics backend

Why Process Metrics in Alloy?​

Before metrics reach storage, it's useful to:

  • Add common labels (environment, region, team)
  • Drop noisy series to reduce cardinality
  • Normalize label names across services

Alloy focuses on label-level processing. Numeric aggregation and math are typically handled by recording rules in Prometheus/Mimir or in Grafana query layers.

Example: Common Labels Pipeline​

The course uses a shared relabeler defined in 01-global.alloy:

docs/grafanaalloy/examples/01-global.alloy (excerpt)
prometheus.relabel "common_labels" {
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]

  rule {
    target_label = "env"
    replacement  = "production"
  }
}

This relabeler adds the env=production label to all metrics flowing through it and forwards them to prometheus.remote_write.grafana_cloud.

Example: Filtering Metrics with prometheus.relabel​

You can add additional rules to drop or keep metrics before they hit remote write:

filtering_metrics.alloy
prometheus.relabel "filter_metrics" {
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]

  rule {
    action        = "drop"
    source_labels = ["__name__"]
    regex         = "go_.*"
  }

  rule {
    action        = "keep"
    source_labels = ["__name__"]
    regex         = "http_.*"
  }
}

Pipeline Composition​

A typical metrics pipeline looks like this:

Exporter -> Scrape -> Relabel -> Remote Write

In the course examples, prometheus.scrape components forward to prometheus.relabel.common_labels, which forwards to remote write.

Where to Do Metric Math and Aggregation​

Alloy doesn't provide metric math or aggregation components. Use:

  • Recording rules in Prometheus or Mimir for derived metrics
  • Grafana queries for calculated views

This separation keeps the collector lightweight and focuses Alloy on collection and labeling.

Common Pitfalls​

  • Rule order matters: drop and keep rules are processed in order.
  • Over-dropping labels: Removing labels too early can make troubleshooting impossible.
  • Mismatch between pipelines: Make sure all scrapes that should be standardized flow through the same relabeler.
  • Confusing target relabeling with metric relabeling: discovery.relabel changes targets; prometheus.relabel changes metric labels.

Summary​

In this lesson, you learned how to process metrics with Alloy by using label-based pipelines:

  1. prometheus.relabel filters and normalizes labels
  2. Pipelines route metrics from scrapes to relabeling to remote write
  3. Aggregation and math should be done in the backend (Prometheus/Mimir)

These practices reduce cost, enforce consistency, and keep your observability data clean.

Quiz​

Metrics Processing Pipelines - Quick Check

What is the primary purpose of the `prometheus.relabel` component?

Question 1/5