Setting Up Service Discovery for Dynamic Targets

Modern infrastructure is dynamic. Containers come and go, pods move between nodes, and services scale automatically. Alloy's discovery components keep your scrape targets up to date without constant manual edits.

Learning Goals

By the end of this lesson, you will be able to:

• Understand the role of service discovery in dynamic environments
• Configure Kubernetes, Consul, Docker, and file-based discovery
• Apply discovery.relabel to normalize target labels
• Feed discovered targets into prometheus.scrape or log sources

How Discovery Works in Alloy

Discovery components emit a list of targets. Those targets are then used by scrapes or log sources. You can optionally run discovery.relabel in between to rename, add, or drop target labels.

Example: Static Target with Relabeling

This example builds a target list and sets a clean instance label before scraping:

docs/grafanaalloy/examples/05-traefik.alloy

// Discovery block to set the address
discovery.relabel "traefik_node" {
  targets = [{
    __address__ = "localhost:8080",
  }]

  // Set a clean instance name
  rule {
    target_label = "instance"
    replacement  = "traefik-proxy"
  }
}

// Scraper block
prometheus.scrape "traefik" {
  targets    = discovery.relabel.traefik_node.output
  // Pointing to your global standardization gate
  forward_to = [prometheus.relabel.common_labels.receiver]

  job_name   = "traefik"
}

Example: Docker Discovery for Logs

Docker discovery is commonly used for container logs. The example below discovers containers and adds a service label before shipping logs:

docs/grafanaalloy/examples/04a-docker-logs.alloy

discovery.docker "linux" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "logs_integrations_docker" {
  targets = []

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex = "/(.*)"
    target_label = "service_name"
  }
}

loki.source.docker "default" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.linux.targets
  labels     = {"platform" = "docker"}
  relabel_rules = discovery.relabel.logs_integrations_docker.rules
  forward_to = [loki.write.grafana_cloud_loki.receiver]
}

Kubernetes Discovery (Metrics)

A minimal Kubernetes discovery configuration looks like this:

kubernetes_discovery.alloy

discovery.kubernetes "pods" {
  role = "pod"
}

prometheus.scrape "k8s_pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.relabel.common_labels.receiver]
}

Consul Discovery (Metrics)

consul_discovery.alloy

discovery.consul "consul_services" {
  server  = "consul-server:8500"
  tags    = ["metrics", "production"]
  services = ["api-.*", "web-.*"]
}

prometheus.scrape "consul_scrape" {
  targets    = discovery.consul.consul_services.targets
  forward_to = [prometheus.relabel.common_labels.receiver]
}

File-Based Discovery

File discovery lets external systems generate targets (JSON/YAML). Alloy watches those files and updates targets automatically.

file_discovery.alloy

discovery.file "file_sd" {
  path_targets = ["/etc/alloy/targets/*.json"]
  refresh_interval = "15s"
}

prometheus.scrape "file_sd" {
  targets    = discovery.file.file_sd.targets
  forward_to = [prometheus.relabel.common_labels.receiver]
}

Common Pitfalls

• Missing RBAC: Kubernetes discovery needs permission to list resources.
• Network access: Discovery components must reach their API endpoints.
• Label explosion: Avoid dumping all metadata into labels.
• File formatting: Invalid JSON/YAML targets are ignored until fixed.

Summary

Service discovery is essential for monitoring dynamic infrastructure. In this lesson, you learned how to:

1. Use discovery.kubernetes, discovery.consul, and discovery.file
2. Apply discovery.relabel to normalize target labels
3. Feed discovered targets into metrics or log pipelines

The discovered targets will serve as input for relabeling, which we'll cover next.

Quiz

Service Discovery in Alloy - Quick Check

What is the primary benefit of using service discovery instead of static target configuration?

Question 1/5