Prometheus relabel_configs: Target Labels Guide

Relabeling is where most Prometheus configuration bugs live. The same eight fields (source_labels, separator, regex, action, target_label, replacement, modulus) behave differently depending on which of the four relabel stages you put them in, and a rule that "looks right" silently drops every target. This guide explains each stage, the actions that matter, and how to verify with the targets API instead of restarting Prometheus repeatedly.

The four stages

• relabel_configs – runs against discovered targets before scraping. This is where you keep/drop targets, rewrite __address__, __metrics_path__ and promote service-discovery metadata (__meta_*) into real labels.
• metric_relabel_configs – runs on samples after scraping but before storage. Use it to drop high-cardinality series, rename metrics and add labels.
• write_relabel_configs – filters what gets forwarded by remote_write.
• alert_relabel_configs – adjusts alerts on their way to Alertmanager.

Labels starting with __ are internal: they must be promoted with target_label or they disappear before the sample is stored.

Anatomy of a rule

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace

source_labels values are concatenated with separator (; by default) and matched against regex, which is fully anchored – it must match the whole string. Capturing groups are referenced as $1, ${1} in replacement, which defaults to $1. The default action is replace, so most rules only need source_labels and target_label.

Actions you will actually use

• keep / drop – filter targets or samples by regex; the fastest way to exclude a namespace or a noisy metric family.
• labelmap – rename all discovered labels matching a regex (typical for __meta_kubernetes_pod_label_(.+)).
• labeldrop / labelkeep – prune label sets before storage.
• hashmod – shard targets across Prometheus instances with modulus and an external label, the standard building block for horizontal scaling.

Common patterns

Scrape only annotated pods

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    regex: "true"
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
    regex: (.+)
    target_label: __address__
    replacement: ${1}

Derive several labels from one filename (file_sd)

relabel_configs:
  - source_labels: [__meta_filepath]
    regex: "/etc/prometheus/targets/hostname_([^_]+)_(dev|tst|uat|prod)\.yml"
    target_label: type
    replacement: $1
  - source_labels: [__meta_filepath]
    regex: "/etc/prometheus/targets/hostname_([^_]+)_(dev|tst|uat|prod)\.yml"
    target_label: env
    replacement: $2

One rule can only write one target label, so splitting a pattern into several labels means repeating the regex with different replacement values.

Debugging without guessing

curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {scrapeUrl, labels, lastError}'
curl -s localhost:9090/api/v1/targets?state=dropped | jq '.data.droppedTargets[] | .discoveredLabels'

The targets API shows both discoveredLabels (before relabeling) and labels (after), plus dropped targets with their reason – that single view answers most "why is my target down" questions. In the UI, the Service Discovery and Targets pages expose the same data.

Mistakes that cost the most time

• Forgetting that regexes are anchored: regex: prod will not match my-prod-cluster.
• Using metric_relabel_configs to filter targets – it runs after scraping, so the scrape still happens.
• Promoting a label that already exists, which overwrites an external label such as instance.
• Ordering: rules execute top to bottom, so a replace that depends on another rule's output must come later.

Related: Alertmanager routing and silences, VictoriaMetrics single-node vs cluster, and Grafana Loki and LogQL.

原文链接：https://prometheus.io/docs/prometheus/latest/configuration/configuration/