Alertmanager Routing, Grouping and Silences That Work

Alert fatigue is not a metrics problem, it is a routing problem. During a rack failure, two hundred hosts going down should produce one notification that says "rack B is unreachable", not two hundred pages. Alertmanager sits between Prometheus and your humans and decides who hears about what and how often, using four mechanisms: routing, grouping, inhibition and silences. Get those right and the same alerts become actionable; get them wrong and the important one hides in the noise.

The Four Mechanisms

A Complete alertmanager.yml

global:
  resolve_timeout: 5m
  slack_api_url_file: /etc/alertmanager/slack_webhook   # keep secrets out of the YAML

route:
  receiver: 'default-slack'          # root route must have a receiver
  group_by: ['alertname', 'service', 'env']
  group_wait: 30s                    # collect related alerts before first notify
  group_interval: 5m                 # batch newcomers into an existing group
  repeat_interval: 4h                # re-notify while still firing
  routes:
  - matchers:
    - team = web
    receiver: 'web-slack'
    routes:
    - matchers:
      - severity = critical
      receiver: 'web-pagerduty'
      group_wait: 10s
      repeat_interval: 1h

  - matchers:
    - team = platform
    receiver: 'platform-slack'
    routes:
    - matchers:
      - severity = critical
      receiver: 'platform-pagerduty'
      group_wait: 10s
      repeat_interval: 1h

  - matchers:
    - severity = info
    receiver: 'info-slack'
    repeat_interval: 12h

receivers:
- name: 'default-slack'
  slack_configs:
  - channel: '#alerts-default'
    send_resolved: true
    title: '{{ template "slack.title" . }}'
    text: '{{ template "slack.text" . }}'

- name: 'web-slack'
  slack_configs:
  - channel: '#alerts-web'
    send_resolved: true

- name: 'web-pagerduty'
  pagerduty_configs:
  - routing_key_file: /etc/alertmanager/pd_web_key
    send_resolved: true

- name: 'info-slack'
  slack_configs:
  - channel: '#alerts-info'

templates:
- /etc/alertmanager/templates/*.tmpl

inhibit_rules:
- source_matchers: [severity = critical, alertname = InstanceDown]
  target_matchers: [severity =~ "warning|info"]
  equal: [instance]                  # only inhibit for the SAME instance
- source_matchers: [alertname = ClusterDown]
  target_matchers: [alertname =~ "HighLatency|HighErrorRate"]
  equal: [cluster]

Grouping: The Highest-Impact Setting

Without grouping, every firing alert is its own notification — a network partition affecting 200 hosts becomes 200 messages in 30 seconds. With grouping, they collapse into one message that lists the affected instances. Choose group_by deliberately:

• ['alertname', 'cluster', 'service'] is the sweet spot for most applications: one notification per distinct failure mode, per service, per cluster.
• Grouping by instance defeats the purpose during infrastructure-wide events — you get the flood back.
• Grouping only by alertname merges unrelated services into unreadable mega-notifications.

Then tune the timings: group_wait buffers the first alerts so a burst arrives as one message, group_interval controls how often newcomers are appended to an existing group, and repeat_interval decides how often a still-firing group is re-sent. Four hours for critical is a common compromise between fatigue and forgotten incidents.

Inhibition: The equal Gotcha

equal lists the labels that must match identically between the source (inhibiting) alert and the target (inhibited) alert. Omit it and inhibition applies far too broadly — one down node can silence warnings on every host; get the labels wrong and inhibition never triggers. Validate the pairs (instance, cluster, datacenter) against your label scheme and test with a real outage simulation.

Silences With amtool

amtool silence add alertname=~"DiskSpace.*" --duration=4h \
  --comment="storage migration on nas-01" --author="oncall"

amtool silence query                 # what is currently silenced
amtool silence expire <silence-id>   # end it early

amtool alert query                   # what Alertmanager currently holds
amtool check-config /etc/alertmanager/alertmanager.yml   # catch typos before reload

Prefer silence matchers that are as narrow as the work: silencing by instance plus alertname is precision, silencing by cluster during a maintenance window is acceptable, silencing everything during a migration is how a real outage gets missed. Automate silences from your change system so they expire with the change.

Wire It to Prometheus and Verify End to End

# /etc/prometheus/prometheus.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets: ["localhost:9093"]
rule_files:
  - /etc/prometheus/rules/*.yml

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

Verification that takes two minutes and saves an afternoon: fire a test alert with a trivial rule such as vector(1) (or stop node_exporter and watch InstanceDown appear), then watch it travel — visible in the Prometheus UI, present in amtool alert query, grouped into a single notification in the channel, and resolved when the condition clears. If any hop is silent, fix it before the next real incident.

Receivers: Where an Alert Actually Lands

A route only decides which receiver fires; the receiver decides what the notification looks like when it arrives. Pick one transport per urgency level and configure the resolution behaviour explicitly, because the default of "send a second message when the condition clears" is exactly what you want in a chat channel and exactly what you do not want in a ticket or an on-call tool.

Keep secrets out of the configuration file. Every transport that needs a token accepts a *_file variant (slack_api_url_file, routing_key_file) that reads the value from a path, which lets the YAML live in version control and the credentials live in a secrets mount with tight permissions. Make sure that file is readable by the alertmanager user and by nobody else — a webhook URL is a credential.

receivers:
- name: 'platform-pagerduty'
  pagerduty_configs:
  - routing_key_file: /etc/alertmanager/pd_platform_key
    send_resolved: true
    severity: '{{ if eq .CommonLabels.severity "critical" }}critical{{ else }}warning{{ end }}'
    description: '{{ .CommonAnnotations.summary }}'
    details:
      runbook: '{{ .CommonAnnotations.runbook_url }}'
      instances: '{{ .Alerts.Firing | len }}'
- name: 'incident-webhook'
  webhook_configs:
  - url: 'https://incident.internal/api/alertmanager'
    send_resolved: true
    max_alerts: 20          # bound the payload for very large groups

Time-Based Routing With mute_time_intervals

Silences handle a one-off maintenance window that you know is coming. Mute timings handle the noise that recurs on a schedule: the nightly batch job that always produces a handful of latency warnings, the Sunday backup window, the monthly report that saturates a link for twenty minutes. A time_intervals block defines named windows and a route references them with mute_time_intervals.

time_intervals:
- name: 'batch-window'
  time_intervals:
  - times:
    - start_time: '01:30'
      end_time: '03:00'
    weekdays: ['monday:thursday']
    location: 'Europe/Berlin'      # always pin the timezone
- name: 'change-freeze-weekend'
  time_intervals:
  - weekdays: ['saturday', 'sunday']

route:
  receiver: 'default-slack'
  routes:
  - matchers: [team = platform, severity = warning]
    receiver: 'platform-slack'
    mute_time_intervals: ['batch-window', 'change-freeze-weekend']
  - matchers: [severity = critical]
    receiver: 'platform-pagerduty'   # critical never muted — that is the whole point

Two rules keep this honest. First, always set location explicitly; a mute window that silently uses the container's UTC clock and your operators' local clock produces a window that mutes the wrong hour. Second, never mute severity=critical on a schedule. If a critical alert is noisy enough that you want it muted every night, the alert rule is wrong, not the calendar.

Notification Templates That Cut Phone Calls

The difference between an alert people act on and an alert people ignore is usually four fields: what is broken, where it is broken, since when, and what to do. Alertmanager templates let you put all four into the first two lines of the message using labels and annotations that the Prometheus rule already carries. Define the templates in files referenced by the templates key, so the main configuration stays readable.

{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} ({{ .CommonLabels.severity }})
{{ end }}

{{ define "slack.text" }}
{{ range .Alerts -}}
*{{ .Annotations.summary }}*
Service: {{ .Labels.service }}  Env: {{ .Labels.env }}
Started: {{ .StartsAt | since }}
{{ if .Annotations.description }}{{ .Annotations.description }}{{ end }}
{{ if .Annotations.runbook_url }}Runbook: {{ .Annotations.runbook_url }}{{ end }}
{{ end }}
{{ end }}

Prefer aggregation over repetition inside the template: use .CommonLabels and .CommonAnnotations for values that are identical across the group, and range .Alerts only for the parts that actually differ, such as the affected instance or mount point. A template that prints the full firing list of two hundred hosts defeats the grouping you configured earlier and recreates the wall of text in a single message, which is worse because nobody can scroll past it.

The Prometheus Side: Labels Are the Routing Contract

Alertmanager can only route on labels that already exist. Every matcher in your routing tree is a promise about a label that some rule must set, so write the rules first and design the routes around what they emit. The labels that earn their place in a routing tree are the ones with low cardinality: severity, team, service, env, cluster. Never route on instance or pod — those are group content, not routing keys.

groups:
- name: host.rules
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
      team: platform
      env: '{{ $labels.env }}'
    annotations:
      summary: 'Instance {{ $labels.instance }} down'
      description: 'node_exporter has been unreachable for 5 minutes.'
      runbook_url: 'https://wiki.internal/runbooks/instance-down'
  - alert: DiskWillFillIn4Hours
    expr: predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0
    for: 30m
    labels:
      severity: warning
      team: platform
    annotations:
      summary: 'Filesystem {{ $labels.mountpoint }} on {{ $labels.instance }} will fill'
      runbook_url: 'https://wiki.internal/runbooks/disk-fill'

An alert that lands without a team label falls through to the root route and pings the general channel — which is how a genuinely urgent page ends up buried under routine noise. Make it a review item: for every rule in the repository, ask which route it lands on and who is expected to act. Any rule you cannot answer that question for is either mislabelled or should not exist. The for duration deserves the same scrutiny; a critical alert with for: 0s fires on a single scrape and will page for every transient blip.

Multiple Alertmanagers Without Double-Paging

One Alertmanager is a single point of failure in the notification path: if it is down, nothing is sent and nobody notices until the next incident. Running two instances in a cluster fixes that. They gossip over TCP and UDP port 9094, replicate the notification log and the shared silence set, and elect which peer sends each notification so a group is not announced twice.

# alertmanager-1
alertmanager --config.file=/etc/alertmanager/alertmanager.yml \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=10.0.0.12:9094 \
  --web.listen-address=:9093

# alertmanager-2
alertmanager --config.file=/etc/alertmanager/alertmanager.yml \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=10.0.0.11:9094 \
  --web.listen-address=:9093

Point Prometheus at both addresses rather than at a load balancer, and give each peer the other as --cluster.peer. Keep the configuration files identical: peers that disagree about receivers will produce different behaviour depending on which one wins the send, which is the hardest kind of alerting bug to diagnose. Cluster members must reach each other on the gossip ports directly and be allowed to keep clocks in sync, so run NTP on both.

Troubleshooting: When Nothing Arrives

Alerts that never reach a human almost always fail at the same handful of places. Walk this list in order instead of restarting services and hoping.

1. Is the alert even firing? Check the Prometheus UI under Alerts. If it is absent, the expression or the for duration is the problem, not Alertmanager.
2. Did Alertmanager receive it? amtool alert query shows what is currently held. An empty result with a firing alert in Prometheus means the alerting: block in prometheus.yml is wrong or the reload did not take effect.
3. Does a route match? Print the configuration with amtool config routes show and walk the tree from the root with the alert's actual labels. A missing label sends the alert to the catch-all receiver, which may be a channel nobody reads.
4. Is it inhibited or silenced? amtool silence query lists active silences. A forgotten silence from last week's maintenance is a classic reason for a quiet pager.
5. Is the receiver healthy? amtool check-config validates the file, and a failed webhook or rotated Slack URL fails silently at send time. Watch the Alertmanager log at --log.level=debug and look for the notification attempt.
6. Was it rate-limited by your own settings? A long group_interval plus a long repeat_interval means a genuinely new alert can wait in a group that already exists. Shorten one of them and re-test.

Once you have walked that list, prove the path works end to end with a synthetic alert rather than waiting for a real failure. A dedicated low-severity rule that fires on demand, or a manual amtool alert add with realistic labels, confirms discovery, routing, grouping and delivery in two minutes. Run it after every configuration change, and after every credentials rotation, because those are the two events that break delivery without producing an error anyone reads.

相关阅读：Prometheus SNMP Exporter 监控网络设备、Grafana Loki 日志聚合与 LogQL 以及 Zabbix SNMP 监控网络设备；告警对接日志侧可参考 rsyslog 中央日志服务器，指标长期存储见 VictoriaMetrics 单机与集群部署对比。

原文链接：A working Alertmanager configuration