Alertmanager Routing, Grouping and Silences That Work

Alert fatigue is not a metrics problem, it is a routing problem. During a rack failure, two hundred hosts going down should produce one notification that says "rack B is unreachable", not two hundred pages. Alertmanager sits between Prometheus and your humans and decides who hears about what and how often, using four mechanisms: routing, grouping, inhibition and silences. Get those right and the same alerts become actionable; get them wrong and the important one hides in the noise.

The Four Mechanisms

A Complete alertmanager.yml

global:
  resolve_timeout: 5m
  slack_api_url_file: /etc/alertmanager/slack_webhook   # keep secrets out of the YAML

route:
  receiver: 'default-slack'          # root route must have a receiver
  group_by: ['alertname', 'service', 'env']
  group_wait: 30s                    # collect related alerts before first notify
  group_interval: 5m                 # batch newcomers into an existing group
  repeat_interval: 4h                # re-notify while still firing
  routes:
  - matchers:
    - team = web
    receiver: 'web-slack'
    routes:
    - matchers:
      - severity = critical
      receiver: 'web-pagerduty'
      group_wait: 10s
      repeat_interval: 1h

  - matchers:
    - team = platform
    receiver: 'platform-slack'
    routes:
    - matchers:
      - severity = critical
      receiver: 'platform-pagerduty'
      group_wait: 10s
      repeat_interval: 1h

  - matchers:
    - severity = info
    receiver: 'info-slack'
    repeat_interval: 12h

receivers:
- name: 'default-slack'
  slack_configs:
  - channel: '#alerts-default'
    send_resolved: true
    title: '{{ template "slack.title" . }}'
    text: '{{ template "slack.text" . }}'

- name: 'web-slack'
  slack_configs:
  - channel: '#alerts-web'
    send_resolved: true

- name: 'web-pagerduty'
  pagerduty_configs:
  - routing_key_file: /etc/alertmanager/pd_web_key
    send_resolved: true

- name: 'info-slack'
  slack_configs:
  - channel: '#alerts-info'

templates:
- /etc/alertmanager/templates/*.tmpl

inhibit_rules:
- source_matchers: [severity = critical, alertname = InstanceDown]
  target_matchers: [severity =~ "warning|info"]
  equal: [instance]                  # only inhibit for the SAME instance
- source_matchers: [alertname = ClusterDown]
  target_matchers: [alertname =~ "HighLatency|HighErrorRate"]
  equal: [cluster]

Grouping: The Highest-Impact Setting

Without grouping, every firing alert is its own notification — a network partition affecting 200 hosts becomes 200 messages in 30 seconds. With grouping, they collapse into one message that lists the affected instances. Choose group_by deliberately:

• ['alertname', 'cluster', 'service'] is the sweet spot for most applications: one notification per distinct failure mode, per service, per cluster.
• Grouping by instance defeats the purpose during infrastructure-wide events — you get the flood back.
• Grouping only by alertname merges unrelated services into unreadable mega-notifications.

Then tune the timings: group_wait buffers the first alerts so a burst arrives as one message, group_interval controls how often newcomers are appended to an existing group, and repeat_interval decides how often a still-firing group is re-sent. Four hours for critical is a common compromise between fatigue and forgotten incidents.

Inhibition: The equal Gotcha

equal lists the labels that must match identically between the source (inhibiting) alert and the target (inhibited) alert. Omit it and inhibition applies far too broadly — one down node can silence warnings on every host; get the labels wrong and inhibition never triggers. Validate the pairs (instance, cluster, datacenter) against your label scheme and test with a real outage simulation.

Silences With amtool

amtool silence add alertname=~"DiskSpace.*" --duration=4h \
  --comment="storage migration on nas-01" --author="oncall"

amtool silence query                 # what is currently silenced
amtool silence expire <silence-id>   # end it early

amtool alert query                   # what Alertmanager currently holds
amtool check-config /etc/alertmanager/alertmanager.yml   # catch typos before reload

Prefer silence matchers that are as narrow as the work: silencing by instance plus alertname is precision, silencing by cluster during a maintenance window is acceptable, silencing everything during a migration is how a real outage gets missed. Automate silences from your change system so they expire with the change.

Wire It to Prometheus and Verify End to End

# /etc/prometheus/prometheus.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets: ["localhost:9093"]
rule_files:
  - /etc/prometheus/rules/*.yml

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

Verification that takes two minutes and saves an afternoon: fire a test alert with a trivial rule such as vector(1) (or stop node_exporter and watch InstanceDown appear), then watch it travel — visible in the Prometheus UI, present in amtool alert query, grouped into a single notification in the channel, and resolved when the condition clears. If any hop is silent, fix it before the next real incident.

相关阅读：Prometheus SNMP Exporter 监控网络设备、Grafana Loki 日志聚合与 LogQL 以及 Zabbix SNMP 监控网络设备。

原文链接：A working Alertmanager configuration