Prometheus Alertmanager Routing and Notification Setup

Alertmanager is where a Prometheus deployment either becomes useful or becomes noise. The difference is entirely in the routing tree: group the right alerts together, send them to the right receiver, suppress duplicates, and repeat only when a problem is still unresolved. Get grouping wrong and a single rack power failure produces four hundred notifications; get inhibition wrong and you will page someone about a service being down while they are already fixing the node it runs on. This guide covers the configuration structure, sensible grouping, inhibition rules, and how to test them without waiting for a real outage.

The Configuration Structure

Alertmanager's configuration has four top-level sections that matter for routing:

• global — defaults such as SMTP settings, Slack API URL, and the resolve timeout.
• route — the root routing node: the default receiver plus grouping, timing, and child routes.
• routes (children) — matching rules that override the parent's receiver and grouping.
• inhibit_rules — conditions under which one alert suppresses another.

How routes match

Routes are evaluated top-down and matching stops at the first route whose matchers all match, unless continue: true is set. That single detail causes most "why did this alert go to the wrong channel" investigations.

Step 1 - Configure Receivers

global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'REDACTED'
        require_tls: true

  - name: 'critical-pager'
    slack_configs:
      - channel: '#incidents'
        send_resolved: true

  - name: 'network-team'
    slack_configs:
      - channel: '#network-alerts'
        send_resolved: true

Keep credentials out of the main file — Alertmanager supports secret files referenced with _file suffixed keys, which is the right approach for anything under version control.

Step 2 - Build the Routing Tree

A practical tree routes by severity first, then by team. Grouping is set at the level where it makes sense, and inherited by children unless overridden:

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - matchers:
        - severity = critical
      receiver: 'critical-pager'
      group_wait: 10s
      repeat_interval: 1h
      continue: false

    - matchers:
        - team =~ "network|datacenter"
      receiver: 'network-team'
      group_by: ['alertname', 'device', 'site']
      group_wait: 1m
      repeat_interval: 6h

    - matchers:
        - severity =~ "warning|info"
      receiver: 'default'
      group_wait: 5m
      repeat_interval: 12h

Read the timing parameters carefully, because they are what determine whether the system feels usable:

• group_wait — how long to wait before sending the first notification for a new group. Short for critical (10s), long for warnings (5m) so that a batch of related warnings becomes one message.
• group_interval — how long to wait before sending an update about a group that has new alerts.
• repeat_interval — how often to re-send a notification that is still firing.

Grouping to reduce notification noise

Grouping by device and site for network alerts is deliberate: a switch that loses power generates dozens of interface-down alerts, and grouping by device collapses them into one notification that an engineer can act on.

Step 3 - Add Inhibition Rules

Inhibition suppresses a target alert while a source alert is firing and both share equal label values for the specified labels. The canonical example: if a node is down, do not also alert on every service running on it.

inhibit_rules:
  - source_matchers: [severity = "critical"]
    target_matchers: [severity =~ "warning|info"]
    equal: ['alertname', 'cluster', 'service']

  - source_matchers: [alertname = "NodeDown"]
    target_matchers: [alertname =~ "InstanceDown|ServiceDown"]
    equal: ['instance']

  - source_matchers: [alertname = "SiteUnreachable"]
    target_matchers: [severity = "warning"]
    equal: ['site']

Be strict with equal. If the label is not present on both sides, the inhibition will never fire — which is a silent failure, since Alertmanager has nothing to complain about. Verify that every label you list actually exists on both alert sets.

Step 4 - Validate Before You Deploy

Never restart Alertmanager with an untested configuration. Validate and test the routing with the bundled tool and the HTTP API:

amtool check-config /etc/alertmanager/alertmanager.yml
amtool config routes show --config.file /etc/alertmanager/alertmanager.yml
amtool config routes test --config.file /etc/alertmanager/alertmanager.yml \
  --verify.receivers critical-pager \
  severity=critical team=network
curl -s http://localhost:9093/api/v2/status | jq
curl -s 'http://localhost:9093/api/v2/alerts' | jq '.[].labels'

Testing a route before a change window

amtool config routes test is the command most people do not know about: give it a set of labels and it prints which receiver the alert would reach. Run it for each of your important alert shapes before a change window, and you will never again discover misrouting during an incident.

Troubleshooting Missing or Duplicated Notifications

1. Alert never arrives. Check whether the alert reached Alertmanager at all with curl http://localhost:9093/api/v2/alerts. If it is not listed, the problem is Prometheus-side — a missing or failing alerting rule.
2. Alert arrives but goes to the wrong place. Run the route test with the alert's real labels. A continue set on an earlier route is the usual culprit.
3. Alert arrives repeatedly. Check repeat_interval on the matched route — an inherited value of 5 minutes instead of 4 hours turns one incident into dozens of messages.
4. Alert never resolves. Confirm the rule has an expr that can evaluate to false, and that send_resolved is enabled on the receiver. An alert that never resolves keeps repeating.
5. Alerts are suppressed unexpectedly. Review inhibition rules, especially any with broad matchers. A rule matching severity = "critical" as a source with a broad target will silence large categories of alerts.

Operational Notes

• Route on matchers that exist on the alerts; a matcher for a label that no alert carries never fires.
• Keep continue: false unless you genuinely want an alert delivered to multiple receivers.
• Set repeat_interval per severity — short for critical, long for warnings.
• Use amtool config routes test before every routing change.
• Keep inhibitors narrow, and verify the equal labels exist on both source and target.

Related reading: our Prometheus relabel_configs and target labels guide, the Grafana Loki log aggregation and LogQL guide, and the Zabbix SNMP switch monitoring with LLD article.

原文链接：https://prometheus.io/docs/alerting/latest/configuration/