Prometheus Alertmanager Routing and Inhibition Rules

Prometheus decides whether something is wrong; Alertmanager decides who hears about it, how often, and in what grouping. Most alerting pain lives in that second half. A rack loses power, three hundred alerts fire, and the on-call engineer learns to ignore the pager. Routing trees and inhibition rules are the tools that turn that event into one actionable page, and they are configured in one file that is worth designing rather than growing by accretion.

The routing tree model

Every alert enters the tree at the top-level route, which matches everything, then traverses child routes. Parameters are inherited from the parent unless the child overrides them. If no child matches, the current node handles the alert. Setting continue: true on a route lets evaluation proceed to the next sibling even after a match — the feature that makes multi-team fan-out possible.

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = critical
      receiver: 'pagerduty'
      continue: true
      routes:
        - matchers:
            - team = network
          receiver: 'network-oncall'
          group_by: ['alertname', 'site']
    - matchers:
        - severity =~ "warning|info"
      receiver: 'slack-noc'
      group_wait: 5m
    - matchers:
        - team = network
        - severity = critical
      receiver: 'network-pagerduty'

The four timing parameters are the ones that determine whether alerting is tolerable:

• group_by — the labels that define a group. Grouping by alertname and site turns a rack failure into one notification per alert type instead of one per device.
• group_wait — how long to collect alerts for the same group before sending the first notification. Too short and a cascading failure pages three times in ten seconds.
• group_interval — minimum time before a new notification for an already-notified group. This is what prevents a slow-burning incident from re-paging on every new symptom.
• repeat_interval — how often an unacknowledged alert repeats. Four hours is a defensible default; a five-minute repeat trains people to ignore the pager.

Inhibition rules

Inhibition mutes a target alert while a matching source alert exists. Both source and target must share identical values for the labels named in equal.

inhibit_rules:
  # If the site is unreachable, do not page for every device in it
  - source_matchers: [ severity = "critical", alertname = "SiteDown" ]
    target_matchers: [ severity =~ "critical|warning" ]
    equal: ["site"]

  # A down host suppresses every alert about that host
  - source_matchers: [ alertname = "HostDown" ]
    target_matchers: [ severity = "warning" ]
    equal: ["instance"]

  # Maintenance window suppresses interface noise
  - source_matchers: [ alertname = "MaintenanceWindow" ]
    target_matchers: [ alertname =~ "InterfaceDown|InterfaceFlapping" ]
    equal: ["instance"]

Three subtleties cause most confusion. A missing label and an empty label value are treated as the same thing, so an inhibition rule whose equal labels are absent from both sides will apply broadly. An alert that matches both the source and target side of a rule cannot inhibit itself, which is deliberate and prevents an alert from silencing its own class. And inhibition only takes effect while the source alert is actually firing — it is not a lasting silence. For planned work use a silence, not a rule.

Matchers and validation

amtool check-config /etc/alertmanager/alertmanager.yml
amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml \
  severity=critical team=network site=dc1
amtool config routes show --config.file=/etc/alertmanager/alertmanager.yml

amtool config routes test is the single most valuable command in this workflow: it tells you which receiver a given label set would reach, without waiting for a real incident. Run it against your planned routes before deploying, and again whenever someone adds a child route. The newer matcher syntax (matchers, source_matchers, target_matchers) is the one to standardise on; the older match/match_re forms are deprecated.

Practical rules of thumb

• Write the routing tree from the on-call engineer's perspective: what single action should each page trigger? If there is no action, it belongs in a dashboard or a low-severity channel.
• Keep continue rare and intentional. An unintended continue on a broad route duplicates every alert.
• Set repeat_interval by severity — fast for page-worthy, slow for ticket-worthy.
• Validate every change with amtool check-config in CI. A broken Alertmanager config means alerts are not delivered at all, which is the worst possible failure mode for a monitoring system.
• Suppress at the source where you can. A for clause in the Prometheus rule, as shown in Zabbix SNMP switch monitoring templates, prevents transient conditions from ever reaching Alertmanager.

For alerting that feeds Alertmanager from a second monitoring stack, see Zabbix proxy in active and passive mode.

原文链接：https://prometheus.io/docs/alerting/latest/configuration