Prometheus snmp_exporter Alerting for Network Devices

SNMP polling in Prometheus is a two-part problem: converting vendor MIBs into something the exporter can walk, and writing alert expressions that are meaningful for devices rather than for hosts. The snmp_exporter generator solves the first part, but it has to be built locally because it depends on Net-SNMP, and that build step is where most first attempts stall. This article covers generating a usable snmp.yml, wiring it into Prometheus, and the alert rules that actually catch interface and device problems.

Build the generator

sudo yum install gcc make net-snmp net-snmp-utils net-snmp-libs net-snmp-devel golang-bin
git clone https://github.com/prometheus/snmp_exporter.git
cd snmp_exporter/generator
make generator mibs

The dynamic dependency on Net-SNMP is why there is no prebuilt generator binary. Once built, organise MIBs one directory per device family so that symbol names cannot collide between vendors, then merge the outputs into a single runtime snmp.yml.

Define modules in generator.yml

modules:
  if_mib:
    walk:
      - 1.3.6.1.2.1.2          # ifTable: interfaces
      - 1.3.6.1.2.1.31.1.1     # ifXTable: 64-bit counters, names
    lookups:
      - source_indexes: [ifIndex]
        lookup: ifAlias
      - source_indexes: [ifIndex]
        lookup: ifDescr
      - source_indexes: [ifIndex]
        lookup: ifName
    overrides:
      ifAlias:
        ignore: true          # used as a label, not a metric
      ifDescr:
        ignore: true
      ifName:
        ignore: true

The lookups turn ifIndex into human labels, which is what makes alerts readable. The ignore overrides stop those strings from being exported as metrics, where they would only inflate cardinality.

./generator generate -m /tmp/mibs -g /tmp/generator.yml -o /tmp/snmp.yml
promtool check config /tmp/snmp.yml 2>/dev/null || true
systemctl restart snmp_exporter

Point Prometheus at it

scrape_configs:
  - job_name: snmp
    metrics_path: /snmp
    params:
      module: [if_mib]
    static_configs:
      - targets:
          - switch-01.example.com
          - router-01.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116

That relabel chain is the canonical snmp_exporter pattern and worth understanding rather than copying: the device address becomes a parameter, the exporter address becomes the actual scrape target, and instance is preserved as the device name so alerts name the switch instead of localhost.

Alert rules that are worth paging on

groups:
- name: network-devices
  rules:
  - alert: InterfaceDown
    expr: ifOperStatus{ifAdminStatus="1"} != 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} {{ $labels.ifName }} is down"

  - alert: InterfaceFlapping
    expr: changes(ifOperStatus[15m]) > 4
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} {{ $labels.ifName }} flapped {{ $value }} times"

  - alert: HighInterfaceErrors
    expr: rate(ifInErrors[5m]) > 1
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Input errors rising on {{ $labels.ifName }}"

  - alert: DeviceUnreachable
    expr: up{job="snmp"} == 0
    for: 3m
    labels:
      severity: critical

Three of these four would be impossible without decent labels. InterfaceDown filters on admin status so decommissioned or administratively shut ports do not generate noise — without that filter, a well-maintained switch looks permanently broken. InterfaceFlapping catches the marginal optic or cable that a simple up/down alert misses entirely. And the rate expression on ifInErrors catches physical-layer degradation before it becomes an outage.

Operational notes

• Poll interval — SNMP walks are expensive. Ten minutes is a common compromise for large devices; faster polling needs narrower walks rather than a shorter interval.
• Timeout and retry — busy devices exceed the default timeout, which produces false up == 0 alerts. Raise the timeout per job before raising the retry count.
• Cardinality — every interface is a time series. On a chassis with hundreds of ports, walking tables you do not alert on is the fastest way to grow your TSDB.
• Counter resets — device reboots reset 64-bit counters. Use rate() and increase(), which handle resets, rather than subtraction.

This pairs naturally with the flow-based view: interface counters show that something is wrong, while NetFlow or IPFIX, covered in NetFlow, IPFIX and sFlow telemetry compared, shows who is generating it. If you run Zabbix alongside Prometheus, the template-based approach in Zabbix SNMP switch monitoring templates is worth comparing on cost of ownership before you standardise on one.

原文链接：https://github.com/prometheus/snmp_exporter/blob/main/generator/README.md