Cisco IOS-XR SNMP Troubleshooting: Commands That Matter

SNMP timeouts and slow polls on Cisco IOS-XR routers are rarely caused by the network management station — they usually point to how the XR SNMP agent processes PDUs. This article explains the SNMP architecture inside IOS-XR, including the mibd process, caching, and parallel versus serialized processing, then walks through the exact show and debug commands used to isolate PDU performance problems on ASR 9000 platforms. You will learn which counters to collect, how to read the trace request log, and how to spot slow MIB objects.

How SNMP Works Inside IOS-XR

When an SNMP GetRequest, GetNextRequest or GetBulk PDU arrives, the SNMP engine parses it and dispatches the variable bindings to the responsible process — for example, IF-MIB objects go to mibd_interface. A look-ahead cache can answer getnext requests without touching the hardware. There is also a statsd cache enabled with snmp-server ifmib stats cache, refreshed every 30 seconds for all interfaces.

The mental model to hold onto is that IOS-XR is a distributed, multi-process operating system in a way classic IOS never was. There is one SNMP daemon (snmpd) that owns the UDP socket, but the data itself lives in a fleet of MIB agent processes, one per subsystem. A single GetBulk from your monitoring station for 200 interface counters is not one operation — it is a request that fans out to several processes, each of which may have to read counters from a line card before it can answer. Every performance question about SNMP on XR comes back to those two facts.

The IOS-XR SNMP Process Tree

The relevant processes, and what they mean for you:

• snmpd — listens on UDP 161/162, decodes PDUs, validates the community or SNMPv3 user, dispatches each variable binding to the owning MIB agent, and reassembles the reply. If snmpd restarts, every community and user configuration is re-applied from the running config, which is why a snmpd crash is survivable but a snmp-server misconfiguration is permanent.
• mibd_* agents — one per MIB domain (mibd_interface, mibd_entity, mibd_system and so on). These are where the actual work happens, and where a slow OID lives.
• Caches — the GetNext look-ahead cache and the interface statistics cache. Cached answers return in microseconds; uncached answers require a round trip to the hardware or a line card.

Enumerate what is actually running on your box rather than trusting a generic list:

RP/0/0/CPU0:router# show processes | include mibd
RP/0/0/CPU0:router# show processes | include snmp

Key SNMP Troubleshooting Commands

show snmp
show snmp trace requests
show snmp mib access
show snmp mib access time
show snmp mib statistics
show snmp queue rx
show snmp queue trap

Technique notes:

• show snmp — global agent counters (incoming/outgoing request and trap PDUs, error PDUs). Periodically collect output to determine the PDU response rate and error rate.
• show snmp trace requests — a sliding window log of PDU processing (Rx, Proc Start, Tx time). Decode it to find poller source IPs, per-source queue lengths, and how long PDUs waited before processing.
• show snmp mib access / access time — per-OID operation counters and timestamps; diffing these shows exactly what was polled during a period.
• show snmp mib statistics — per-OID transaction time statistics (count + min/max/avg). High averages or large variance point to a slow MIB object.
• show snmp queue rx / queue trap — min/max/avg sizes of the PDU receive, pending and trap queues, with real-time and 5-minute views.

A few more that repay their typing time when you are narrowing a fault:

show snmp request duplicates
show snmp request incoming-queue detail
show snmp engineid
show snmp host
show snmp users
show snmp group
show snmp interface
show snmp mib bulkstat
clear snmp counters

show snmp request duplicates deserves a special mention. A monitoring station that has timed out and retried produces duplicate request IDs, and each duplicate costs the router another full variable-binding lookup. On a router that is already slow, duplicate requests turn a slow poll into a failed poll, which produces more retries, which produces more duplicates. Breaking that feedback loop by fixing the NMS retry settings is frequently the whole fix.

Baselining with show snmp

Raw counters mean nothing on their own. The workflow that actually finds problems is to take two samples a known interval apart and divide.

RP/0/0/CPU0:router# show snmp
! wait 300 seconds
RP/0/0/CPU0:router# show snmp
RP/0/0/CPU0:router# clear snmp counters

Collect the input, output, bad-version, unknown-community and "too big" counters at both samples. Then:

• Input PDUs per second tells you the real poll load. If this is above a few hundred on a low-end route processor, you are approaching the platform's PDU processing capacity and no amount of NMS tuning will help — you need bulkstat or a longer poll interval.
• Input versus output PDUs should be roughly equal for GetRequests. A large gap means responses are being generated but not delivered — look at the queue counters and at the return path.
• "Too big" errors are the classic snmpwalk-succeeds-but-data-is-missing symptom. The response exceeded the maximum packet size, so the router dropped it. On XR the default is 1500 bytes, which is small for a GetBulk response; raising it with snmp-server packetsize is usually the fix.
• Unknown community and bad version counters climbing steadily mean the NMS is sending the wrong credentials, often after a config change on one side only.

Decoding show snmp trace requests

The trace log is a fixed-size ring buffer, so the older entries roll off — read it soon after the fault, not the next morning. Each entry gives you a receive timestamp, a processing-start timestamp and a transmit timestamp for one PDU, plus the source address. Read it three ways:

• Rx to Proc Start is queueing delay. If this is milliseconds, the router is busy. Sort the entries by source IP and you will usually find a single poller whose queue depth is starving everyone else.
• Proc Start to Tx is the processing time for that PDU, which is the sum of the lookups for every OID in it. A single entry taking hundreds of milliseconds with a normal OID count points at a slow MIB agent; the same delay spread across a huge OID count points at an over-ambitious GetBulk.
• Response gaps per source let you tell an overloaded router apart from a leaky network. If every poller shows the same delay, the router is the bottleneck. If one poller shows delay and the others do not, the problem is on that poller's path.

RP/0/0/CPU0:router# show snmp trace requests
RP/0/0/CPU0:router# show snmp trace requests | include 10.10.10.10

Finding the Slow MIB Object

Once you know a delay exists, show snmp mib statistics names the culprit. It reports, per OID, the number of transactions and the minimum, maximum and average time each took. The pattern to look for is not a uniformly high average — that usually means a large table being walked — but a small set of OIDs with a very high maximum and a low count, which indicates an operation that occasionally blocks on hardware.

RP/0/0/CPU0:router# show snmp mib statistics
RP/0/0/CPU0:router# show snmp mib access
RP/0/0/CPU0:router# show snmp mib access time

show snmp mib access gives operation counts per OID, and access time adds timestamps, so diffing two samples tells you precisely what was polled in between. That is how you answer the question "is anything actually walking the full routing table every minute?" — which is one of the most common hidden causes of SNMP overload on a BGP-heavy router.

Queue Depth: Where Backlog Becomes Visible

Queues are the router's pressure gauge. show snmp queue rx shows the receive queue, and show snmp queue trap the trap queue, both as min/max/average over a real-time and a 5-minute window.

RP/0/0/CPU0:router# show snmp queue rx
RP/0/0/CPU0:router# show snmp queue trap

A receive queue that sits at one or two for a moment is normal bursting. A receive queue whose 5-minute average is climbing, or whose maximum hits the configured queue length, means PDUs are being discarded before snmpd ever processes them — and the NMS sees that as a timeout, not as a drop. Tune the queue length with snmp-server queue-length, but treat it as a symptom fix: a permanently deep queue means the poll load exceeds the platform, and the durable answer is fewer, larger polls or scheduled bulk retrieval.

The trap queue is a separate failure mode. If traps are being generated faster than they can be sent — a flapping interface with link-up/down traps enabled will do it — the trap queue fills and later traps are lost. Interface flaps plus SNMP polling on the same box is worth watching as a combined problem.

Configuring SNMP on IOS XR

The configuration that avoids most of these problems is short. Two lines do most of the work: a statistics cache so interface counters are served from memory, and a realistic packet size so GetBulk responses are not discarded.

snmp-server community PUBLIC RO
snmp-server community N0tPublic RW
snmp-server ifmib stats cache
snmp-server ifindex persist
snmp-server packetsize 4000
snmp-server queue-length 100
snmp-server trap-source Loopback0
snmp-server contact netops@example.com
snmp-server location AMS-DC1-RR01
snmp-server host 10.10.10.10 version 2c PUBLIC udp-port 162
snmp-server enable traps bgp
!

Two of these are worth calling out. snmp-server ifmib stats cache is the single most effective change on a router with many interfaces, because interface counters are what virtually every monitoring system polls first; the 30-second refresh means counters are up to 30 seconds stale, which is a trade most operators accept for the reduction in hardware reads. snmp-server ifindex persist keeps interface index numbers stable across reloads, without which every reload invalidates your monitoring history and can shuffle data from one interface into another's series.

For monitoring systems that walk large tables, IOS XR's Bulkstat feature exists precisely for this: instead of walking a table interactively, you define a schema, a transfer and a scheduled interval, and the router assembles the data itself. Inspect it with show snmp mib bulkstat. Modern alternatives worth evaluating are streaming telemetry and gNMI, which push data instead of being polled — see our gNMI dial-in versus dial-out comparison.

Debug Commands (Use with Care)

debug snmp request
debug snmp packet

debug snmp request logs every OID processed by every PDU to syslog, and debug snmp packet logs the same data as show snmp trace requests. Disable logging trap debug if snmp trap syslog is configured, or you will create a trap storm. On NMS side, review timeout and retry settings — row traversal with get-bulk and adequate retries avoids most apparent SNMP failures.

On a busy router, debug snmp request can produce megabytes of log per minute. Treat it as a 30-second instrument: enable it, capture to a bounded syslog target, and turn it off. If you need continuous visibility of which OIDs are being requested, show snmp mib access diffing gives you the same answer with no performance cost.

Common SNMP Problems and Fixes

• Timeouts on a subset of OIDs. One MIB agent is slow or overloaded. Use show snmp mib statistics to identify the OID, then check whether the underlying table is huge or being refreshed too aggressively.
• Timeouts on everything at the same time of day. Two systems polling on overlapping schedules. Stagger the poll intervals; uncorrelated large walks are what saturate snmpd.
• Missing rows in an otherwise successful walk. Response too large — raise snmp-server packetsize or lower the bulk size on the NMS.
• Traps arriving but polls failing, or vice versa. Different UDP ports and different queue paths; a device or firewall ACL permitting 162 but not 161 is a common cause.
• Everything stops after a reload. Interface indexes changed. Enable snmp-server ifindex persist.
• A polling storm after an outage. Every NMS retried simultaneously when the router returned. Fix timeouts and retry counts so they back off rather than synchronise.

Frequently Asked Questions

Why is SNMP on XR slower than the same poll on an IOS switch? On XR the data comes from distributed MIB agent processes and often from line card hardware, not from a single monolithic agent reading a shared counter block. Caching and bulkstat exist to bridge that gap.

Does snmp-server ifmib stats cache reduce accuracy? It makes interface counters up to 30 seconds stale and reduces their granularity. For capacity planning that is irrelevant; for sub-minute traffic spike analysis it is not, and streaming telemetry is the better tool.

Can I raise the SNMP packet size indefinitely? No. Large responses still have to be fragmented at the IP layer and received by the NMS, and a fragment lost in transit fails the whole response. 4000 to 8000 bytes is a practical working range.

What does a snmpd restart mean for my configuration? None of it is lost — it is all in the running config and re-applied automatically. Your polling gaps are the only impact.

Is it safe to run clear snmp counters in production? Yes, but do it deliberately as part of a baseline, and record the pre-clear values first, otherwise you lose the history you were trying to build.

More IOS-XR troubleshooting: input drops in IOS-XR, CRC errors on IOS-XR, the bundle troubleshooting guide, and SNMPv3 security levels and engine ID verification. To send the same data without polling, compare ASR 9000 show commands with the telemetry approach.

原文链接：https://community.cisco.com/t5/service-providers-knowledge-base/asr9000-xr-understanding-snmp-and-troubleshooting/ta-p/3144621