ASR9000 IOS-XR Fabric Troubleshooting: Commands and Drops

2026年8月24日 LOG admin

Packet drops on a Cisco ASR 9000 can be caused by anything from a failing optical module to fabric-level congestion, and knowing where to look first saves hours of troubleshooting. This article explains the ASR9000 fabric architecture — the FIA (fabric interface ASIC), Trident versus Typhoon line cards, and how packets traverse the fabric — then gives you the exact command sequence used to debug fabric-related packet drops. It is written by a Cisco engineer who maintains the well-known IOS-XR troubleshooting series on the Cisco community.

Why Fabric Troubleshooting Is Different on the ASR 9000

On a fixed pizza-box router, a drop is a drop: you inspect the interface, then the network processor, then you are done. On a modular chassis such as the ASR 9000 the packet path is much longer, and every hop in that path has its own counters, its own alarms and its own failure modes. A single dropped packet can be discarded at ingress policing, inside the network processor's input queue, on the backplane between the line card and the fabric, inside the fabric card itself, at the egress VOQ (virtual output queue), or at the egress interface. If you only look at the interface counters you will see a small, steady trickle of output drops and never find the cause, because the counter that is incrementing sits three stages away from the port you are staring at.

Three properties make fabric faults particularly frustrating. First, they are usually partial — the fabric continues to forward most traffic, so routing protocols stay up and the box looks healthy from the outside. Second, they often correlate with traffic patterns rather than with time, so an incident appears only under a specific mix of destinations or at a specific offered load. Third, they frequently follow an event that is easy to forget: an online insertion and removal (OIR), a software upgrade that left an FPD image outdated, or a line card that came up after a power event.

The practical consequence is that you need a consistent order of operations. Start with the alarms that the platform raises for you, verify that the programmable hardware is running the image the software expects, and only then walk the packet path hop by hop with counters. That order is what the rest of this article follows.

ASR9000 Fabric Architecture Basics

Two line card families matter for fabric troubleshooting. Trident-based cards (A9K-40GE, A9K-2T20G, A9K-4T, A9K-8T/4, A9K-8T, A9K-16T/8) have no fabric stage on the line card, but include bridge ASICs that convert between the NPU memory interface (XAUI) and the FIA (DDR2). Typhoon-based cards (A9K-24x10, A9K-36x10, A9K-MOD80/160, A9K-2x100) include a line card fabric stage to reduce the number of fabric links. The 9922 and 9911 chassis use separate fabric cards (FCs), giving higher fabric capacity per slot.

The distinction matters because it changes what the fabric link count looks like and therefore what a healthy topology looks like in the show output. On a Trident card the NPU reaches the fabric through a bridge ASIC; on a Typhoon card the line card fabric ASIC aggregates internal traffic before it ever touches the backplane. When you compare a working chassis against a suspect one, compare like with like — a Trident card that shows fewer fabric links than a Typhoon card is normal, not a fault.

The FIA itself is a set of ASICs that terminate the fabric links coming from every line card and switch cells between them. It is the component that raises most of the platform alarms you will read later, and it is also the component whose parity errors most often justify an RMA rather than a configuration change.

Prerequisites and Safety Checks Before You Start

Before you touch anything, confirm four things. First, that you have an out-of-band path to the console or the management interface — a fabric-heavy fault can disturb in-band management, and you do not want to lose the box mid-investigation. Second, that you know the chassis type and the line card mix, because the expected fabric topology depends on both. Third, that you have the exact IOS-XR release and the FPD versions, since a mismatch between the running software and the programmable hardware is one of the most common causes of fabric-path failures after an upgrade. Fourth, that you have a maintenance window or at least the authority to reload, because some of the recovery actions are disruptive.

Collect a baseline before you change anything. The platform is far more informative when you can compare a healthy snapshot with a suspect one, and a baseline taken on a quiet network is more useful than one taken during an incident.

show version
show platform
show interfaces summary
show controllers np summary all
admin show hw-module fpd all location all
show tech-support fabric location all

Save the output to a file rather than scrolling through it on the terminal. Fabric investigation involves a lot of counting, and counting is easier in a text editor.

Step-by-Step Fabric Packet Drop Debugging

Step 1: Look for active PFM alarms on LC and RSP

PFM (platform fault management) is the subsystem that watches the hardware and raises alarms when a monitored condition goes out of range. It runs on both the line cards and the route switch processors, which is why the command is issued with location all — a fabric problem is often reported by a line card that is not the one dropping packets.

show pfm location all

Read the output for entries that are not OK. A transient entry that appears once and clears is usually benign; the same entry repeating with a rising count is not. Pay particular attention to any alarm whose text mentions parity, CRC, or a specific ASIC name, since those point at silicon rather than at software.

Step 2: Verify hardware FPGA versions are up to date

The ASR 9000 contains several field-programmable devices — the NPU, the bridge ASIC, the fabric ASIC and the FIA — each of which carries its own image. If the router was upgraded but the FPD images were not, the software may be running against programmable hardware that does not implement the features the software assumes, and the symptom is exactly the kind of subtle, partial drop you are chasing.

show controllers np summary all
admin show hw-module fpd all location all

Compare the running and committed versions column by column. A row showing a different running version against the expected one, or the SW_INACTIVE state, means the device needs a reflash. See the ASR 9901 SW_INACTIVE fix guide for the reflash procedure, and remember that upgrading FPDs is service-affecting on the affected card.

Step 3: Follow the packet drop debugging sections

Check NP drop counters to see whether drops occur at ingress, in the fabric, or at egress, then correlate with fabric-specific counters. The NP summary output shows per-NPU, per-interface counters in buckets, and the bucket names tell you where in the pipeline the discard happened. An ingress-side drop bucket points at policing, queue limits or an oversubscribed input; a fabric-side bucket points at the backplane; an egress bucket points at the VOQ or the egress port itself.

show controllers np counters all location 0/0/CPU0
show interfaces <interface> | include drops
show qos interface <interface> input
show qos interface <interface> output

Do not try to interpret every counter at once. Pick one flow that is known to be affected — ideally a ping or a test stream you control — and follow only the counters that change while that flow is running. That turns an overwhelming wall of numbers into a short list of suspects.

Step 4: Exclude known fabric issues

Two important syslog messages to recognize:

%PLATFORM-PFM_NP-0-TMA_CLUSTER_PARITY — signifies a hardware error
%PLATFORM-DIAGS-3-PUNT_FABRIC_DATA_PATH_FAILED — diag packets from the RP were not returned properly by the NP hardware; check the dedicated Cisco doc for this issue

Both messages indicate that the data path itself failed a self-test, not that a policy or a queue limit discarded traffic. If either appears repeatedly on the same line card, treat it as a hardware fault: reseat the card once if the platform permits it, and if the message returns, open a TAC case with the collected evidence rather than continuing to tune configuration.

Reading PFM Alarms and NP Drop Counters

There is a skill to reading these two outputs quickly. PFM alarms are named by subsystem and severity; the subsystem tells you where the monitor sits, and the severity tells you whether the platform has already decided the condition is service-affecting. A CLEAR alarm states, a RAISED one does not. The NP counters, by contrast, are cumulative since the last counter clear, so an absolute number means very little — what matters is the delta between two samples taken a known interval apart, ideally while a known load is running.

clear controller np counters all location 0/0/CPU0
... wait a known interval, ideally 5 minutes ...
show controllers np counters all location 0/0/CPU0

Compare the delta against the traffic volume you expected. A drop counter that climbs in step with offered load is congestion; a counter that climbs while the load is trivial is a fault. That single distinction removes most of the guesswork from the rest of the investigation.

Fabric Link and FIA Health Verification

The physical fabric is a set of links between every line card and either the fabric cards or the RSP. Verifying those links is a matter of checking that the expected number are up, that they are carrying traffic, and that none of them is reporting a rising error count. This is also where a badly seated card or a marginal backplane connector shows up after an OIR.

show controllers fabric fia link-info all location all
show controllers fabric fia stats all location all
show controllers fabric plane all
show controllers fabric health

Compare the link count against the expected count for the card type and the chassis. A fabric plane that is administratively up but carrying no cells is a strong pointer at either an FIA fault or a card that failed to join the fabric during bring-up. For a structured walkthrough of line card state transitions, including what a card looks like when it is stuck between states, see IOS-XR line card diagnostics with show diag and safe OIR.

Fabric and Backpressure Troubleshooting

When backpressure is suspected, identify the VOQ (virtual output queue) for an interface and check for FIA drops. Backpressure is the fabric's way of telling a line card to stop sending because the destination cannot keep up, and it is a legitimate mechanism — the problem only arises when the backpressure is caused by something you did not intend, such as a single slow egress port being fanned in from many high-speed ingress ports.

show controllers np voq-stats location 0/0/CPU0 interface <if>
show controllers fabric fia drops ingress location all
show controllers fabric fia drops egress location all
show controllers fabric fia stats plane-all

Useful hardware error checks include fabric statistics and VOQ depth monitoring. If one egress port shows a persistently deep VOQ while its neighbours are shallow, the congestion is local to that port: check the interface speed, the negotiated duplex, the line card's shaper configuration, and whether the port is running below its configured rate because of a bad optic. If instead every VOQ on a line card is deep, look at the fabric links serving that card.

Do not overlook the mundane causes. A single member of a bundle that is flapping causes a repeated rehash of traffic across the remaining members, and that rehash is enough to generate fabric-level bursts that look like congestion. Check bundle membership and member flap counters before you conclude the fabric is at fault.

show bundle <bundle-id>
show interfaces <bundle-id> member
show interfaces <interface> | include "flaps|drops"

Building the Evidence Bundle Before a Reload or a TAC Case

Reloading a suspect line card destroys the counter state that would have proven the fault, so collect before you act. The order below captures everything a TAC engineer will ask for, in an order that survives an interrupted session.

show tech-support fabric location all
show tech-support platform location all
show pfm location all
show controllers np summary all
admin show hw-module fpd all location all
show logging
show install active summary

Note the exact timestamp of the collection and the router's uptime, because a TAC engineer will want to correlate the counters with the log timeline. If the fault is intermittent, take two collections an hour apart: the deltas are more valuable than either snapshot.

Common Failure Signatures and FAQ

Drops appear only under high load, and every counter looks mild. This is classic congestion, not fabric failure. Review the QoS policy and the VOQ depths, and confirm that the egress port is actually running at the speed you think it is.

Drops started immediately after an upgrade. Check FPD versions first. A software upgrade that did not reflash the programmable devices is one of the most common post-maintenance faults on this platform.

A single line card shows fabric link errors while its neighbours are clean. Treat it as a card or connector fault. Reseat once, verify the link counts return to the expected value, and escalate to hardware if the errors resume.

Input drops are climbing but the fabric is healthy. Then the fabric is not your problem — the discard is happening earlier in the pipeline. The counter-level walkthrough in IOS-XR input drops and NP counter troubleshooting covers that path stage by stage.

How long should a fabric investigation take before escalating? If you have confirmed the alarms, verified the FPD images, walked the counters and checked the fabric links, you have done roughly an hour of work and you will have either a suspect component or a clean bill of health. Anything past that is TAC territory, and the evidence bundle above is what they will ask for anyway.

Should I clear counters during an investigation? Only deliberately. Clearing fabric or NP counters re-bases every delta you might want later, so take the snapshot first, then clear, then sample again over a known interval.