Cisco ASR9000 Fabric Troubleshooting: Show Commands

Packet drops on a Cisco ASR 9000 running IOS XR can live anywhere along the data path: ingress NPU, bridge ASIC, fabric interface ASIC (FIA), crossbar or egress linecard. Cisco's service-provider knowledge base documents a structured method to isolate fabric-related drops using a specific set of show commands, and this article walks that procedure from clearing counters to reading NPU and FIA statistics.

What makes fabric troubleshooting on the ASR 9000 hard is not the command syntax — it is the fact that several different failures produce the same symptom. An RSP punt path problem, a FIA parity error and a linecard fabric stage fault can all present as "traffic enters the router and never leaves". The method below works because it divides the data path into a small number of checkpoints and compares counters at each hop; the first checkpoint where the numbers stop matching is where the fault lives.

Background: The ASR 9000 Data Path

A packet entering a linecard normally traverses: the PHY/optics, the ingress NPU (network processor, where parsing, lookup, ACL, QoS classification and rewriting happen), optionally a bridge ASIC, then the Fabric Interface ASIC (FIA) which serialises the packet onto the fabric link, the crossbar on the RSP (via the fabric arbiter), and the same chain in reverse on the egress linecard.

Control traffic and exception packets take a different branch: the NPU punts them to the RSP across the same FIA and crossbar infrastructure. That shared infrastructure is why a fabric fault can look exactly like an RSP failure from the interface's point of view, and why the counter walk below is the only reliable way to tell them apart.

Know Your Linecard: Trident vs Typhoon

Trident-based cards (A9K-40GE, A9K-2T20G, A9K-4T, A9K-8T/4, A9K-8T, A9K-16T/8) have a bridge ASIC between the NPU and the FIA, while Typhoon cards (A9K-24x10, A9K-36x10, A9K-MOD80/160, A9K-2x100) carry a linecard fabric stage to reduce fabric link count. The forwarding architecture differs but the troubleshooting commands are the same.

The distinction still matters in two places. First, on Trident cards there is one additional hop where a packet can be lost — the bridge ASIC — so any counter analysis that shows packets entering the NPU but never reaching the FIA must be narrowed to that bridge. Second, fabric scale differs: Typhoon cards aggregate their fabric links through an on-card fabric stage, so a single failed fabric link has proportionally less impact than on a Trident card where fewer links carry the same bandwidth. Always identify the card type before interpreting fabric link health numbers.

Prerequisite Checks Before You Start

• Confirm the router is not simply rate-limiting or policing a class of traffic — check QoS policy counters first, they are cheaper to read than fabric counters.
• Record the exact ingress and egress interfaces and the flow direction that is failing; "packet loss" is not reproducible unless you can name the flow.
• Confirm you can afford to clear counters. Clearing is the recommended practice because absolute counters accumulate since last reload and are meaningless as deltas, but it destroys any in-progress analysis.
• Have the current time and an NTP-synchronised clock, so alarm timestamps can be lined up with the incident report.
• Never reload or OIR a card before collecting show tech-support fabric — the state is gone afterwards.

Step 1: Look for Active PFM Alarms

RP/0/RSP0/CPU0:router# show pfm location all

Pay attention to alarms such as %PLATFORM-PFM_NP-0-TMA_CLUSTER_PARITY (hardware error) and %PLATFORM-DIAGS-3-PUNT_FABRIC_DATA_PATH_FAILED (diag packets not returned by NP hardware).

PFM (Platform Fault Management) is the ASR 9000 subsystem that correlates hardware error signals into named alarms, and it is the single highest-value command in this procedure because it points directly at hardware instead of making you guess from counters. A parity alarm usually means a real ASIC or memory fault and is a strong candidate for card replacement; the diag-punt alarm means the platform's own diagnostic packets never came back through the fabric, which localises the problem to the fabric or the punt path rather than to customer traffic.

RP/0/RSP0/CPU0:router# show alarms brief system active
RP/0/RSP0/CPU0:router# show logging | include PFM
RP/0/RSP0/CPU0:router# show pfm location 0/1/CPU0

Cross-reference the alarm against the syslog to get the timestamp, then read the fabric counters for the same window — an alarm plus a matching counter increment is a defensible TAC finding, while a counter increment alone is not.

Step 2: Check Hardware FPGA Versions

RP/0/RSP0/CPU0:router# show controllers np summary all
RP/0/RSP0/CPU0:router# admin show hw-module fpd all location all

On Trident linecards, ensure the bridge FPGA revision is above 0.41 for 40G cards and above 0.5 for 80G cards to avoid known memory errors.

FPD (Field Programmable Device) versions are the hardware sibling of a software image, and a mismatch between linecards in the same chassis can produce fabric errors that look like random corruption. Run upgrade hw-module fpd all location all only during a maintenance window, because FPD upgrades reset the affected hardware. When checking, note that show hw-module fpd requires admin mode on many releases and that the output lists each device with a current and required version — any device showing a mismatch is a candidate root cause.

Step 3: Clear All Counters Before Testing

RP/0/RSP0/CPU0:router# clear counters all
RP/0/RSP0/CPU0:router# clear controller np counters all
RP/0/RSP0/CPU0:router# clear controller fabric fia location
RP/0/RSP0/CPU0:router# clear controller fabric crossbar-counters location

Then start the traffic pattern that caused the drop and walk the path.

Be precise about scope when clearing: clear controller np counters all clears every NPU counter in the chassis, which is what you want before a controlled test, but on a production router carrying live traffic it also destroys the historical baseline that would have told you when the problem started. If the fault is intermittent and you do not know when it began, read first, snapshot the values, and only then clear and re-test.

Step 4: Walk the Data Path

! input interface counters
RP/0/RSP0/CPU0:router# show interfaces tenGigE 0/1/0/0

! NPU counters
RP/0/RSP0/CPU0:router# show controllers np counters all location

! NPU to interface mapping
RP/0/RSP0/CPU0:router# show controllers np ports all

! NPU - bridge fabric counters per NPU
RP/0/RSP0/CPU0:router# show controllers np fabric-counters all np0 location

Key NPU counters: 800 PARSE_ENET_RECEIVE_CNT (packets from external interface), 970 MODIFY_FABRIC_TRANSMIT_CNT (packets sent to fabric), 801 PARSE_FABRIC_RECEIVE_CNT (from fabric), 971 MODIFY_ENET_TRANSMIT_CNT (to external interface). Compare them to find where packets stop moving.

The comparison logic is straightforward once the four checkpoints are in place. On the ingress card, if 800 is incrementing but 970 is not, the packet is being dropped inside the ingress linecard before the fabric — look for an ingress policy, an MTU or FCS failure, or a bridge/FIA problem. If 970 increments on the ingress card but 801 does not increment on the egress card, the packet was lost in the fabric itself, which points at FIA links, the crossbar, or the arbiter. If 801 increments but 971 does not, the egress card is dropping internally — the mirror image of the ingress case.

Extend the walk with the fabric-side commands:

RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/1/CPU0
RP/0/RSP0/CPU0:router# show controllers fabric link health
RP/0/RSP0/CPU0:router# show controllers fabric plane all detail
RP/0/RSP0/CPU0:router# show controllers fabric connectivity detail
RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics location 0/0/CPU0

show controllers fabric link health is the fastest way to spot a degraded fabric link — it reports per-link error rates and whether a link has been taken out of service. A link that is administratively up but showing continuous CRC or sync errors is worse than one that has failed outright, because the arbiter may still be scheduling traffic onto it.

Understanding the Fabric Architecture

• Octopus - the Trident fabric interface ASIC on the linecard and RSP.
• SCZ (Santa Cruz) - the actual fabric chip, two on the RSP4/RSP440.
• BE (Bellagio) - the fabric arbiter granting access to the crossbar; both arbiters see every request, so failover is seamless.

The RSP punts and fabric paths share the same FIA, so RSP failures and fabric problems can look identical from the interface - the counter walk above separates them.

The dual-arbiter design means a single arbiter failure does not stop forwarding: both Bellagio arbiters receive every scheduling request, and if one stops responding the other continues to grant access. This is why fabric plane redundancy on the ASR 9000 rarely produces a visible outage — what it produces instead is a performance degradation that is easy to mistake for congestion. When throughput drops without any interface errors or policy counters, check fabric plane state before assuming oversubscription.

Common Failure Signatures

Verification After a Fix

Once a card has been reseated, an FPD upgraded or a fabric link restored, verify rather than assume. Re-run show pfm location all to confirm the alarm is gone and has not reappeared in the log, verify that all fabric planes report up and that no link is flagged, and then repeat the counter walk under the same traffic pattern. Success looks like 800 and 970 on the ingress card and 801 and 971 on the egress card all incrementing at matching rates, with a drop counter that stays flat across the test window. If any of those four counters disagrees, the fault has not been fixed, only moved.

Collecting Evidence for TAC

Before opening a case, gather show tech-support fabric, show pfm location all, show controllers fabric link health, the per-location NPU counter output, and the syslog window around the alarm. Include the linecard type (Trident or Typhoon), the FPD versions, and the exact flow that is failing. A TAC engineer can work from counters and a flow definition; they cannot work from "the router is dropping packets".

FAQ

Do fabric drops always raise an alarm? No. Some fabric faults only show up as counter deltas, particularly when a link is marginal rather than failed. That is precisely why the counter walk is the method and PFM is the shortcut, not the other way round.

Can I clear counters without affecting forwarding? Yes, clearing counters is a statistics operation and does not interrupt traffic. It does destroy your baseline, so record the values first.

Why do RSP problems and fabric problems look the same? Because punt traffic and fabric traffic share the FIA and the crossbar. The only way to separate them is to compare the counters on both sides of the shared element.

Is a single degraded fabric link a service-affecting event? Usually not, because the arbiter reschedules around it — but it does reduce available fabric bandwidth and can produce tail drops under load, which is what makes it worth fixing proactively.

Related articles: Cisco IOS XR configuration management, Cisco Nexus NX-OS VLAN and trunk configuration, and Data center ToR switch selection. For the counter-level detail behind the NPU checkpoints, see Cisco IOS-XR Input Drops: NP Counter Troubleshooting, and for the physical-layer side of the same data path, Cisco IOS-XR Interface CRC Errors: How to Troubleshoot. Linecard hardware faults that surface as fabric errors are covered in ASR9000 IOS-XR Fabric Troubleshooting: Commands and Drops.

原文链接：https://community.cisco.com/t5/service-providers-knowledge-base/asr9000-xr-understanding-the-fabric-and-troubleshooting-commands/tac-p/3812960