Cisco ASR 9901 SW_INACTIVE Fix: FPD Firmware Reflash Guide

While maintaining a Cisco ASR 9901 core router (ASR 9902 chassis), I hit a nasty low-level hardware bus deadlock. After a routine firmware adjustment and reload, the most critical line card in the box — 0/0/CPU0 (ASR9001-LC) — refused to come up and stayed stuck in SW_INACTIVE (software not activated).

Digging through the logs, the system kept throwing low-level hardware errors during boot, related to backplane cluster communication and Gigabit Ethernet controller programming failures:

BCM5482 cluster port-1 reset done !!
Port 0. SFP Not Ready (0:0). Init failed using TSEC.
No MBI confirmation received from dSC

Root cause: the cluster/slot internal communication bus between the route processor (dSC) and the line card had deadlocked because of outdated microcode. The control plane could no longer push the IOS-XR 6.3.3 MBI (boot image) to the line card, so the card and all its sub-card interfaces stayed completely down.

This article documents the full field procedure: how to break the package-add restrictions of the classic 32-bit IOS-XR (cXR) minimal image, force-refresh the low-level FPD (Field Programmable Device) firmware to Rommon 3.04, and finally bring the line card and every port back to life.

Phase 1: Breaking the cXR Limit to Import the FPD Package

This ASR 9901 only had the minimal base package activated (disk0:asr9k-mini-px-6.3.3), so the software layer lacked the components needed to parse compound .pie packages.

1. The Pitfall: Silent Asynchronous Failure

My first attempt added the package from admin mode using a local relative path:

RP/0/RSP0/CPU0:ios(admin)# install add disk0:/asr9k-fpd-px.pie-6.3.3

The system printed The install operation will continue asynchronously., but show install log 5 detail revealed the task had been blocked by the underlying security mechanism: No log information available. It was a silent failure — the package was never actually added.

2. The Fix: Forcing Distribution with the Source Path Argument

On classic cXR systems you must use the source syntax in admin mode. It forces the lowest-level transfer and file-release mechanism, bypassing the higher-level component dependency checks:

RP/0/RSP0/CPU0:ios(admin)# install add source disk0: asr9k-fpd-px.pie-6.3.3

Note: source disk0: must be followed by exactly one space, then the bare filename.

This time the system actually invoked the extraction engine in the background, parsed the 512 MB FPD package and imported it into the inactive software repository. Operation 6 completed with a success log.

3. Activate and Commit

With the package imported, I issued the asynchronous activation from admin mode. Because the package is huge, activation takes a few minutes — watch progress with show install request:

# Activate the FPD package (Operation ID: 6)
RP/0/RSP0/CPU0:ios(admin)# install activate id 6
# Once progress hits 100% and success is reported, commit immediately
# so the activation survives the next reload
RP/0/RSP0/CPU0:ios(admin)# install commit

Phase 2: Forcing the Physical Firmware Reflash (Brace for the Fan Storm)

With the FPD library active in software, it was time to program the physical chip flash. Note: on classic 32-bit cXR systems the FPD hardware refresh must be run from admin mode.

1. Triggering the Full-Chassis Forced Upgrade

RP/0/RSP0/CPU0:ios(admin)# upgrade hw-module fpd all force location all

The system shows a long 90-minute timeout safety warning. Answer y firmly and press Enter.

2. The Painful Burn: Brief Fan Ramp-Up During CBC Upgrade

During the burn the console scrolled firmware comparisons for the route processor (RP), line card (LC), sub-card (SPA) and fan tray (FAN):

• RP/LC Rommon: force-upgraded from the ancient 1.29 / 1.30 to the brand-new 3.04!
• FPGA devices: fully refreshed (e.g. line card fpga4 from 2.06 to 2.10).

When the burn reached the fan module 0/FT0/SP CBC chip (24.114 to 24.115), the system suddenly raised a Major red alarm:

canb-server[155]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_RED_FAILED

At the same time the fans spun up to full speed, roaring loudly with the red LED on.

💡 Field lesson: do NOT panic at this point! The fan controller chip auto-reloads after its upgrade, which causes a few seconds of CAN bus communication turbulence. As a safety mechanism the fans switch to physical maximum speed (protective runaway). Whatever you do, do NOT cut power, do NOT reseat any card, and do NOT cancel the command — interrupting the Rommon burn will brick the device!

Phase 3: Cold Boot — Watching Everything Come Back Alive

When FPD upgrade has ended. appeared, all core physical chips had been programmed 100% successfully.

1. Inspecting the Staged Firmware State

RP/0/RSP0/CPU0:ios(admin)# show hw-module fpd location all

Rommon already showed 3.04, but the other components (fpga2, fpga4, etc.) — although written Successfully — still showed their old Current SW Version, and the Upg/Dng? column still said Yes.

Key insight: the new microcode is already in flash, but the running memory still holds the old logic. Only a full cold boot / reload — forcing the physical chips to power-cycle and re-initialize — makes the new microcode take effect.

2. Reloading the Router

Exit admin mode, hand the CLI back to the normal XR layer and reload safely:

RP/0/RSP0/CPU0:ios(admin)# exit
RP/0/RSP0/CPU0:ios# reload

3. The Result After Boot

After the 5–10 minute cold re-initialization, logging back in was extremely satisfying:

1. Fans back to quiet: the new Rommon 3.04 handshakes cleanly with the new fan CBC firmware — the red alarm clears, the fans go silent and return to healthy green.
2. Firmware fully in place: back in admin mode, show hw-module fpd location all showed every FPGA running the latest version (line card fpga4 now 2.10), and the whole table flipped to a perfect No in the Upg/Dng? column.

The Finale: Line Card and All 24 Ports Fully Restored

Back in normal mode, the core platform verification command showed the previously deadlocked card reborn:

RP/0/RSP0/CPU0:ios# show platform
Node Type State Config State
-----------------------------------------------------------------------------
0/RSP0/CPU0 ASR9001-RP(Active) IOS XR RUN PWR,NSHUT,MON
0/0/CPU0 ASR9001-LC IOS XR RUN PWR,NSHUT,MON
0/0/1 A9K-MPA-20X1GE OK PWR,NSHUT,MON

Line card 0/0/CPU0 was finally in IOS XR RUN state, and the 20-port Gigabit sub-card showed OK. (For routine post-bring-up checks see ASR9000 line card bring-up verification; for the full show-command reference see ASR9000 IOS XR troubleshooting show commands.)

Checking the interface descriptions again, all 24 high-bandwidth physical ports (20× Gigabit + 4× 10GE) had been released from the deadlock and appeared neatly in the list:

RP/0/RSP0/CPU0:ios# show interfaces description
Interface Status Protocol Description
--------------------------------------------------------------------------------
Gi0/0/1/0 admin-down admin-down
... [Gi0/0/1/1 through 19 omitted]
Te0/0/2/0 admin-down admin-down
Te0/0/2/1 admin-down admin-down
Te0/0/2/2 admin-down admin-down
Te0/0/2/3 admin-down admin-down

The admin-down state at this point only reflects the software-level default shutdown — the hardware layer was fully repaired. The complete low-level FPD firmware refresh succeeded, the ASR 9901 is back at full strength, and the box is ready for the business configuration handover.