Cisco ASR 9000 Boot Loop After Upgrade: IOS XR Recovery

A Cisco ASR 9000 that boots into a loop after an IOS XR upgrade is one of the more alarming failure modes in a service provider network — line cards come up, crash, and reload endlessly while the router never reaches a stable forwarding state. Cisco's IOS XR system setup troubleshooting guide identifies a software synchronization problem after upgrades performed with customized GISO images containing updated SMUs on top of a base ISO. This article translates that official procedure into a practical recovery run book, with the checks that confirm recovery and the steps to take when the loop persists.

What a Persistent Boot Loop Looks Like

Before applying the fix, confirm you have the right failure. A genuine post-upgrade boot loop has a characteristic signature:

• One or more line cards repeatedly reach a working state and then reset, rather than staying failed once.
• show platform shows a card cycling between IOS XR RUN and a bring-up or failure state; the card's uptime never grows past a few minutes.
• The console or logging buffer shows repeated bring-up messages from the same location, with a restart reason on each cycle.
• Traffic on the affected card's interfaces flaps in step with the reloads, while cards in other slots stay stable.
• The problem survived at least one full router reload — this is the key distinction from a transient bring-up failure.

If the card plateaus in a failed state instead of cycling, you are looking at a different problem, usually a hardware or FPD mismatch. Our IOS XR line card diagnostics and OIR guide covers that path.

Why Line Cards Boot-Loop After an Upgrade

When a router is upgraded using a customized GISO image that layers updated SMUs over a base ISO image, the software set on the route processor and the software set expected by the line cards can fall out of sync. The line cards then repeatedly fail bring-up and reload. The fix is not to re-copy the image — it is to force a software synchronization so every node reloads from the network boot media with the correct software set.

It helps to understand where the mismatch comes from. Each node in the chassis boots its own image from a copy on the local boot media, and the route processor is the authority on which packages are active. A customized GISO changes the composite set the RP is running, and if a line card's local copy or its cached package manifest does not line up with what the RP expects, the card's bring-up fails a consistency check and it restarts. Because the card retries forever, the symptom is an endless loop rather than a clean failure — and because a plain reload does not force the cards to re-fetch their software, the loop survives reboot after reboot.

Before You Start: Prerequisites and Safety

Three things make this procedure safe rather than risky:

1. Know which nodes are healthy. Capture show platform and show install active before you touch anything, so you can tell the affected from the unaffected nodes afterwards.
2. Understand the blast radius. The all form of the command reloads every node, including the route processor and therefore the whole router. If only one line card is looping, target that location instead.
3. Have out-of-band access. A reload of all nodes will drop every management connection that terminates on the chassis. Use a console server or a management path you know survives a control-plane reload.

Recovery Procedure: Software Synchronization

Execute the hw-module location bootmedia network reload command to enforce software synchronization on the affected nodes. This reloads the node using the network boot media, which pulls the active software set and brings the card back in sync:

Router# hw-module location all bootmedia network reload
Router# hw-module location 0/0/CPU0 bootmedia network reload
Router# hw-module location 0/1/CPU0 bootmedia network reload

The first form targets every node; the second form is for a single line card or route processor specified in rack/slot/module notation. Run it from admin or EXEC mode according to your IOS XR release, then let the node complete the reload cycle before checking state.

A few operational notes on the command itself. The location is written in rack/slot/module form where the module is CPU0 for a line card or route processor — 0/0/CPU0 is the first line card slot on a single-rack chassis, and 0/RSP0/CPU0 is the primary route processor. The bootmedia network keywords are what force the fetch from the RP instead of the local copy; omitting them only reboots the card from the same stale image and the loop returns. Run the command, then watch the node rather than the prompt: the CLI returns immediately while the reload runs asynchronously in the background.

Expected timeline on an ASR 9000 is several minutes per node, longer if the software set must be transferred over a busy management path. Wait for the card to reach IOS XR RUN and for its uptime to keep climbing before you declare the node fixed — a card that returns to the loop within a few minutes tells you the synchronization did not take, which almost always means an FPD mismatch prevented it.

Make FPD Auto-Upgrade Work for You

Cisco explicitly recommends enabling the FPD auto-upgrade feature for a successful software synchronization. If FPD images on the line cards are older than what the new software expects, the card can fail bring-up again after the sync:

Router(config)# hw-module location 0/0/CPU0 fpd auto-upgrade
Router# show fpd package
Router# show hw-module fpd location 0/0/CPU0
Router# show hw-module fpd location all

Confirm the FPD package versions with show fpd package before and after the reload so you can prove the upgrade completed. The comparison that matters is between the "Min Required" column of show fpd package for your new IOS XR release and the versions reported by show hw-module fpd location all. If any entry is below the minimum, the card will keep failing — and the fix is then the dedicated reflash procedure rather than another synchronization attempt.

FPD issues have their own dedicated recovery path — see our ASR 9901 SW_INACTIVE FPD firmware reflash guide and the SW_INACTIVE FPD recovery guide for that scenario.

Verifying the Router Recovered

After the sync completes, verify that every node is stable and running the committed software set:

Router# show platform
Router# show platform | include IOS XR RUN
Router# show install active summary
Router# show install committed
Router# show redundancy
Router# show logging | include %PLATFORM

All line cards should show IOS XR RUN or a steady OK state in show platform, and the active software set should match the committed set. The include filter on the platform output gives you a fast count of healthy nodes and highlights the one that is missing from the list. show install active summary and show install committed should agree; if they do not, commit the install before doing anything else, otherwise the next reload will reintroduce the mismatch you just fixed.

The logging check is the tie-breaker. Scan for platform messages timestamped after the synchronization and confirm there are no further bring-up failures from the affected location. A silent log for several hours is the real evidence that the loop is gone.

If cards still flap, escalate with show tech-support collected from the affected location before contacting the TAC. Related show-command fundamentals for ASR 9000 IOS XR are covered in our ASR 9000 show command guide and the ASR 9000 line card bring-up verification checklist.

If the Loop Continues

Work through these in order, because each one eliminates a whole class of causes:

1. Compare active and committed software sets. If they disagree, commit and reload — a large share of "the fix did not work" cases are uncommitted package state.
2. Check FPD versions against the package table. A card that satisfies the software check but fails the FPD check will loop forever.
3. Inspect the card's restart reason. Repeated identical reasons from the same node point at that card, not at the chassis; a card that fails bring-up on a known-good software set is a hardware candidate.
4. Re-verify the image. A corrupted or partially transferred GISO produces exactly this symptom, and re-copying the image after verifying its checksum is cheap insurance.
5. Isolate the node. If the card is causing traffic instability on a production chassis, shutting it down stops the flapping while you continue the investigation, and the rest of the chassis keeps forwarding.

Why a GISO Upgrade Is Different from a Plain ISO Upgrade

A customized GISO is a composite: a base IOS XR ISO plus one or more SMUs and package updates layered on top, packaged so the router activates the whole set in one operation. That layering is what makes it convenient and what creates the synchronization hazard. With a service-pack or plain ISO upgrade, the delta between the old and new software set is well defined and the release was tested as an integrated whole. With a customized GISO, the delta is whatever the engineer assembled — and a line card whose local package manifest predates the composite set has no way to validate it.

The practical consequence is that a GISO upgrade deserves more verification than a standard one, not less: check the FPD table before activation, activate and commit without delay, and never leave the chassis in an activated-but-uncommitted state across a maintenance window boundary. If you need to prepare images weeks in advance, stage them on the RP's harddisk and activate later — but keep the composite set recorded in the change ticket so the next engineer knows exactly what "the running software" means on this chassis.

Alternate Recovery Paths

If the command-line synchronization is not available — because the router is not reachable through the management plane, or because the route processor itself is the node that is looping — you still have options before you consider a re-image:

• Console access into ROMmon. Every ASR 9000 route processor presents a boot loader on the console. From there you can inspect the boot media and start the node with a specific image, which is enough to get a working RP from which the synchronization commands become available again.
• Boot the previously committed software set. If show install inactive listed the previous set before the upgrade, restoring it returns the chassis to a known-good state and gives you a stable platform to plan the upgrade from.
• Re-stage the image. A corrupted transfer reproduces this failure exactly. Re-copying the GISO after verifying its checksum, then activating and committing cleanly, resolves a surprising number of "boot loop" cases that were never actually a synchronization problem.

Whichever path you take, record the software set and FPD versions you ended up with. Recovery by trial and error leaves a chassis in a state that nobody can describe later — and the next upgrade will be planned from that undocumented baseline.

Preventing the Next Occurrence

• Run the FPD comparison before every upgrade window, not after the first line card fails.
• Enable FPD auto-upgrade as part of the upgrade procedure so the router performs the step it can do itself.
• Always finish an upgrade with install commit, and record show install committed output in the change ticket.
• Verify the GISO checksum after transfer and before activation.
• Capture show platform before and after, so a recurring problem is provably not caused by the change.

Frequently Asked Questions

Does hw-module location bootmedia network reload wipe the configuration?

No. It reloads the node's software from the network boot media; the running configuration on the route processor is unaffected. That is precisely why it is the preferred fix for a software synchronization problem over a full re-imaging.

Can I run the command on a single line card instead of the whole chassis?

Yes, and you should whenever only one location is affected. Replacing all with a specific rack/slot/CPU0 limits the impact to that node and keeps the rest of the chassis forwarding.

How long should a synchronized line card take to come back?

Plan for several minutes per node. The card must reload, re-fetch the active software set and re-run its bring-up sequence. If it returns and loops again within a couple of minutes, the synchronization itself failed and the FPD check is your next step.

原文链接：https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/26xx/system-setup/configuration/guide/b-system-setup-cg-asr9000-26xx/m-troubleshooting.html