HPE 3PAR Error Codes: checkhealth and Disk Troubleshooting

When a 3PAR array starts emailing error alerts, the instinct is to fix the specific error - but an isolated alert only shows a narrow slice of what is happening. The right move is to run checkhealth first to get a full picture of hardware, software, and configuration issues, then drill into the specific problem. This guide, based on a practical 3PAR troubleshooting reference, walks through checkhealth, degraded disk diagnosis, cage loop offline errors, PSU replacement, and how to prioritize issues by severity.

Start with checkhealth

Log into the array via SSH with admin credentials and run:

checkhealth -detail -svc

The -svc and -detail flags give a more detailed output covering components, software, and configuration. Use this as the baseline before chasing any individual alert. checkhealth is deliberately broad: it inspects nodes, cages, disks, ports, the software version and the configuration, and reports anything that is degraded or inconsistent. Because it is broad, it is also the best way to answer the question an individual alert never answers - is this one problem, or one symptom of many?

Understanding 3PAR Alert Vocabulary

The array's alerts use a small vocabulary that is easy to misread. Getting the terms right avoids chasing the wrong component:

• Degraded: a component is not performing at full capability but is still in service. A degraded disk can still be serving data, and a degraded node can still be online - neither has failed yet.
• Failed: a component has stopped functioning and the array has compensated. A failed disk has already had its data moved to spare space.
• Missing: a component the array expects to see is not present or not responding - often a cabling, seating or port problem rather than a dead part.
• Offline: a path or loop is out of service entirely, which is more serious than a single degraded component because it removes redundancy.

Reading an alert with this vocabulary in mind immediately tells you whether you are looking at a part that needs replacing, a part that needs reseating, or a path that needs investigating.

Degraded vs Failed Disks

A "degraded" disk does not necessarily mean the disk has failed. There are 40+ reasons a disk can go degraded - including a failed node that makes all disks sharing its path appear degraded (which can produce 40+ alarming messages at once). Diagnose with:

showed -failed -degraded

Common causes: a disk that is physically missing (but may be fine if still seated), or a missing A/B port that points to a cage or node error. A fully failed disk has already had its data evacuated from normal space to spare space, and the array will move data from used to spare space in preparation for replacement. That last point is important operationally: when you see a disk fail, the array is already rebuilding, so the priority is to replace it promptly with a like-for-like drive so the spare pool is not exhausted by a second failure.

Cage Loop Offline Errors

Each disk cage has two connections - left port (A) and right port (B). If either goes offline you get a cage loop offline error, which can be caused by a failed cable, the port itself, the whole cage, or the node. First determine whether the node or the cage caused the outage: an amber LED on the node or cage port shows the failure point. Then inspect with:

showcage -d
showport

These commands show which cage ports and node ports are involved (for example, cage0/cage1 plugged into ports 1:2:1, 1:2:2, 0:2:1, 0:2:2 of type disk). The A/B design exists so that a single cable or port failure does not take the loop down; a loop-offline alert therefore means both paths to that point were lost, which is why it ranks higher than an isolated degraded disk.

Reading checkhealth Output: What to Prioritize

checkhealth output can be long, and the temptation is to work top to bottom. Resist it. Instead, scan for severity and for the components that remove redundancy, in this order:

• Failed nodes first - a node failure is the most consequential event on the array and can explain a cascade of downstream "degraded" reports.
• Offline loops and failed cages next - these remove a path, so a second failure in the same area takes data offline.
• Failed disks after that - serious, but the array is designed to tolerate them and is already rebuilding.
• Degraded disks and ports last - investigate, but they are not actively endangering data.

Grouping the output this way turns a wall of text into a work queue, and it stops you from replacing a disk while a node is quietly the real problem.

Node Failure: The Highest-Severity Case

A failed node is rated the most critical of the common 3PAR faults, and for good reason: with a node down the array is running with reduced controller capability, and the fix requires real expertise rather than a part swap. If the failed node is a second-hand or secondary-market unit, be especially careful - replacement nodes often need their configuration checked and corrected before they will join the cluster cleanly, and an incorrectly configured node can cause more disruption than the failure it was meant to fix. Confirm the node state, capture the logs, and treat node recovery as an engineering task with a maintenance window, not a hot-swap.

Other Common Issues

• Host port CRC errors: these are external errors from the connected host or upstream switch - rarely a 3PAR problem. Investigate the host identified by the port in the alert, then re-run checkhealth.
• PSU replacement: confirm the fault via the amber LED, power off and unplug the faulty PSU, swap it, power on, and verify with showcage -d cage0.

The CRC example is a useful reminder that not every alert is the array's fault. A host with a marginal HBA, a dirty fibre connector or a mis-seated SFP will generate port errors that the array faithfully reports. Fixing the host is the correct response; replacing array hardware would be wasted effort.

How Nodes, Cages and Loops Connect

It is worth picturing the physical topology, because most cage and port errors only make sense once you do. Disks live in drive cages. Each cage connects to the controller nodes by two separate paths - the left port is the A path and the right port is the B path. The node sees those paths through its own ports, and the array addresses a specific path as node:slot:port:type, which is why an alert can name something like 1:2:1 of type disk. Because every cage has two paths, no single cable or port failure should disconnect a disk. When both paths to a cage are lost you get a loop-offline condition, and the array treats that seriously because redundancy is gone.

When you see a cage or port alert, work from the outside in: confirm which path is affected, check the amber LED on the node port and the cage port to find the failed end, then verify the cable between them. A surprisingly large share of "cage" faults turn out to be a cable that was disturbed during a neighbouring maintenance task, or a connector that was never fully seated. Only after the path is proven good should you suspect the port or the cage hardware itself.

Worked Scenarios: From Alert to Action

Three examples show how the pieces fit together:

• Forty-plus degraded-disk alerts at once: the instinct is a batch of failed drives, but the real cause is usually a single failed node or a lost path that makes every disk behind it appear degraded. Run showed -failed -degraded and look for a shared node or port; fix the shared component and the disk alerts clear together.
• A single disk reports missing but is still seated: check the A/B port for that disk. A missing port is a cage or node signalling problem rather than a dead disk, so reseating the drive will not help - you need to investigate the path.
• Host port CRC errors with healthy array components: this is an external fault. Identify the host and port named in the alert, inspect the HBA, cable and SFP on that host, and only re-run checkhealth once the host side is clean.

Each of these follows the same discipline: run checkhealth, read the alert with the correct vocabulary, and look for the shared cause before you start replacing parts. The replacement is the easy part - finding the right component to replace is where the time is saved.

Preventive Maintenance and Monitoring

Most of these faults are cheaper to prevent than to fix. A short recurring routine catches the early warning signs:

• Schedule a regular checkhealth rather than relying on alerts alone - a component that is trending towards failure usually shows up in checkhealth before it raises an alarm.
• Keep the alert email routing current so the right person sees a node failure at 2 a.m., not a disk replacement the next morning.
• Track spare-space consumption; a shrinking spare pool means every future disk failure is more dangerous.
• Keep firmware consistent across nodes and cages, and record every part you replace for warranty and failure-trend analysis.

Collecting Support Data Before You Escalate

When a fault exceeds what you can fix, the quality of your support case determines how fast it resolves. Before you escalate, gather the checkhealth output, the specific alert strings with timestamps, the output of showed -failed -degraded, showcage -d and showport, and a note of any recent changes - a firmware update, a cable move or a component replacement. A case that arrives with this data already attached lets support confirm a known issue rather than asking you for a file it will need anyway.

Severity Ranking

On a 1-3 scale (1 = severe): a failed node is the most critical because it needs real expertise (and secondary-market nodes need careful configuration); a failed cage is mid-level; a failed disk is the most straightforward to replace. For more storage content, see our HPE 3PAR 8400 configuration principles, the NetApp ONTAP network best practices, Dell PowerStore best practices, and HPE 3PAR Primera Peer Persistence failover.

原文链接：https://mglobalservices.com/3par-error-codes-troubleshooting-guide/