Diagnosing HPE 3PAR Storage Issues in a VMware Environment

VMware environments backed by HPE 3PAR storage are highly available on paper, but the integration surface is large: storage latency, SAN connectivity, multipathing, thin provisioning, and even Windows domain services all interact. When a VM slows down or an ESXi host drops a datastore, the fault can live in any one of those layers. This guide provides the structured, layer-by-layer troubleshooting sequence used to isolate and fix 3PAR-related problems in a VMware environment.

The important mindset is that almost every "3PAR problem" reported by a virtualization team turns out to be a problem in one of four places: the guest itself (queue depth, drivers, anti-virus or backup I/O storms), the ESXi host (HBA, path state, SATP/PSP configuration), the fabric (zoning, ISL congestion, iSCSI network), or the array (physical disk degradation, node failover, controller firmware). Working the layers in a fixed order stops you from chasing symptoms on the wrong side of the SAN. Change one variable at a time, record what you changed, and only then move on — a troubleshooting session that changes the path policy, the firmware and the queue depth simultaneously teaches you nothing about the root cause.

Initial Troubleshooting Steps

1. Gather logs and alerts from vCenter/ESXi, the 3PAR, and Windows Event Viewer.
2. Check health of all components — ESXi hosts, 3PAR nodes, domain servers — via vSphere Client, 3PAR CLI and Server Manager.
3. Validate firmware and software compatibility — outdated firmware on either side is a frequent root cause of odd storage behavior.

Capture a baseline before you touch anything: the time the problem started, which VMs and datastores are affected, whether the symptom is latency, packet loss or a complete outage, and whether it correlates with a backup window, a patch cycle or a storage maintenance event. This timeline is what turns a vague complaint into a reproducible fault.

Map the Data Path Before You Change Anything

A VMware write travels through a long chain, and each hop can add latency:

1. Guest OS filesystem → virtual SCSI controller (PVSCSI or LSI Logic SAS)
2. VMFS datastore on a given LUN → ESXi storage stack and the NMP (Native Multipathing Plugin)
3. Path Selection Plugin (PSP) → a specific HBA port or software iSCSI initiator
4. Fabric: FC switch zoning or iSCSI Ethernet segments
5. 3PAR host port → ASIC → node owning the LUN → CPG → physical disks

Write this chain down with real values (HBA WWPN, 3PAR host port WWN, LUN, CPG) for one affected VM. When you know the exact path, showhost, showport, showvlun and showpath output stops being noise and starts being evidence.

Storage Latency: Find the Affected LUN

Use vCenter performance metrics to identify VMs with high latency and note their LUNs. On the 3PAR (Management Console or CLI), review read/write latency, IOPS and throughput per LUN and correlate spikes. If latency is chronically high, review thin provisioning and deduplication settings — both save space but add overhead under load; schedule heavy storage processes like dedupe off-peak.

Distinguish read latency from write latency, because they point at different sub-systems. High read latency usually means cache miss and disk seeking; high write latency often means queue saturation, a busy remote-copy link, or a backend that has run out of free chunklets in the CPG. Also check whether the elevated latency is array-wide or isolated to a single host — array-wide points to the array, single-host points to a path, driver or queue-depth issue.

Reading 3PAR Performance Counters from the CLI

Performance data is available without the GUI. statvlun shows per-VLUN latency and IOPS, statport shows front-end port load, and statcpu/statpd show node and physical disk pressure. A practical starting point:

# Per-LUN read/write latency and IOPS (5-second sample, 12 iterations)
statvlun -vvsum -iter 12

# Front-end port throughput and queue depth
statport -port -iter 12

# Node CPU busy percentage
statcpu -iter 12

# Physical disk utilisation — find the hot spindle or SSD
statpd -p -iter 12

# Host-level view: which initiator is generating the load
stathost -iter 12

# Is a remote copy link adding write latency?
statrcopy -iter 12

Compare the front-end port utilisation against the back-end disk utilisation. If ports run at 80–90% while disks idle, you have a fabric or host-port bottleneck. If disks sit at 100% while ports are quiet, the CPG layout, RAID level or drive count is the constraint, and the fix is adding capacity or rebalancing with tunesys or showbalance.

Connectivity: iSCSI / FC / Multipathing

• iSCSI: verify IP configurations on both 3PAR and ESXi; FC: verify zoning and masking on the switches.
• Multipathing: confirm each ESXi host has multiple paths to 3PAR and the correct path selection policy (e.g. Round Robin).
• HBA/network adapter health: run diagnostics — faulty adapters cause intermittent connectivity that is easy to misread as storage failure.

For FC, a single-zone-per-host design with the correct WWPNs beats a big "all-in-one" zone: it keeps RSCN storms small and makes path loss unambiguous. For iSCSI, keep the initiator and target on separate subnets or VLANs from general traffic, enable jumbo frames end to end only if every switch port in the path supports it, and verify that the 3PAR iSCSI ports and the ESXi VMkernel ports are on the same VLAN with MTU matched. A single mis-set MTU on one switch port produces exactly the symptom profile of a failing array: intermittent, high-latency, sometimes successful I/O.

ESXi-Side Checks: Adapters, Paths and Queues

Most path failures are visible from the ESXi shell within seconds:

# List HBAs and their state
esxcli storage core adapter list

# Show all paths to all devices, including dead ones
esxcli storage core path list

# Check NMP device state and SATP/PSP for one naa ID
esxcli storage nmp device list -d naa.60002ac0000000000000012300000123

# Set Round Robin with a sane IOPS limit
esxcli storage nmp device set -d naa.60002ac... -P VMW_PSP_RR
esxcli storage nmp psp roundrobin deviceconfig set -d naa.60002ac... -t iops -I 1

# Rescan when a new LUN is masked
esxcli storage core adapter rescan --all

Set the Round Robin IOPS limit rather than leaving it at the default 1000 for flash-backed 3PAR volumes: the default sends a large number of I/Os down one path before switching, which hides a dead path and skews performance. A limit of 1 to 10 is typical for all-flash. Also confirm the queue depth: if a VM has many outstanding I/Os and the guest HBA queue is throttled, the VM sees the latency while the array reports idle disks.

Firmware, Drivers and the Compatibility Matrix

Before escalating to HPE, cross-check both sides against the current support matrix: 3PAR Inform OS version, HBA firmware and driver on each ESXi host, ESXi build, and vCenter version. Interoperability gaps produce symptoms that look like hardware faults — paths that flap, datastores that go APD (All Paths Down) and recover, or VMs that stun under load. A driver that is newer than what the array's host persona supports can also cause the array to reject the initiator entirely, which shows up as a host in a "missing" or "degraded" state rather than as an outage.

Thin Provisioning and Space Reclamation

Monitor free capacity in 3PAR — over-provisioning can exhaust physical space and crash VMs. When VMs are deleted or migrated, space may not be reclaimed automatically: use UNMAP in VMware to reclaim space and keep the array optimized.

Enable the VAAI UNMAP plugin, but schedule the reclaim pass outside production hours because it is I/O intensive:

# Enable VAAI primitives (if disabled)
esxcli system settings advanced set -o /VMFS3/HardwareAcceleratedLocking -i 1
esxcli system settings advanced set -o /VMFS3/EnableBlockDelete -i 1

# Manually reclaim dead space on one datastore
esxcli storage vmfs unmap -l datastore-name

# Check array-side free chunklets and CPG growth
showspace
showcpg -sdg

Set a capacity alert on every CPG — 70% used is a sensible warning line, 80% a hard trigger — so thin provisioning never becomes an outage. Also watch the "compaction" ratio: if a CPG was provisioned to 100% of physical space and dedupe savings shrink, the array can run out of space even though no VM grew.

Priority Optimization and Host I/O Limits

When a single noisy workload (a backup job, a replica seed, a VDI storm) is degrading everything else, the answer is not more cache — it is a limit. 3PAR Priority Optimization lets you cap IOPS or bandwidth per host, per VV set or per VM volume, guaranteeing latency for the tier-1 workloads. Create a VV set for the noisy workload, assign a QOS rule with a max IOPS target, and confirm with showqos. On the VMware side, if you cannot use array QOS, the equivalent blunt instrument is a per-device I/O limit via a storage policy, but array-side QOS reacts to actual back-end pressure and is the better tool.

Domain Services and DNS

Slow logins or failed authentication point to AD replication delays or DNS misconfiguration — review security logs and confirm domain controllers are reachable from ESXi. Check DNS settings on ESXi and vCenter, and flush stale DNS caches on hosts and domain controllers.

It is worth separating the two failure modes explicitly. If authentication is slow but storage metrics are normal, the problem is almost certainly domain-side and 3PAR is a red herring. If storage metrics are bad and logins are slow, fix the storage first — domain controllers running on starved datastores produce cascading authentication timeouts that look like an AD fault. A common trap is a 3PAR-hosted domain controller whose VMDK lives on the affected LUN: every storage stall then becomes a login stall, and the AD team and the storage team each blame the other.

When to Escalate: Log Collection Checklist

If the array itself is the suspect, collect the following before opening a case so that HPE support has something to work with: the checkhealth output, an advisories summary, shownode and showpd -s output, an inventory dump, an srhist (service request history) extract, and a captured performance report (statvlun/statpd at 5-second granularity covering the incident window). From VMware, add a vm-support bundle from one affected host and vCenter, plus the exact timestamps. This bundle is the difference between a two-day and a two-week resolution.

Troubleshooting Summary

• Establish the timeline and the blast radius before changing anything.
• Identify the affected LUN, then compare front-end port, back-end disk and host-side latency.
• Validate paths and the path selection policy on every host, not just the reporting one.
• Check firmware/driver interoperability on both sides of the wire.
• Treat thin provisioning as a capacity risk, not a free lunch, and reclaim space with UNMAP.
• Use QOS to protect latency-sensitive workloads from noisy neighbours.
• Remember that domain services and DNS can mimic storage faults — and vice versa.

See HPE 3PAR 8400 配置原则 for 3PAR configuration fundamentals and NetApp ONTAP 端口排障 for the equivalent NetApp troubleshooting path. For the day-to-day array commands referenced above, the HPE 3PAR CLI cheat sheet collects them in one place, and HPE 3PAR error codes and checkhealth covers how to read the alert output.

原文链接：https://vstack.it/troubleshooting-hpe-3par-storage-issues-in-a-vmware-environment/