RHEL DM-Multipath Troubleshooting: Paths and Queueing

DM-Multipath exists so that a failure at any point in the I/O path — HBA, Fibre Channel cable, SAN switch, array controller port — causes the host to switch to an alternate path instead of losing the LUN. When it works, nobody notices. When it half-works, the symptom is usually an application hung in uninterruptible sleep rather than a clean error, which is why the two most common failures — a stopped daemon and the queue_if_no_path feature — deserve to be first in any troubleshooting routine.

Start with the daemon, always

multipathd must be running for multipathed devices to work at all. A configuration that is perfect in multipath.conf produces nothing if the daemon is stopped after a maintenance reboot:

systemctl status multipathd
systemctl enable --now multipathd
systemctl reload multipathd.service

Confirm the module is present with lsmod | grep dm_multipath and that the package itself is installed (rpm -q device-mapper-multipath). A surprising number of "multipath stopped working" tickets end here.

Reading multipath -ll correctly

multipath -ll

Sample output structure:

mpatha (3600d0230000000000e13954ed5f89300) dm-4 WINSYS,SF2372
size=233G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 6:0:0:0 sdf 8:80 active ready running

Read it in three blocks. The header line gives the alias and WWID, the dm device name, vendor and product, size, features, hardware handler and write permissions. The path-group line gives the path-selector policy and priority. The path line gives host:channel:target:lun, the kernel device, its priority, the path status and the path state.

The distinction that matters: active ready running is healthy. active faulty or failed means this path has been dropped — if every path for a map shows faulty while the storage is up, suspect zoning, an array port or the HBA before suspecting multipath itself. If a path drops and comes back repeatedly, that is a physical or SAN-side flapping issue that multipath is faithfully reporting.

The queue_if_no_path trap

When a map carries features='1 queue_if_no_path' — the default for most arrays — any process issuing I/O to that device blocks until at least one path returns. That is deliberate: it prevents a filesystem from being marked read-only during a transient SAN blip. It also means that if all paths are genuinely gone, applications hang in D state and cannot be killed.

# release waiting I/O for one map, or for every map
multipathd disablequeueing map <device>
multipathd disablequeueing maps
# restore the previous behaviour
multipathd restorequeueing map <device>

Queueing stays disabled until multipathd is restarted or reloaded. For a permanent change, set no_path_retry in multipath.conf instead of disabling at runtime:

defaults {
    no_path_retry           queue   # or a number of retries, or "fail"
    polling_interval        5
}
devices {
    device {
        vendor   "NETAPP"
        product  "LUN.*"
        no_path_retry  10
        path_checker    tur
    }
}

no_path_retry 10 retries ten times and then fails I/O — the choice between "hang forever" and "fail cleanly" is a workload decision, and it should be made deliberately per array class rather than left at the default.

Troubleshooting with the interactive console

multipathd -k
> help
> show maps
> show paths
> show path group
> show config
> reconfigure

multipathd -k gives a live console into the running daemon; reconfigure re-reads multipath.conf without a restart. This is the fastest way to test a config change before committing it to a service reload on a production host.

Blacklisting and the find_multipaths mistake

The other frequent cause of trouble is over-eager creation: multipath tries to build devices for local disks and for every pseudo-device in the system. As of RHEL 6 and later, with find_multipaths yes, DM-Multipath creates a device only when a defined condition is met — a valid WWID, a matching udev property, or multiple paths to the same device — which removes most of the noise. When you do need an explicit exclusion, prefer the WWID:

blacklist {
    wwid 3600508b4000156d700012000000b0000
}
blacklist_exceptions {
    wwid 3600d0230000000000e13954ed5f89300
}

Check which protocol each path uses before blaming the policy: multipathd show paths format "%d %P" prints the device and protocol (fcp, iscsi, sas, nvme and so on), which tells you whether you are chasing a fabric issue or an iSCSI session issue.

A five-minute routine for a hung application

1. systemctl status multipathd — is the daemon alive?
2. multipath -ll — do the affected maps have any active ready running path?
3. If all paths are faulty, go to the SAN/array before touching the host: zoning, array ports, link state.
4. If paths are healthy but processes are stuck, check features='1 queue_if_no_path' and use disablequeueing to release I/O.
5. Fix the underlying path, then multipathd reconfigure and re-check path status, not just map status.

For the storage-side view of the same infrastructure see HPE 3PAR thin provisioning and space reporting; for replication lag on file and block workloads see SnapMirror lag troubleshooting.

原文链接：https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_device_mapper_multipath/troubleshooting-dm-multipath