FC vs iSCSI vs NVMe-oF: Choosing a Storage Network Protocol

Block storage for virtualization clusters and databases travels over one of three fabrics - dedicated Fibre Channel, Ethernet-based iSCSI, or the newer NVMe-over-Fabrics family - and the choice shapes both your latency budget and your day-2 operations for years. The common trap is treating the decision as "FC is old, NVMe is new, so NVMe wins": each protocol optimizes a different constraint. This article compares FC, iSCSI and NVMe-oF (including NVMe/TCP, NVMe/RoCE and FC-NVMe) so the transport matches the workload.

Protocols at a Glance

iSCSI: The Pragmatic Workhorse

iSCSI wraps SCSI in TCP/IP, so the entire Ethernet toolkit applies: standard switches, jumbo frames, multipathing, existing monitoring. It is the right answer for small/medium virtualization, VM file services, backups and test environments - and it can serve databases if traffic is isolated on a dedicated VLAN or network. The costs are CPU per I/O (TCP processing and SCSI translation) and latency that depends heavily on network tuning; vendor tests consistently show iSCSI delivering the lowest IOPS and highest host CPU of the common transports.

Fibre Channel: Isolation as a Feature

FC is a purpose-built lossless SAN with its own switches, HBAs and terminology (WWPN, zoning, fabric). Because nothing else shares the network, latency is predictable and operational surprises are fewer in large, critical environments - finance, healthcare, dense VDI. The price is a separate infrastructure to buy and skills to keep; FC's bandwidth roadmap (32GFC mainstream, 64GFC arriving) also trails Ethernet.

NVMe-oF: Native Flash Semantics over a Network

NVMe-oF keeps the NVMe command set end to end, avoiding the SCSI translation every iSCSI or FCP I/O pays. The three transports answer different questions:

• NVMe/TCP - lowest barrier: standard Ethernet and NICs, routable, troubleshot with familiar tools. Measurably better IOPS and CPU than iSCSI on the same network.
• NVMe/RoCE (RoCEv2) - RDMA moves data between host and target memory directly; lowest Ethernet latency, but requires RDMA NICs and a lossless fabric tuned with PFC and ECN. Published tests put 25G RoCE near 32GFC on IOPS and latency.
• FC-NVMe - NVMe over existing FC infrastructure; deterministic and low-risk for shops that already run FC well, letting them adopt NVMe semantics without ripping out the SAN.

Where the Latency Actually Comes From

An I/O between a host and an array crosses a fixed chain of components: the application's I/O stack, the operating system's block layer, the protocol driver, the HBA or NIC, the fabric, and the array's front-end ports and cache. The protocol you choose changes only the middle of that chain, but the middle is exactly where microseconds are won or lost. iSCSI adds a full TCP/IP send-and-receive path plus a SCSI command translation at both ends. Fibre Channel and FC-NVMe skip IP entirely and run over a credit-based fabric that never buffers overflow, so it does not drop frames by design. NVMe/TCP keeps the TCP stack but removes the SCSI translation. NVMe/RoCE removes both by moving data directly between host and target memory with RDMA. When a vendor quotes “30 µs for RoCE versus 100 µs for iSCSI”, that gap describes the software stack, not the flash media underneath it.

Host CPU and Queue Depth: The Hidden Cost

Every protocol consumes host CPU per I/O, and the cost is not equal. Software iSCSI burns the most cycles because the CPU runs the TCP stack and translates SCSI commands. NVMe/TCP is measurably lighter on the same network because the command set is native NVMe. Fibre Channel uses an HBA that offloads the protocol, and NVMe/RoCE offloads to the RDMA NIC, so both free the host CPU almost entirely. That difference becomes the real bottleneck in dense virtualization hosts: a server that could sustain 200,000 IOPS on FC may top out near half that on iSCSI purely on CPU, before the fabric is stressed at all. Queue depth matters for the same reason — NVMe supports far deeper queues than SCSI (thousands of commands per queue versus a few hundred), so a single NVMe-oF connection can keep a modern all-flash array busy without provisioning hundreds of LUNs.

Lossless Ethernet: What RoCE Actually Demands

RoCEv2 runs over ordinary Ethernet but assumes the network never drops a packet, because RDMA retransmission is expensive. To get there you must configure Priority Flow Control (PFC) to pause one traffic class, and Explicit Congestion Notification (ECN) so switches mark rather than drop as queues build. Both endpoints and every switch in the path must agree on the DSCP-to-priority mapping. Get it half right and the symptom is not an obvious outage — it is a handful of flows that stall for milliseconds under load, which users report as “random” application latency. This is why NVMe/RoCE demands more network engineering than NVMe/TCP: you are not buying a faster NIC, you are committing to run a lossless fabric correctly.

Multipathing and Resilience on Every Fabric

Whichever transport you pick, the host must present more than one path to the array, or a single HBA, cable or switch failure takes the LUN with it. On iSCSI and NVMe/TCP this is the Linux or ESXi multipath layer: two NICs, two subnets, two switches, and a policy that spreads or fails over as required. On Fibre Channel it is MPIO over two fabrics (A and B), each with its own switches and zoning. NVMe-oF adds native multipath inside the protocol, so a single namespace can be reached over several controllers — but the underlying discipline is identical. Plan the same way for all three: two independent paths, no shared cable or switch, and periodic failover testing so the second path is proven to work.

Bandwidth, Oversubscription and the Fabric Budget

Latency gets the attention, bandwidth decides how many workloads fit. Fibre Channel's roadmap moved through 16GFC, 32GFC and 64GFC, while Ethernet jumped to 25G, 100G and 400G in the same period, so for raw port bandwidth Ethernet has pulled ahead. In practice you rarely need to saturate a port; you need enough headroom that bursts do not queue. Size the fabric to at least 2:1 oversubscription from array ports to host ports for general virtualization and closer to 1:1 for latency-sensitive databases, and remember that iSCSI and NVMe/TCP will consume CPU before they consume link bandwidth. Fibre Channel's deterministic credit system makes its throughput more predictable under load, which is part of why it persists in environments that refuse to chase every Ethernet tuning knob.

Migration Paths Between Protocols

Most environments do not rip and replace; they add. A common, low-risk progression starts on iSCSI with a dedicated VLAN and multipathing, moves the most latency-sensitive workloads to NVMe/TCP once the array supports it and the hosts have CPU headroom, and keeps Fibre Channel where it already works well. Shops with an existing FC fabric can adopt NVMe semantics through FC-NVMe without new cabling. Migrate per workload, keep the old path configured as a fallback during the transition, and validate with real I/O — not a synthetic benchmark — before cutting over. Moving the data (LUN copy, array-based replication, or host-side copy) is often a longer project than the protocol change itself.

Operational Cost and Skills

The purchase price is the smallest number in the total cost of ownership. Fibre Channel concentrates cost in switches, HBAs and specialized skills; iSCSI and NVMe/TCP concentrate it in network tuning and the risk of a shared, noisy fabric if isolation is sloppy; NVMe/RoCE concentrates it in lossless-fabric engineering. Whichever you choose, the day-2 tasks rhyme — zoning or ACLs, multipath management, firmware, monitoring — but the tooling and the failure modes differ, and those determine how quickly your team recovers at 02:00. Pick the fabric your team can operate, not the one with the best spec sheet.

Common Mistakes When Choosing

• Choosing by generation instead of workload. “Newer is better” ignores that a medium VMware cluster is perfectly served by well-built iSCSI.
• Running storage over the production data VLAN. iSCSI and NVMe/TCP belong on a dedicated segment or a separate physical path.
• Enabling RoCE without a lossless design. PFC and ECN configured on one switch and not the rest guarantees intermittent stalls.
• Forgetting host CPU. A software initiator can run out of CPU long before the fabric or the array is busy.
• Single-pathing “temporarily”. Temporary tends to become permanent until the day the one path fails.

How to Decide

• Fast start on familiar Ethernet, cost-sensitive, medium scale - start with iSCSI (segmented, redundant, validated), and treat NVMe/TCP as the natural next step.
• Maximum predictability, isolation and zero-tolerance downtime - Fibre Channel (or FC-NVMe) if you already operate FC.
• Very fast all-flash arrays with latency-critical workloads (high-load databases, analytics, AI) - NVMe-oF, picking the transport by your team's ability to run lossless Ethernet (RoCE) or by existing FC practice (FC-NVMe).
• Most shops run a mix: NVMe/TCP for general workloads, RoCE islands for the latency-critical tier, FC-NVMe for the legacy SAN - matching transport to workload rather than standardizing on one fabric.

Troubleshooting Cheat Sheet by Transport

The commands you reach for differ on every fabric, and knowing them in advance is what turns a storage incident into a five-minute fix. On the host the goal is always the same: confirm the session or login exists, confirm the paths are healthy, then confirm the array sees the initiator.

# iSCSI (Linux)
iscsiadm -m session -P 3          # sessions, target portals, negotiated parameters
iscsiadm -m node                  # discovered targets and their state
multipath -ll                     # paths per LUN and their status

# NVMe-oF (Linux)
nvme list-subsys                  # controllers, transports, live paths
nvme list                         # namespaces presented to the host

# Fibre Channel (Linux)
systool -c fc_host -v             # port state, speed, WWPN
cat /sys/class/fc_host/host*/port_state

On the switch side the questions are just as specific: for iSCSI and NVMe/TCP, whether the VLAN or VRF is actually carrying storage traffic and whether any interface is dropping frames; for NVMe/RoCE, whether PFC and ECN counters are incrementing, because a pause or mark counter that climbs under load is the fingerprint of a lossless design that is not quite right; and on an FC fabric, the port state and error counters of both the initiator and the array ports. VMware shops see the same picture through esxcli storage core path list and esxcli iscsi session list. Whatever the transport, if the host shows no path, look at the fabric before the array — and if the host shows a path but the array does not show the initiator, suspect zoning or VLAN isolation rather than host drivers.

Security, Isolation and Encryption

Block storage carries your most sensitive data across a network, and each transport answers “who may see this LUN” differently. iSCSI has CHAP authentication and target-side access control; without them, anyone on the storage VLAN who discovers the target can attempt a login. NVMe/TCP provides an equivalent in-band authentication option, and both depend on segmentation more than on any credential: a dedicated VLAN or VRF that no user traffic touches is the real boundary. Fibre Channel uses zoning plus host mapping on the array, so a LUN is visible only to the WWPNs that both the zone and the array’s mapping permit — a stronger default, because the fabric itself enforces it. For data in flight, arrays and HBAs increasingly offer encryption terminating at the array port, and IP transports can be wrapped in IPsec where regulation demands it, at a measurable cost in latency and CPU. Decide which of those requirements is real before choosing a transport, because retrofitting encryption into a busy iSCSI path is a project, not a setting.

Benchmarking Honestly Before You Cut Over

Synthetic numbers from a slide deck do not predict your workload. Before migrating a tier, measure the storage you have and the storage you are proposing with the same tool, the same queue depth and the same access pattern:

# random read: the pattern most databases care about
fio --name=randread --ioengine=libaio --direct=1 --rw=randread \
    --bs=4k --iodepth=32 --numjobs=4 --time_based --runtime=300 \
    --group_reporting --filename=/dev/mapper/mpathX

# sequential write: closer to backup and log workloads
fio --name=seqwrite --ioengine=libaio --direct=1 --rw=write \
    --bs=1M --iodepth=8 --numjobs=1 --time_based --runtime=300 \
    --group_reporting --filename=/dev/mapper/mpathX

Three rules keep the results honest. Test through the multipath device rather than a single path, or you are benchmarking one HBA. Watch host CPU alongside IOPS and latency — a run pinned at 100 percent CPU on the initiator has found the real limit, which is the protocol stack, not the array. And repeat the test with one path failed, so the surviving path is proven. A tier that meets its latency target on one path and collapses on the other has not been validated at all.

Whichever protocol you pick, the fabric side needs the same discipline: two independent paths, redundant switches and cabling. See our Brocade FC zoning CLI guide for FC fabrics, HPE 3PAR administration commands for array-side operations, and the NetApp ONTAP NFS export policy guide when the workload prefers file (NFS) semantics over block. For a hands-on walkthrough of the NVMe/TCP transport covered above, see our NVMe over TCP target and initiator setup guide.

原文链接：https://gse.kz/en/blog/storage-network-for-san-iscsi-fc-nvme-of