NVMe-oF: Choosing Between TCP and RoCEv2

NVMe over Fabrics is transport agnostic by design, which sounds like a gift until you have to pick one. In practice the choice is between TCP, which runs on the Ethernet you already own, and RoCEv2, which delivers RDMA latency but demands a lossless fabric and RDMA-capable adapters. The decision is less about raw latency and more about what your switch fleet, NICs and operations team can actually sustain. This article lays out the measured differences, the configuration each transport demands, and the questions to answer before committing.

The two transports at a glance

NVMe/RoCEv2 encapsulates RDMA over UDP/IP, so it is routable, but UDP provides no flow control. RoCEv2 therefore expects a lossless Ethernet domain, achieved with Priority Flow Control (PFC) and often complemented by Explicit Congestion Notification and DCQCN. It also requires RDMA capable NICs to offload the transport, keeping CPU cost near zero and the latency in the single-digit microsecond range adapter-to-adapter.

NVMe/TCP was added in the NVMe-oF 1.1 specification and uses ordinary TCP sockets. It runs on any Ethernet adapter with no special switch configuration or lossless domain. The cost is paid on the host: the TCP stack consumes CPU per gigabit, and total latency is higher because data is copied through the stack unless the adapter offers TCP offload.

Measured differences worth quoting in a design review

• Two transports on identical hardware: roughly 11–34 percent lower latency with NVMe/FC class transports, and up to 58 percent higher IOPS in one vendor audit — the audit explicitly notes it was not a max-performance test.
• One vendor benchmark reports RoCE about 49 percent lower read latency and 43 percent lower write latency than TCP for sequential I/O, and 38/30 percent for random 4K.
• CPU scaling tells the more practical story: saturating the link took roughly 4–8 cores with RDMA versus 16–24 cores with TCP in independent testing on the same class of server.

Those CPU numbers are the ones that change architectures. On a hyperconverged node where the CPUs are already sold to tenant workloads, spending 16 cores on storage transport is a direct capacity tax.

What each transport needs configured

RoCEv2 — the fabric side

# Switch side (Cumulus/NVIDIA-style Linux switch example)
# Enable PFC on the storage priority, typically 802.1p 3
lldptool -T -i swp1 -V PFC willing=1 enabled=3
# ECN for DCQCN marking
tc qdisc add dev swp1 root handle 1: htb default 0
# Host side
cma_roce_mode -d mlx5_0 -p 1 -m 2      # RoCEv2
cma_roce_tos -d mlx5_0 -t 106          # DSCP 26

PFC makes the priority lossless; without it RoCEv2 will retransmit and the latency advantage erodes. PFC also creates a new failure mode — a stalled priority can propagate head-of-line blocking backwards — which is why watchdog and PFC storm detection matter as much as the PFC configuration itself.

NVMe/TCP — the host side

modprobe nvmet-tcp
# Target
mkdir /sys/kernel/config/nvmet/subsystems/nqn.2026-01.io.example:ssd1
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.2026-01.io.example:ssd1/attr_allow_any_host
mkdir /sys/kernel/config/nvmet/subsystems/nqn.2026-01.io.example:ssd1/namespaces/1
echo -n /dev/nvme0n1 > /sys/kernel/config/nvmet/subsystems/nqn.2026-01.io.example:ssd1/namespaces/1/device_path
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.2026-01.io.example:ssd1/namespaces/1/enable
mkdir /sys/kernel/config/nvmet/ports/1
echo 10.10.20.1 > /sys/kernel/config/nvmet/ports/1/addr_traddr
echo 4420 > /sys/kernel/config/nvmet/ports/1/addr_trsvcid
echo ipv4 > /sys/kernel/config/nvmet/ports/1/addr_trtype
ln -s /sys/kernel/config/nvmet/subsystems/nqn.2026-01.io.example:ssd1 \
      /sys/kernel/config/nvmet/ports/1/subsystems/ssd1

# Initiator
nvme discover -t tcp -a 10.10.20.1 -s 4420
nvme connect -t tcp -a 10.10.20.1 -s 4420 -n nqn.2026-01.io.example:ssd1

Note the difference in operational surface: NVMe/TCP needs no switch change at all, only host kernel modules and addressing. That is the whole argument for TCP in brownfield environments.

A decision framework

• Greenfield rack, RDMA NICs standard, switch team can run PFC/ECN with monitoring — RoCEv2 gives the lowest latency and frees host CPU.
• Existing standard Ethernet, mixed NIC vendors, no appetite for lossless fabric operations — NVMe/TCP, and budget CPU accordingly.
• Cross-rack or data-centre-wide reach — TCP, or RoCE confined to a lossless domain. RoCEv2 outside a controlled lossless island is a latency lottery.
• CPU-constrained nodes — RDMA or TCP with hardware offload; software TCP on a busy hypervisor node is the worst combination.

Both transports also benefit from the same diligence on multipathing and MTU. Multi-path configuration and path checking follow SAN conventions described in Fibre Channel vs iSCSI vs NVMe-oF compared, and the MTU and fragmentation issues across underlay and overlay are the same ones discussed in VXLAN MTU fragmentation and jumbo frames. Get those wrong and the transport comparison becomes academic.

原文链接：https://www.snia.org/sites/default/files/education/snia-optimizing-nvme-over-fabrics-nvme-of.pdf