RoCEv2 Lossless Ethernet: PFC and ECN Configuration Guide

RDMA over Converged Ethernet (RoCEv2) turns ordinary Ethernet switches into an RDMA transport, but only if you make the fabric lossless - otherwise a single dropped packet collapses throughput on every RDMA queue and training jobs stall. This guide explains PFC, ECN and the priority mapping that ties them together, shows working configurations on both an NVIDIA/Cumulus style switch and a standard Cisco/NX-OS platform, and lists the counters that tell you whether the fabric is stable before your GPUs find out.

Why RoCE needs lossless treatment

RoCEv1 rides directly on Ethernet; RoCEv2 carries the InfiniBand transport over UDP/IP so it can be routed. Both use a credit-based sender/receiver model with go-back-N retransmission, which means a dropped packet is expensive. The industry answer is a two-part mechanism: PFC (802.1Qbb) pauses a specific priority class hop by hop when buffers fill, and ECN marks packets instead of dropping them so the endpoints slow down gracefully. PFC alone causes head-of-line blocking and pause storms; ECN alone cannot cope with microbursts. Use both.

Pick a priority and mark consistently end to end

Whatever you choose, the value must be identical on the NIC, the access switch, the spine and the far-end host. Renumbering one tier is how fabrics end up "mostly lossless".

NVIDIA/Cumulus style configuration

cumulus@switch:~$ net add roce lossless
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

! per-interface and detailed QoS objects
cumulus@switch:~$ net add qos roce
cumulus@switch:~$ sudo mlnx_qos -i swp1 --trust dscp
cumulus@switch:~$ sudo mlnx_qos -i swp1

net add roce lossless configures PFC on the RoCE priority and ECN/DCTCP on the switch in one shot, which is exactly why NVIDIA documents it as the fast path. Because it manipulates several objects at once, commit it alone - mixing it with other pending configuration is a documented way to produce a partially applied lossless setup.

Standard NX-OS style configuration

switch(config)# class-map type qos match-all ROCE-CLASS
switch(config-cmap-qos)# match cos 3
switch(config)# policy-map type qos ROCE-TRUST-DSCP
switch(config-pmap-qos)# class class-default
switch(config-pmap-c-qos)# set dscp 26
switch(config)# system qos
switch(config-sys-qos)# service-policy type qos input ROCE-TRUST-DSCP

switch(config)# class-map type queuing ROCE-Q
switch(config-cmap-que)# match qos-group 3
switch(config)# policy-map type queuing ROCE-QOS
switch(config-pmap-que)# class ROCE-Q
switch(config-pmap-c-que)# priority level 1
switch(config-pmap-c-que)# pause buffer-size 64000 pause-threshold 32000 resume-threshold 16000
switch(config-pmap-c-que)# random-detect minimum-threshold 1 mbytes maximum-threshold 2 mbytes
switch(config-pmap-c-que)# hardware qos wred threshold 20000
switch(config-if)# priority-flow-control mode on
switch(config-if)# priority-flow-control watch-dog interval 100

Tune the pause thresholds against the port speed: too low and you never pause, too high and one aggressive sender can starve the queue. Always enable the PFC watchdog - a misconfigured or wedged peer link that keeps asserting pause will otherwise silently drop a whole port class out of the fabric.

Host-side settings that matter as much as the switch

# set the RoCE priority mapping (Mellanox/NVIDIA NIC)
sudo mlnx_qos -i ens1f0 --trust dscp
sudo cma_roce_mode -d mlx5_0 -p 1 -m 2          # RoCEv2

# ECN/DCTCP via sysctl
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_ecn_fallback = 0

# lossless counters
ethtool -S ens1f0 | grep -Ei 'pause|prio|pfc'
mlnx_qos -i ens1f0
perfquery -x -C mlx5_0 1

If the host does not trust DSCP, the switch's careful markings are overwritten with zero and RoCE traffic lands in the best-effort queue. perfquery and ethtool -S are your ground truth: rising prioX_pause or rx_pause_duration counters mean the fabric is congested even while throughput looks fine.

Bring-up checklist

Verify MTU end to end (9000 on every hop including the NIC), verify the RoCE priority mapping on both ends, verify PFC is negotiated on the link (show interface priority-flow-control / lldp PFC TLVs), then measure: run ib_write_bw or perftest between two GPUs and watch for retransmissions. Only after that should you plug in the training job. Further reading on this site: spine-leaf design, ToR switch selection, PAM4 vs NRZ optics and Linux bonding modes for host-side redundancy.

原文链接：https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-43/Network-Solutions/RDMA-over-Converged-Ethernet-RoCE