GPUDirect RDMA and NCCL Tuning for GPU Clusters

In a multi-node training cluster the network is no longer a supporting system — it is part of the compute path. GPUDirect RDMA lets the NIC read and write GPU memory directly, removing the host-memory copy from every collective operation, and NCCL is the library that orchestrates those collectives. When training throughput is below expectation, the fault is often not the fabric at all: it is a PCIe topology that never enabled peer-to-peer, or a performance test that was never run at the right size. This article covers the enabling steps and the validation that separates a working cluster from a fast one.

What GPUDirect RDMA actually removes

Without it, a cross-node exchange is: GPU memory → host memory → NIC → wire. With GPUDirect RDMA, the NIC accesses GPU memory over PCIe directly. The result is lower latency, fewer CPU cycles consumed, and higher effective bandwidth at small message sizes — the sizes that dominate gradient-synchronisation traffic.

Prerequisites that are usually the real fault

• PCIe ACS settings — GPUDirect implementations require specific ACS (Access Control Services) bits so that peer-to-peer routes work. On Grace Blackwell systems with CX8 Data Direct Interfaces the required ACS configuration is documented per bridge type (GPU-attached bridges, data-direct-interface bridges, Grace root ports), with specific bits to enable and disable. Incorrect ACS silently disables P2P, and the cluster still works — just slower.
• Correct GPU↔NIC affinity — the NIC a GPU uses must be on the same PCIe switch or NUMA domain. Use the vendor topology tool (rdma_topo on Grace Blackwell platforms) to list the valid GPU, NIC and Data Direct Interface combinations rather than assuming the ordering.
• NIC firmware and driver — GPUDirect requires a matching generation of MLNX OFED driver and firmware on both the GPU and NIC sides.

Validating the fabric before blaming the model

The OSU micro-benchmarks with CUDA buffers are the standard test for GPUDirect RDMA, and HPC-X ships a prebuilt copy:

source <path/to/hpcx>/hpcx-init.sh
hpcx_load
export LD_LIBRARY_PATH=<cuda/install/path>/lib64:$LD_LIBRARY_PATH

# device-to-device bandwidth
$MPI_HOME/bin/mpirun -np 2 --hostfile hosts ./osu_bw D D

Expected device-to-device bandwidth rises steeply and then flattens: single-digit MB/s at tiny sizes, tens of GB/s once the transfer is large enough to amortise latency. If the curve flattens far below your link rate — for example around 24 GB/s on a 400G fabric that should reach roughly 45 GB/s — the bottleneck is the PCIe/NVLink path or the P2P configuration, not the switch.

NCCL: measure first, tune second

NCCL performs topology detection and internal tuning automatically, selecting parameters based on GPU count, NVLink domain size, HCA speed and PCIe topology. NVIDIA's guidance is explicit that users generally do not need to set NCCL environment variables to reach peak performance. The productive order of operations is:

1. Run nccl-tests (all_reduce_perf, reduce_scatter_perf, all_gather_perf, alltoall_perf) from 1 node up to the full cluster, recording bandwidth at each size.
2. Compare against the fabric's theoretical bandwidth, remembering that bus bandwidth in the test output is not the same as link rate.
3. Only then look at environment variables — and treat any unset variable as a suspicious change.

#!/bin/bash
export NCCL_DEBUG=WARN
TEST_DIR=./build
for TEST in all_reduce_perf reduce_scatter_perf all_gather_perf alltoall_perf; do
  srun --nnodes ${NODES} --ntasks ${GPUS} --ntasks-per-node ${GPUS_PER_NODE}        --cpu-bind=none --mpi=pmix        ${TEST_DIR}/${TEST} ${TEST_ARGS} 2>&1 | tee ${LOGDIR}/LOG_${TEST}_N${NODES}n${GPUS}.txt
done

A sweep that halves the node count on each iteration is the most informative single run you can do: it shows whether bandwidth degrades linearly (expected) or collapses (a topology or routing problem).

Diagnostic checklist

• Training slower than the fabric allows — run nccl-tests at the job's real scale, not the smallest one available.
• Small-message collectives slow, large transfers fine — latency-bound: check PCIe topology and P2P, not switch buffers.
• Bandwidth halves when a node is added — an ECMP or rail-alignment problem; some flows are transiting a shared spine while others stay local.
• NCCL falls back to TCP — the RDMA device was not visible to the process: check container device passthrough, NCCL_IB_DISABLE, and that the NIC used by the job is the one with GPUDirect configured.
• Performance fine on the bench, bad in production — other tenants sharing the fabric; enable ECN/PFC-aware congestion monitoring on the leaf uplinks and look for queue drops during the training window.

See also: InfiniBand vs RoCE vs Ethernet for AI clusters, RoCEv2 lossless tuning with PFC and ECN, and throughput testing methodology with iperf3.

原文链接：https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/nccl.html