SMB Multichannel and SMB Direct RDMA Tuning

File servers are usually judged by one benchmark and one user complaint: a large copy that runs at 3 Gbit/s on a 10 GbE link, and an application that becomes unusable while that copy runs. Both are often the same root cause — a single TCP connection and a single CPU core doing all the work. SMB Multichannel and SMB Direct fix it, but only if the adapters, the RSS configuration and the switch fabric all cooperate. This article covers the requirements, the configuration and the verification counters that prove the change worked.

What Each Feature Actually Does

• SMB Multichannel makes the SMB client open multiple TCP connections for a single session, using all available adapters or all RSS queues of one adapter. Two effects: more aggregate bandwidth, and per-CPU distribution of processing, which removes the single-core bottleneck on small-I/O workloads.
• SMB Direct (enabled by Multichannel) detects RDMA-capable adapters and creates multiple RDMA connections for the session — two per interface — giving high throughput, low latency and very low CPU utilisation.
• Important interaction: disabling SMB Multichannel also disables SMB Direct. They are not independent knobs.

Requirements are minimal in principle: at least two machines running Azure Local, Windows Server or Windows 10+, and at least one of: multiple network adapters, adapters that support RSS, a NIC team, or an RDMA-capable adapter.

Step 1 — Check What the Server Has

# Adapter capabilities
Get-NetAdapter | ft Name,InterfaceDescription,LinkSpeed,Status
Get-NetAdapterRss -Name * | ft Name,Enabled,NumberOfReceiveQueues
Get-NetAdapterRdma -Name * | ft Name,Enabled,OperationalState

# What SMB thinks it can do
Get-SmbServerConfiguration | Select EnableMultiChannel
Get-SmbClientConfiguration  | Select EnableMultiChannel,EnableBandwidthThrottling
Get-SmbMultichannelConnection

If Get-NetAdapterRdma shows OperationalState: Not Ready, the RDMA stack is not bound — usually an adapter driver/firmware item, not an OS one. Fix that before touching SMB settings; SMB Direct cannot compensate for a missing NDKPI provider.

Step 2 — Configure the Server

# Enable (default) or re-enable Multichannel - required after it has been disabled
Set-SmbServerConfiguration -EnableMultiChannel $true
Set-SmbClientConfiguration -EnableMultiChannel $true

# Manually configure a fast server for high-throughput file service
Set-SmbServerConfiguration -AsynchronousCredits 2048 -AutoDisconnectTimeout 15

# Per-role tuning: file server workload
Set-SmbServerConfiguration -Smb2CreditsMax 8192 -Smb2CreditsMin 512
Set-SmbServerConfiguration -MaxWorkItems 8192 -MaxThreadsPerQueue 20

Two settings cause most enterprise interruptions. AsynchronousCredits and Smb2CreditsMax raised too far on a server with many concurrent clients will increase memory pressure and can look like a memory leak. Change one value at a time, and only after a baseline with Get-SmbServerConfiguration saved to file.

Step 3 — Constrain Channels When You Must

Multichannel discovers every usable path between client and server, which on a multi-homed host can mean traffic over a backup or management VLAN — usually not what you want.

# Restrict which interfaces SMB may use on the client
Set-SmbClientConfiguration -EnableMultiChannel $true

# Per-interface RDMA control
Disable-NetAdapterRdma -Name "Ethernet 2"          # stop RDMA on one adapter
Set-NetOffloadGlobalSetting -NetworkDirect Disabled # kill RDMA globally

# Force a client to a specific adapter/IP for a share (per-share path preference)
New-SmbMapping -LocalPath Z: -RemotePath \fs01\data -UserName corp\user

# Inspect the chosen connections - the fastest way to see what is in use
Get-SmbMultichannelConnection | ft ServerName,ClientIPAddress,ServerIPAddress,ClientLinkSpeed,ServerLinkSpeed

If a client shows only one connection to a server with two 10 GbE links, the two usual causes are different subnets with no route between them, or RSS queues configured as 1. Verify the fabric first: Multichannel does not invent connectivity that the network does not provide.

Step 4 — Verify the Speedup

# Which connections exist, and what are they carrying
Get-SmbMultichannelConnection -IncludeNotSelected
Get-SmbMultichannelConnection -IncludeNotSelected | ft ClientIPAddress,Selected,ClientLinkSpeed,ServerLinkSpeed

# Per-connection counters, including RDMA
Get-SmbMultichannelConnection -IncludeNotSelected | fl *
Get-SmbClientNetworkInterface | ft InterfaceIndex,IPv4Address,RssCapable,RdmaCapable,LinkSpeed

# Throughput proof
# On the server: create a 20 GB test file
fsutil file createnew C:	estGB.bin 21474836480
# On the client (Measure-Command + copy), then inspect the share counters
Get-SmbServerShare -Name data

A successful RDMA configuration typically shows near line-rate throughput with single-digit CPU utilisation on both ends. A Multichannel-only configuration shows similar throughput but visibly higher CPU — if CPU is still pinned, RSS is not spreading load and you should check Get-NetAdapterRss per adapter.

Failure Modes and the Usual Suspects

• RDMA "Not Ready". Driver/firmware mismatch, or the adapter is not connected to a switch that supports the required flow control (PFC/ECN for RoCEv2).
• Instability after enabling RoCEv2. The switch needs PFC on the correct priority and ECN marking; without it, RDMA retransmits look like packet loss and throughput collapses — see our RoCEv2 lossless Ethernet notes.
• Multichannel creates paths over the wrong network. Constrain adapters or fix routing; SMB will use whatever the OS considers reachable.
• Performance worse after "tuning". Roll back credits and MaxWorkItems first. The defaults are tuned for general workloads and are usually within a few percent of optimal.

Checklist Before Declaring Victory

1. Get-NetAdapterRdma reports Enabled/Ready on both ends.
2. Get-SmbMultichannelConnection shows the expected number of connections on the intended subnets.
3. Throughput measured after the change, with CPU utilisation recorded alongside.
4. A deliberate adapter-down test proves the session survives on the remaining path.
5. Settings documented, because the next admin will find Smb2CreditsMax non-default and want to know why.

Related reading: RoCEv2 PFC and ECN configuration, iSCSI LUN and MPIO for ESXi and Unraid vs TrueNAS comparison.

原文链接：https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/manage-smb-multichannel