Linux Bonding 802.3ad: LACP Rate and Hash Policy

Mode 4 (802.3ad) is the most requested and most misunderstood Linux bonding mode. It is not a magic throughput doubler: it negotiates a LACP aggregator with the switch, and the real behaviour - which flows use which member, how fast a failure is detected, and whether a single conversation can be spread - is decided by a handful of options that most guides leave at their defaults. This article covers the options that matter, how to verify them from /proc, and the switch-side settings they depend on.

The Options That Decide Behaviour

• lacp_rate: slow (default, LACPDUs every 30 s) or fast (every 1 s). Fast matters when you want sub-second failure detection from the negotiation itself.
• xmit_hash_policy: layer2 (default, MAC XOR, 802.3ad-compliant), layer3+4 (uses IP and ports), layer2+3 (MAC plus IP). Higher layers give better spread across many flows but can break a single flow across members, which reordering-sensitive applications dislike.
• miimon: link monitoring interval in milliseconds. 100 ms is the usual starting point.
• arp_interval / arp_validate: ARP-based monitoring that detects failures the driver's carrier signal misses (for example a switch that keeps the link up but stops forwarding).
• fail_over_mac: irrelevant for 802.3ad, essential for active-backup on switches that lock MAC addresses.

Configuration Examples

# Debian/Ubuntu via ifupdown
auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-mode 802.3ad
        bond-miimon 100
        bond-lacp-rate fast
        bond-xmit-hash-policy layer3+4

# NetworkManager
nmcli con add type bond ifname bond0 mode 802.3ad miimon 100 lacp_rate fast \
  xmit_hash_policy layer3+4
nmcli con add type ethernet ifname eno1 master bond0
nmcli con add type ethernet ifname eno2 master bond0

Runtime tuning exists too, via /sys/class/net/bond0/bonding/, but a value written there is lost on reboot - always mirror it in the persistent configuration.

Verification

cat /proc/net/bonding/bond0
ip -brief link show bond0
ethtool bond0
ethtool -S eno1 | grep -Ei 'pause|lacp|link'
ip -s link show bond0

In /proc/net/bonding/bond0 confirm: bonding mode 802.3ad, both slaves with MII status up, "Aggregator ID" identical on both slaves, and a partner MAC address that is not all zeros. A zero partner MAC means no LACP partner was negotiated - the switch LAG is missing or in the wrong mode.

Switch-side Requirements

1. The switch ports must be in one LACP/802.3ad aggregation, not a static EtherChannel or a set of independent ports.
2. Both ends must agree on speed and duplex, and on the LAG's hashing expectations; a switch that only supports layer2 hashing pairs badly with layer3+4 on the host.
3. If the two member NICs go to different switches, those switches need MLAG/VPC/VSX - a plain LACP LAG cannot span chassis.
4. VLAN tagging must be consistent: tag on the bond (bond0.100) and bridge it, rather than tagging the slave NICs.

Troubleshooting Order

1. Is the bond up and are both slaves "up"? If not, it is cabling or the switch port state.
2. Is the Aggregator ID identical on both slaves? If not, LACP is only half negotiated.
3. Is the partner MAC non-zero? If zero, fix the switch side before touching bonding options.
4. Is throughput still single-link? Check flow distribution - a single flow will not be split by design with layer2 hashing.

Related reading: Proxmox VE networking with VLAN-aware bridges and LACP, ethtool for Linux network diagnostics and Linux multipath (multipath.conf) for iSCSI SAN.

How the transmit hash decides which slave carries a flow

Once LACP has formed an aggregator, the kernel picks a slave for every packet by hashing selected header fields. The policy therefore determines everything about how traffic spreads, and the three policies behave very differently:

• layer2 (default): XOR of the source and destination MAC addresses. This is the 802.3ad-compliant setting and the only one guaranteed to be understood by every switch. It spreads well between routers and badly between two hosts, because a conversation between two MAC addresses always hashes to the same member.
• layer2+3: MAC plus source and destination IP. A good compromise when traffic is routed but you cannot afford the reordering that port-based hashing occasionally causes.
• layer3+4: IP plus source and destination ports. Best spread for a server talking to many clients, but a single connection can be split across members, so packets can arrive out of order. Most hypervisors and databases tolerate it; some latency-sensitive appliances do not.
• encap2+3 and encap3+4: use the encapsulated headers of VXLAN and similar overlays rather than the outer ones, which is essential when the transport is a tunnel and every outer header looks identical.

The policy is pointless on the host alone. If the switch uses a different hash, return traffic is spread differently than forward traffic, and with any stateful middlebox in the path this shows up as asymmetric routing rather than lost throughput. Agree the policy on both ends before you tune anything else.

# check what the kernel is really using
cat /proc/net/bonding/bond0 | grep -i 'hash\|lacp rate\|MII Status'
cat /sys/class/net/bond0/bonding/xmit_hash_policy
cat /sys/class/net/bond0/bonding/lacp_rate

lacp_rate, the switch timers and what fast actually buys you

lacp_rate controls how often the host sends LACPDUs: slow sends one every 30 seconds (the 802.3ad default), fast sends one every second. Fast gives sub-second detection from the negotiation itself, which matters on top-of-rack links where you want reconvergence without waiting for the 90-second slow timer to expire. It is not free: every member sends 30 times as many control frames, which is irrelevant on a 25G link and not irrelevant on a large chassis with thousands of ports.

Critically, the host and the switch must both be configured for fast. If only one end requests a 1-second periodic rate, LACP simply keeps the slow timer, and engineers spend an afternoon wondering why failover still takes 30 seconds. Verify from both ends:

# Linux
cat /proc/net/bonding/bond0 | grep -i 'LACP rate'
# Arista EOS
show lacp neighbor detail
show lacp interface Ethernet1
# Cisco NX-OS
show lacp interface Ethernet1/1
show port-channel summary

Monitoring deeper than carrier: arp_interval and arp_validate

miimon asks the driver whether the link is up. That question is answered by the PHY, so it will happily report "up" on a link whose switch is in a broken state, whose VLAN is misconfigured, or whose upstream port is err-disabled. ARP monitoring catches those cases because it tests reachability rather than carrier.

# ARP-based monitoring with per-slave validation
bond-arp-interval 1000
bond-arp-ip-target 10.0.0.1
bond-arp-ip-target 10.0.0.2
bond-arp-validate active      # only the active slave is validated
# alternatives: all, backup, rarp, filter

cat /proc/net/bonding/bond0 | grep -i 'ARP'   # confirm polling is running
ethtool -S eno1 | grep -Ei 'rx_missed|rx_no_buffer|rx_errors'

Use two or three arp_ip_target values so a single unreachable neighbour does not declare the whole bond dead, and prefer arp_validate active unless you specifically need the others. Combining miimon 100 with ARP monitoring costs almost nothing and closes the most common false-positive failure mode.

VLANs, bridges and where to put the tag

Tag on the bond, never on the slaves. A tagged sub-interface on a member NIC creates a configuration that the aggregator cannot consistently represent, and the result is flapping traffic or a slave that silently never joins the LAG.

# correct: 802.1Q on the bond, then bridge the tagged interface
ip link add bond0 type bond mode 802.3ad miimon 100 lacp_rate fast xmit_hash_policy layer3+4
ip link set eno1 down && ip link set eno1 master bond0 && ip link set eno1 up
ip link set eno2 down && ip link set eno2 master bond0 && ip link set eno2 up
ip link add link bond0 name bond0.100 type vlan id 100
ip link add br100 type bridge
ip link set bond0.100 master br100
ip link set br100 up

# verify
bridge link show
bridge vlan show
ip -d link show bond0 | grep -i 'mode\|miimon'

When the bond carries a bridge for virtual machines, the host also needs the bridge's own forward delay and STP settings to be considered: a bridge with STP enabled will block the second member path, which is correct for a looped topology and wrong for a single LACP LAG where the aggregator already handles redundancy. The Proxmox-specific end-to-end example, including the VLAN-aware bridge, is in Proxmox VE with VLAN-aware bridges and LACP.

Measuring which member a flow takes

# per-slave counters: a healthy LAG shows traffic on both members
ip -s link show eno1 | tail -4
ip -s link show eno2 | tail -4
ethtool -S eno1 | grep -E 'tx_packets|rx_packets'

# prove the distribution with several parallel flows
iperf3 -c 10.0.0.20 -t 20 -P 8
# then, for a single flow, watch which slave counts up
iperf3 -c 10.0.0.20 -t 20
watch -n1 'cat /proc/net/bonding/bond0 | grep -A1 "Slave Interface"'

Eight parallel flows should show traffic on both members; a single flow will land on exactly one member, which is not a fault but the documented behaviour of every hash policy in the kernel. If a single flow that gets close to line rate is the requirement, LACP is the wrong tool and you need a routed design with equal-cost multipath or a flow-based load balancer.

What throughput to expect, and what to tell the application team

An LACP bond does not multiply the speed of a single conversation, and saying so before the change avoids an argument afterwards. With any hash policy in the Linux bonding driver, one flow hashes to one member and is limited to that member's line rate. Eight parallel flows spread across two 10G members give close to 20G aggregate; one SCP copy over the same bond gives close to 10G. That is expected behaviour, not a misconfiguration.

• Aggregate capacity: the sum of the members, available to many independent flows.
• Single-flow capacity: one member's line rate, regardless of policy.
• Redundancy: sub-second failover with lacp_rate fast and miimon 100, or up to 30 seconds with the default timers.
• Asymmetry risk: host-side and switch-side hash policies must match, or forward and return traffic take different members.

If a single flow genuinely has to exceed one member, the answer is a routed design with equal-cost multipath so the transport layer can spread a single connection across paths, or a flow-based load balancer in front of the bond. Both are larger changes than a bonding option, and both should be chosen before the hardware is racked rather than after the first performance complaint.

Failure patterns and what they mean

1. Partner MAC all zeros: no LACP partner was negotiated. The switch port is in the wrong mode, in a static EtherChannel, or administratively down. Fix the switch before touching any bonding option.
2. Aggregator ID differs between slaves: LACP is only half negotiated, usually because one member's VLAN membership or speed differs. Identical aggregator IDs on every slave is the single best health check.
3. Both slaves up, no throughput gain: check the hash policy on the host and the switch together. Mismatched policies explain this far more often than a broken bond.
4. Members to different switches with no MLAG: a plain LACP LAG cannot span chassis. If the design needs that, the peer switches must form MLAG, VPC or VSX, and the comparison of those designs is in EVPN multihoming versus MLAG and ESI DF election.
5. Link flap on one member takes down the bond: LACPDUs are being lost. Check for a duplex mismatch, an mtu mismatch, or a switch port with a rate limit that drops control frames.

For the switch-side reference configuration and the equivalent Junos LAG syntax, see the Junos aggregated Ethernet LACP example. When the LAG carries block storage, the multipath layer above it has its own requirements, documented in Linux multipath configuration for iSCSI SAN.

原文链接：Linux Ethernet Bonding Driver HOWTO