MC-LAG Explained: ICCP, BFD and LACP System ID Deep Dive

MC-LAG (Multi-Chassis Link Aggregation) is the Juniper answer to the single-active-routing-engine problem: instead of one primary RE with all PFEs connected to it, two devices run two active routing engines and present a single LAG to the downstream server. The magic happens in three mechanisms — ICCP for control-plane coordination, BFD for fast failure detection, and a shared LACP system ID so the server sees one logical bundle. This article explains each pillar in depth, based on the Packet-Expert MC-LAG design write-up, and adds the configuration, timer arithmetic and verification commands that matter once you build it for real.

The Problem MC-LAG Solves

High availability features like GRES, NSR and NSB still leave one RE active at a time; when the primary RE fails, all PFEs must reconnect to the backup, causing momentary disruption. MC-LAG offers two active REs in two different chassis, eliminating the switchover disruption entirely.

Look at the alternatives a dual-homed server actually has. Connect both NICs to one switch and you have bundled bandwidth with LACP, but the switch is a single point of failure and a software upgrade costs an outage. Connect one NIC to each of two independent switches and 802.3ad stops working: LACP requires both links to report the same partner system ID, so each switch is treated as a separate aggregator, one link is marked individual, and spanning tree blocks the redundant path. The result is active/standby uplinks, half the usable bandwidth, and a reconvergence that is measured in seconds.

MC-LAG breaks that constraint by making two chassis look like one LACP partner. The server runs ordinary LACP, receives the same system ID on both member links, and bundles them; both chassis forward traffic. There is no spanning tree blocking, no RE switchover, and no single point of failure below the server.

Pillar 1: ICCP (Inter-Chassis Control Protocol)

ICCP is the TCP-based protocol MC-LAG peers use to exchange control information and coordinate forwarding state. It replicates control traffic and forwarding states across peers and communicates the operational state of MC-LAG members. Key parameters:

• ICCP peer IP — the address used to establish the ICCP session (loopback lo0 recommended).
• session-establishment-hold-time — 50 seconds recommended for faster ICCP establishment.
• redundancy-group-id-list — must be identical on both peers and referenced in the MC-AE configuration.
• liveness-detection minimum-interval / multiplier — BFD timers that detect peer failure; e.g. 60 ms × 3 = 180 ms detection.

A representative Junos configuration looks like this — the peer address is a loopback, the redundancy group ties ICCP to the MC-AE, and BFD liveness is enabled on the same peer:

set protocols iccp local-ip 10.0.0.1
set protocols iccp session-establishment-hold-time 50
set protocols iccp peer 10.0.0.2 redundancy-group-id-list 1
set protocols iccp peer 10.0.0.2 liveness-detection minimum-interval 60
set protocols iccp peer 10.0.0.2 liveness-detection multiplier 3

ICCP is not carried on a magic out-of-band link: it needs IP reachability between the two routing engines, which is why the peer is normally reached over the dedicated Inter-Chassis Link (ICL) plus loopbacks, or through the core if the ICL is down. The ICL itself must be a plain link or a static aggregate Ethernet — never an MC-AE and never an interface that depends on ICCP being up, or you build a circular dependency in which the session that should protect the link can only be established over the link it protects.

Because ICCP rides over TCP, its own failure detection is slow: a silent peer can leave the TCP session established from the local perspective while the remote node is forwarding nothing. That is the entire reason BFD exists in the design.

Pillar 2: BFD Liveness

Because ICCP rides over TCP, it needs fast failure detection: BFD liveness detection with a 60 ms minimum interval and multiplier 3 gives 180 ms convergence — fast enough to fail over before the server's own LACP timers expire.

The arithmetic is worth spelling out because it drives your choice of values. BFD negotiation ends up with the slower of the two peers' requested transmit intervals multiplied by the multiplier, so 60 ms × 3 means the session is declared down after roughly 180 ms without a valid packet. Compare that with LACP fast timers (30 s timeout) and with the ICCP/ICL hold timers, which run in the tens of seconds — a peer failure detected by TCP alone can be two orders of magnitude slower. Pairing BFD with ICCP is what turns MC-LAG from "eventually converges" into a sub-second failover.

Two practical constraints apply to the timer values. First, both peers must be able to sustain the packet rate: 60 ms means roughly 16 packets per second per session in each direction, which is trivial for a modern RE but not free if you run many MC-AEs with aggressive timers. Second, a single lossy ICL will cause the session to flap, and every flap is a potential dual-active window — so do not tune below what the physical path can carry reliably. Juniper's own design guidance lands on 60 ms × 3 for MC-LAG, and that is a sound default.

For split-brain protection you can add backup-liveness-detection on the ICCP peer. The backup session probes the peer's alternative path; if primary ICCP is up but the backup probe fails, the peer can safely take over instead of assuming the remote node is healthy and leaving the MC-AE down. This is the knob that prevents the classic "both peers think the other is gone" outcome.

Pillar 3: LACP System ID and MC-AE

The Multichassis Aggregated Ethernet interface (MC-AE) takes one interface from each peer and bundles them; the connected device treats them as a normal LAG. The critical requirement is a matching LACP system ID on both peers — unique per MC-AE but identical across the two peers — so the downstream device considers both links members of the same bundle and load-balances across them.

Mechanically, LACP compares the partner's system ID and key when it decides whether a link may join an aggregator. If peer A advertises its own chassis MAC and peer B advertises a different one, the server computes two different partners, marks the links as belonging to separate aggregators, and — because the system priority and key still collide with an existing aggregator — leaves one link in standby or unselected. Setting an explicit system-id on both MC-AE interfaces, and a distinct chassis-id on each peer, tells the server that two physically separate devices are the same logical partner.

set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-id 00:11:22:33:44:55
set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae0 aggregated-ether-options mc-ae mode active-active
set interfaces ae0 aggregated-ether-options mc-ae status-control active
set interfaces ae0 aggregated-ether-options mc-ae chassis-id 0
set interfaces ae0 aggregated-ether-options mc-ae init-delay-time 180

Notice the division of labour: mc-ae-id is the internal index of this MC-LAG, redundancy-group must match the group advertised over ICCP, chassis-id is different on each peer (0 and 1), and status-control decides how that peer drives the bundle — active in active/active mode, standby in active/standby mode. The init-delay-time keeps the bundle down for a period after boot so both peers and the server agree on the LACP state before traffic starts flowing.

Every MC-AE needs its own unique LACP system ID. Two MC-AEs sharing one system ID create one destination MAC/LAG identity on the downstream side, and LACP will fight over which links belong where. Keep the system IDs documented per MC-AE and duplicate them only across the two peers of the same MC-LAG.

How the Three Pillars Interact

The mechanisms are not independent. ICCP carries the operational state of MC-AE members between peers, BFD decides whether that information can be trusted at all, and the shared LACP system ID is what lets the server keep both links bundled while the peers sort out which one forwards. Fail one pillar and the design degrades in a specific, predictable way: lose the system ID match and you get active/standby; lose BFD and failover latency jumps to seconds; lose ICCP entirely and both peers may try to forward, which is the packet-loop scenario covered in our MC-LAG ICCP failure troubleshooting guide.

Failure Handling

When ICCP fails, the peers cannot coordinate — see our MC-LAG ICCP failure troubleshooting guide for the packet-loop scenario and fixes. Compare vendor implementations in Arista EOS MLAG and MLNX-OS LAG/LACP configuration.

Walk the failure cases deliberately and decide, per case, what you want the bundle to do:

• ICL fails, ICCP fails. The worst case. Each peer now believes the other is gone and keeps its own MC-AE member active, while the server keeps both links bundled. Traffic arriving on one chassis is flooded to the MC-AE on the other, and the ICL — the path that should have absorbed it — is down, so the frames recirculate. This is exactly where BFD timers, backup liveness detection and a well-documented "hold the bundle down on ICCP loss" policy earn their keep.
• ICCP fails, ICL up. The peers can often still exchange enough information over the ICL to avoid dual-active; the forward traffic path is intact. Treat this as a warning state and alert on it, because the next failure has no safety net.
• One MC-AE member link fails. LACP removes that link from the bundle; the partner continues forwarding over the remaining member. Recovery is sub-second with fast timers, and no ICCP action is needed.
• A peer's routing engine crashes. BFD declares the ICCP peer down in milliseconds, the surviving peer takes over forwarding responsibility for the affected MC-AE, and the server keeps its bundle up on the surviving member.
• Server reboots. LACP renegotiates and both members return to the bundle; init-delay-time prevents a race where traffic is forwarded before the fabric is ready.

The design rule that follows from all of this: never let the ICL share fate with the MC-AE member links, never run LACP on the ICL, and always size the ICL for the full MC-AE bandwidth, because a single failed member can push that member's traffic across the chassis interconnect.

Verification Commands

MC-LAG problems are almost always visible in a handful of operational commands. Run these on both peers and compare:

show iccp
show interfaces mc-ae
show interfaces mc-ae redundancy-group 1
show lacp interfaces ae0
show lacp interfaces ae0 extensive
show bfd session
show configuration protocols iccp
show log messages | match ICCP

show iccp confirms the session state, redundancy group and BFD status. show interfaces mc-ae shows the MC-AE role, peer state, ICL status and whether the local interface is active or standby. show lacp interfaces proves the server-facing side is bundled with the expected partner system ID, and show bfd session confirms the negotiated intervals. If the MC-AE is up on one chassis and standby on the other when you expect active/active, compare the LACP system IDs first, then the redundancy group IDs.

Design Guidelines and Common Pitfalls

• Use loopbacks for ICCP peer addresses and make sure they are reachable over the ICL, not only through the core.
• Keep redundancy-group-id-list on the ICCP peer and redundancy-group on the MC-AE identical; a mismatch keeps the bundle down with no obvious error.
• Give each peer a different chassis-id and the same lacp system-id. Getting this backwards is the single most common MC-LAG misconfiguration.
• Set the same mc-ae-id and the same AE unit number on both peers; use interface descriptions that name the peer.
• Enable BFD liveness on ICCP (60 ms × 3 is a good starting point) and consider backup liveness detection for split-brain protection.
• Dedicate at least two physical links per MC-AE member where the budget allows, and keep the ICL out of any MC-AE.
• Verify with show lacp interfaces ae0 extensive that the partner's system ID is the configured one rather than the chassis MAC — this catches a missing system-id statement immediately.
• Update both peers in the same maintenance window; a version skew that changes LACP or ICCP behaviour can silently break the bundle.

For the timer-by-timer view of the same design, see MC-LAG design: ICCP liveness, ICL and hold timers.

原文链接：https://packet-expert.org/2016/09/15/multi-chassis-lag/