Juniper MC-LAG ICCP Failure: LACP System ID Behavior on QFX

Juniper's Multichassis Link Aggregation (MC-LAG) relies on the Inter-Chassis Control Protocol (ICCP) to keep two QFX switches behaving like one logical device. When that ICCP session drops, the way each peer treats its LACP system ID decides whether the attached server keeps a stable bundle or suffers a painful re-convergence. This article walks through the official ICCP failure scenarios documented for QFX switches, explains what each one does to the LACP system ID, and shows the Junos statements engineers use to avoid split-brain states in the first place.

ICCP Failure Scenarios and LACP System ID Behavior

Juniper's failure-handling table for QFX series summarises what happens to a multichassis aggregated Ethernet (MC-AE) interface whose status is set to standby. The three inputs are the ICCP connection state, the ICL (inter-chassis link) state and the backup liveness peer status:

• ICCP down, backup liveness not configured — the LACP system ID is changed to the default value, so the downstream device keeps only the links of the first peer that comes up.
• ICCP down, backup liveness Active — the LACP system ID is changed to the default value for both the active and the standby MC-AE interfaces. Both peers still believe the other side is alive, so neither can safely take over.
• ICCP down, backup liveness Inactive — no change is made to the LACP system ID. Liveness has confirmed the remote peer is really gone, so the surviving switch can keep the configured system ID and the server bundle stays up.
• ICCP up but ICL down — the LACP state is set to standby and the MUX state moves to waiting, preventing loops over a broken inter-chassis link.

The default system ID change is the switch's safety mechanism

Read the table as a decision tree rather than a list. The first question is whether ICCP is up; if it is, the ICL state decides what the data plane does. If ICCP is down, the only thing that determines whether the server keeps a working bundle is whether liveness has confirmed the peer is really gone. Every other behaviour in the matrix is a fallback that exists to prevent loops, and each one costs you something at the server.

: when a true split-brain exists, forcing both sides back to the default LACP system ID makes the server select a single peer's links instead of forwarding through both. See our related breakdown of MC-LAG ICCP, BFD and LACP system ID behavior for the protocol fundamentals.

What the server side actually experiences

The LACP system ID is not just a label in a show command; it is the identifier a server uses to decide that a set of physical links belongs to one logical aggregator. An LACP-speaking server - a Linux bond in 802.3ad mode, a VMware vSwitch with LACP teaming, a Windows SET or LBFO team - keys its aggregator on the system ID and the port key it receives in the LACPDU. Change the system ID and the server concludes it is talking to a different device, tears the aggregator down, and restarts the selection state machine from scratch.

That restart is visible to users. The server's ports move through the LACP states - detached, waiting, collecting, distributing - and no traffic passes until they reach collecting and distributing. With fast periodic LACP (1 second) that is a couple of seconds of loss; with the default slow periodic timers (30 seconds) the bundle can take considerably longer to settle, and any flow that was riding a link that is no longer selected is broken and has to recover through the transport layer. In a virtualised environment the effect cascades: a vSwitch that loses its uplink lag can take management reachability and vMotion traffic with it.

This is why the "ICCP down, backup liveness Inactive" case matters so much in practice. It is the only combination where nothing changes at the server, because the surviving switch keeps the configured system ID and the server never notices that its second peer disappeared. Compare that with the "not configured" case, where the system ID reverts to the default and the server reselects links - a visible event that shows up in switch logs as a LACP partner change and in a ticket as "the application blipped".

Minimal MC-LAG configuration reference on QFX

The following is the core configuration for one peer of a two-member MC-LAG. The statements that matter for this article are status-control, prefer-status-control-active, the two timers and the ICCP liveness block; everything else is the usual aggregated Ethernet plumbing.

set chassis aggregated-devices ethernet device-count 4
set interfaces xe-0/0/1 ether-options 802.3ad ae0
set interfaces xe-0/0/1 description "to server NIC1"
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp periodic fast
set interfaces ae0 aggregated-ether-options lacp system-id 00:11:22:33:44:55
set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae0 aggregated-ether-options mc-ae mode active-active
set interfaces ae0 aggregated-ether-options mc-ae chassis-id 0
set interfaces ae0 aggregated-ether-options mc-ae status-control active
set interfaces ae0 aggregated-ether-options mc-ae init-delay-time 240
set interfaces ae0 aggregated-ether-options mc-ae icl-down-delay 60
set interfaces ae0 aggregated-ether-options mc-ae prefer-status-control-active
set protocols iccp local-ip 10.10.10.1
set protocols iccp peer 10.10.10.2
set protocols iccp peer 10.10.10.2 liveness-detection minimum-interval 8000
set protocols iccp peer 10.10.10.2 liveness-detection multiplier 3
set protocols iccp peer 10.10.10.2 session-establishment-hold-time 340
set protocols iccp backup-liveness-detection backup-peer-ip 10.10.10.2

The second switch uses the same statements with three values swapped: local-ip 10.10.10.2, peer 10.10.10.1 and chassis-id 1. The LACP system ID is deliberately identical on both peers - that is the whole point of MC-LAG - and it is exactly that shared value that has to be protected when ICCP fails, which is what the status control statements are for.

Set periodic fast unless you have a reason not to. One-second LACPDUs cost almost nothing and cut the time the server spends in the collecting and distributing states after any change, which directly reduces the user-visible impact of the failure scenarios above.

Why Backup Liveness Detection Matters

ICCP runs over TCP, so a plain link failure between the peers can take the session down while both switches are still powered and forwarding. Without extra state, both MC-LAG peers may stay active and send duplicate traffic to the connected server — the "packets loop on the server" failure mode documented in Juniper's MC-LAG troubleshooting guide. Backup liveness detection gives each peer an out-of-band heartbeat over the management network so a peer can distinguish "ICCP lost" from "peer is dead".

set protocols iccp backup-liveness-detection backup-peer-ip 10.10.10.2
set protocols iccp peer 10.10.10.2 liveness-detection minimum-interval 8000
set protocols iccp peer 10.10.10.2 session-establishment-hold-time 340

Best practice notes from Juniper: configure the liveness interval to at least 8 seconds when ICCP runs over an IRB interface so graceful Routing Engine switchover (GRES) works seamlessly, and add the master-only statement to the management IP used for liveness on both routing engines so the heartbeat survives a GRES. Our article on ICCP backup liveness configuration covers the full setup.

Verifying ICCP and MC-AE State

When a failure is suspected, start with the operational commands that expose each layer of the MC-LAG control plane:

user@qfx> show iccp
user@qfx> show iccp detail
user@qfx> show mc-ae
user@qfx> show lacp interfaces ae0 detail
user@qfx> show interfaces ae0 extensive | match "LACP|MC-AE"

show iccp reports the TCP connection state between the peers, the BFD liveness state and the client applications (MCSNOOPD, LACPD, ESWD). show mc-ae shows which role each MC-AE is currently playing. For deeper failure hunting, see our standby and split-brain scenario guide.

Timers, BFD and the convergence budget

Three timers decide how each failure behaves, and they interact, so it is worth setting them with a purpose rather than leaving the defaults.

init-delay-time controls how long a rebooting peer that is configured as status-control active waits before claiming its active role. It exists to stop the standby peer from flapping to active and back while its partner comes up, but the cost is straightforward: for the duration of the delay, neither peer considers itself active and traffic through the MC-AE can be interrupted. Juniper's guidance for QFX is 240 seconds for the init delay with a session-establishment hold time of 340 seconds, which keeps the hold time comfortably more than 100 seconds above the init delay so that the ICCP session does not time out while the peer is still waiting.

icl-down-delay is the other side of the same problem: it defines how long the chassis waits before treating a lost inter-chassis link as down, giving the ICL a chance to recover from a transient failure without triggering a role change. Prefer this timer over broad hold-time adjustments when the status-control-active peer reboots, because it targets the specific event that causes momentary traffic loss.

Finally, liveness detection ties it all together. ICCP runs over TCP, so a peer that is powered but unreachable looks exactly like a peer that has failed. Configure BFD-based liveness on a management path, keep the interval at 8 seconds or longer when ICCP runs over an IRB interface so that graceful Routing Engine switchover still works, and add master-only to the management address used for the heartbeat on both routing engines. With those three settings you have a system that can tell "I lost the ICCP session" apart from "my peer is gone", which is the distinction every row of the failure table depends on.

Design Rules That Prevent the Failure Modes

• Run ICCP over the peer loopback addresses instead of a direct link, so a single cable failure does not drop the session.
• Use prefer-status-control-active on the active peer so the LACP system ID does not revert to default during an ICCP failure.
• Prefer the icl-down-delay timer over hold-time when the status-control-active peer reboots, to avoid momentary traffic loss on the standby peer.
• Keep the session-establishment hold time (default 300 s) at least 100 s above the init delay time — Juniper suggests 340 s hold with 240 s init delay on QFX.
• Enable backup liveness detection on both peers; configure it on a dedicated out-of-band management path whenever possible.

Troubleshooting checklist

Work down this list in order when an MC-LAG incident lands on your desk. Each step answers a question the next one assumes.

show iccp
show iccp detail
show mc-ae
show mc-ae status
show lacp interfaces ae0 detail
show interfaces ae0 extensive | match "LACP|MC-AE"
show log messages | match "ICCP|mc-ae|lacp"

Start with show iccp detail to establish whether the TCP session between the peers is up, whether liveness is being reported and which client applications have registered. Then use show mc-ae to see which peer currently believes it is active and whether the MC-AE is in a standby state. Only after both of those do you look at show lacp interfaces ae0 detail, which is where you will see the system ID actually being advertised and the MUX state of each member link.

If the switches look healthy and the server does not, the next useful step is on the server rather than the switch: check the bond or team state, the partner system ID it has learned, and its own log for a partner-change event. A surprising number of MC-LAG incidents are servers that never converged after an unrelated maintenance event and are simply waiting for the LACP partner they remember. Confirm the management path used for liveness is actually usable - the right VRF, no ACL blocking it, and a routing engine that is primary on the correct RE after a failover. Configuration changes should be the last resort, not the first, because the matrix above shows that most of these states are working as designed rather than broken.

Understanding exactly which LACP system ID behavior follows each ICCP failure state is what separates a clean MC-LAG failover from a server-side LACP storm. Verify the state with the commands above before touching any configuration. For the traffic-plane side of the same failure - what happens to ARP, PIM and forwarding across a broken ICL - see MC-LAG ICCP failure and ARP/PIM behaviour, and for the timer side of the same design, the liveness-interval guidance in the configuration sections above is the companion read.

原文链接：https://www.juniper.net/documentation/us/en/software/junos/mc-lag/topics/concept/best-practices-usage-notes.html