MC-LAG ICCP Failure: LACP System ID Behavior on Junos

MC-LAG (Multi-Chassis Link Aggregation) on Junos OS keeps two QFX or EX switches acting as one logical switch, but the whole design depends on the ICCP control channel staying healthy. When ICCP goes down, the peers must change behavior deterministically or the downstream server will see two different LACP partners and start flapping links. This article explains the ICCP failure scenarios documented by Juniper, what happens to the LACP system ID in each case, and how backup liveness detection, BFD timers and GRES change the recovery story. It then goes one level deeper than a configuration guide: what actually changes inside the LACPDU on the wire when the system ID flips, how to capture those frames, and how the change propagates to the downstream device.

ICCP Failure Scenarios and the LACP System ID Matrix

Juniper documents a clear action matrix for QFX Series switches. The outcome depends on three inputs: ICCP connection status, ICL (inter-chassis link) status, and whether backup liveness detection reports the peer as Active or Inactive.

• ICCP Down, backup liveness not configured: the LACP system ID is changed to the default value, so each MC-LAG peer stops pretending to be one switch.
• ICCP Down, backup liveness Active: the LACP system ID is changed to the default value for both the active and the standby MC-AE interfaces.
• ICCP Down, backup liveness Inactive: no change to the LACP system ID.
• ICCP Up but ICL Down: LACP state is set to standby and the MUX state machine moves to the waiting state.

The logic behind the matrix is simple: when the peer is confirmed dead (backup liveness Inactive), keeping a shared LACP system ID is safe because only one switch remains. When the peer may still be alive but ICCP is lost, each switch reverts to a unique default LACP system ID so the downstream device brings up only one link instead of forming an inconsistent bundle. For a deeper look at adjacent failure modes such as standby state and split-brain, see our earlier breakdown of MC-LAG ICCP failure scenarios: standby and split-brain.

Reading the three inputs on a live box

All three inputs are visible without packet capture, and reading them in the right order avoids most misdiagnosis. ICCP status comes from the session table, ICL status from the inter-chassis link interfaces, and backup liveness from the management-network heartbeat that only runs while ICCP is down. In practice you want to know two things: is ICCP operationally up, and does the peer still believe there is a live partner? If ICCP is up but the ICL is down, the matrix does not touch the system ID at all — it moves LACP into standby, which is a completely different symptom (traffic is forced to one side rather than the bundle flapping).

show iccp
show iccp peer-state
show interfaces ae0 extensive | match "ICL|LACP|MC-AE"
show lacp interfaces ae0 extensive

Packet-Level Behavior: What Actually Changes in the LACPDU

MC-LAG does not invent a proprietary field on the wire. The illusion that two chassis are a single LACP partner is created entirely with the standard IEEE 802.1AX Actor and Partner information TLVs inside every LACPDU. That is why an ICCP failure has a very specific, very observable packet-level signature: one six-byte field changes, and the downstream switch reacts to it.

Where the system ID lives on the wire

A LACPDU is an Ethernet slow-protocol frame: EtherType 0x8809, slow-protocol subtype 0x01 for LACP, version 0x01. After the 14-byte Ethernet header the layout is fixed and easy to read from a decode window:

• Bytes 14–15: slow-protocol subtype and version (1 and 1).
• Byte 16: TLV type 0x01 (Actor Information), byte 17: TLV length 20.
• Bytes 18–19: Actor System Priority (2 bytes, default 127).
• Bytes 20–25: Actor System ID — the six-byte MAC that defines the LACP partner.
• Bytes 26–27: Actor Key (the aggregation key, i.e. the LAG it wants to join).
• Bytes 28–29: Actor Port Priority, bytes 30–31: Actor Port Number, byte 32: Actor State flags, bytes 33–35: reserved.
• Bytes 36–51: the Partner Information TLV, which repeats the same layout (system priority, system ID, key, port, state) for the remote device.

When ICCP fails and Junos reverts the MC-AE interface to its default LACP system ID, the bytes that change are 18–25 of every LACPDU the peer transmits — the Actor System Priority and Actor System ID of the frame source. Everything else in the frame can stay identical: same aggregation key, same port number, same state flags, same rate. The downstream device compares the Actor System ID in the incoming frame with the Partner System ID it has stored for that port. A mismatch is a hard break of the LACP conversation, and that is precisely what turns a working bundle into a flapping one.

There is a second, equally important detail: the system ID change is emitted continuously, once per second in fast mode or once every 30 seconds in slow mode. A 30-second periodic timer is why you sometimes see the downstream bundle survive an ICCP failure briefly and then collapse. The peer starts sending the wrong identity immediately, but the downstream switch only discovers it when the next LACPDU arrives and its partner state no longer matches.

Capturing and reading the frames

On Junos, the fastest packet-level check is a monitor session filtered on the slow-protocol EtherType. Filtering matters: an unfiltered monitor session on a busy AE interface will drop frames and distort what you see.

monitor traffic interface ae0 no-resolve matching "ether proto 0x8809" write-file /var/tmp/lacp.pcap
show lacp interfaces ae0 extensive
show lacp interfaces ae0 extensive | match "System"
show iccp session

In Wireshark, open the capture and use the lacp display filter, then expand Actor Information and Partner Information for one frame before and one frame after the failure. For a scripted comparison, export the raw header fields instead of the whole decode:

tshark -r /var/tmp/lacp.pcap -Y lacp -T fields -e frame.time -e eth.src -e eth.dst
tshark -r /var/tmp/lacp.pcap -Y "ether proto 0x8809" -V | grep -E "System (Priority|ID)|Port State"

What you should see is unambiguous. In steady state, both MC-LAG peers emit LACPDUs whose Actor System ID is the same shared MC-AE value, so show lacp interfaces ae0 extensive on either peer prints that value as the local system ID. After the failure, each peer prints its own default system ID, and the captures taken on the two peers no longer agree. If you also capture on the downstream server or switch, you will find it receiving two competing Actor System IDs on two links that it believes belong to one bundle.

Failure Timeline at Packet Level

1. T0 — steady state. ICCP up, both peers advertise the shared system ID. The downstream device sees one partner, all member links collecting and distributing.
2. T1 — ICCP session breaks. TCP session down (control link failure, keepalive expiry, or peer control-plane restart). Data plane keeps forwarding; nothing changes in the LACPDUs yet, because the failure action has not fired.
3. T2 — failure action evaluated. Backup liveness decides the branch: liveness Inactive means the peer is dead and the system ID is left alone; Active or not configured means the system ID is reverted to default. Session-establishment-hold-time keeps the session from churning on transient TCP resets during this window.
4. T3 — system ID flips. The next LACPDU from each peer carries the default Actor System ID. From this instant, both peers are lying about being a single chassis — deliberately.
5. T4 — downstream detects the mismatch. The first LACPDU received with the new Actor System ID invalidates the stored partner state. Depending on the downstream LACP timeout (3 seconds in fast/active mode, 90 seconds in slow/passive mode), the bundle either renegotiates immediately or keeps forwarding on the old view until the partner times out.
6. T5 — convergence. One link stays collecting and distributing, the other goes standby or individual. The failure is contained to a single-path outage instead of a permanent flapping loop.

The timeline explains why the same ICCP outage can look like a two-second blip on one network and a five-minute flapping storm on another: it is the interaction between the failure action, the LACP rate configured on the downstream device, and how long the downstream device holds stale partner state.

Backup Liveness Detection Configuration

Backup liveness detection gives the peers an out-of-band channel through the management network that is used only when the ICCP TCP connection is operationally down. The peers exchange liveness requests, and if no response arrives within the configured timeout for a set number of consecutive attempts, the peer is declared down and the failure action above is applied. If backup liveness detection is configured, peer status is always Up when either the ICCP TCP connection is established or BFD is up; the backup liveness check only runs while ICCP is down. That single rule is what prevents a working ICL from being ignored after a TCP reset.

set protocols iccp peer 1 session-establishment-hold-timer 340
set protocols iccp peer 1 liveness-detection backup-liveness-detection
set protocols iccp peer 1 liveness-detection minimum-interval 1000
set protocols iccp peer 1 liveness-detection multiplier 3

Juniper recommends configuring ICCP connectivity over an aggregated interface whose child links are spread across multiple FPCs, and adding the backup liveness channel so a single link or FPC failure cannot silently split the pair. See the dedicated reference on ICCP backup liveness detection for MC-LAG for more detail, and the timer design discussion in Junos BFD liveness detection timers and MC-LAG integration when you need to size minimum-interval and multiplier together.

ICCP over IRB and BFD Timer Guidance

When ICCP is carried over an IRB interface, configure the ICCP liveness-detection interval (the BFD timer) to at least 8 seconds. An interval below 8 seconds can break graceful Routing Engine switchover (GRES): during GRES there is a control-plane gap that a fast BFD timer mistakes for a peer failure, forcing an unnecessary LACP system ID change and traffic disruption. An 8-second or longer interval lets GRES complete seamlessly, at the cost of a slower genuine failure declaration. The trade-off is worth stating plainly: a fast timer buys you a faster detection of a real outage and a higher risk of a false positive during a planned control-plane event.

GRES, ARP and MAC Synchronization Notes

• During GRES, ARP entries learned remotely on the MC-LAG peer are purged and then re-learned; plan for the brief re-population window.
• ARP and MAC tables normally stay synchronized, but link flapping can desynchronize them; Juniper recommends the arp-l2-validate statement on IRB interfaces as a workaround, disabling it again in normal operation because it can cost performance at scale.
• Dynamic ARP resolution over the ICL interface is not supported; ARP replies received on ICL are discarded.
• DHCP relay combined with MAC address synchronization is not supported; use VRRP over IRB/RVI when Layer 3 DHCP relay is required, and prefer the bootp forwarding helper over the extended jdhcp process.

Verification Checklist

show iccp
show lacp interfaces ae0 extensive
show interfaces ae0 extensive
show log messages | match iccp
monitor traffic interface ae0 no-resolve matching "ether proto 0x8809"

Run the first three commands on both peers and compare the printed LACP system ID. If the two peers disagree before any failure has occurred, you have a configuration problem (mismatched mc-ae-id, chassis-id or LACP mode) rather than an ICCP problem, and no amount of capture analysis will fix that. Compare the system ID before and after a controlled ICCP shutdown to validate that the failure action you intend is the one that actually fires.

Split-Brain Prevention Summary

Split-brain occurs when ICCP adjacency is lost while both peers are still forwarding. The combination of aggregated ICCP links, backup liveness detection over the management network, a master-only management address, and a conservative BFD liveness interval is what keeps the LACP system ID behavior deterministic. Apply the same system ID and administration key on both peers, verify with show iccp and show lacp interfaces, and always test peer failure in a maintenance window. For general design guidance, review our Juniper MC-LAG best practices article as well, and the system ID matrix walkthrough in Juniper MC-LAG ICCP failure scenarios and LACP system ID.

原文链接：https://juniper.net/documentation/us/en/software/junos/mc-lag/topics/concept/best-practices-usage-notes.html