Juniper MC-LAG ICCP Failure Scenarios and LACP System ID

A Juniper MC-LAG keeps two switches acting as one Link Aggregation Group toward a server or access switch, but that illusion depends entirely on the Inter-Chassis Control Protocol (ICCP) connection between the peers. When ICCP fails, the LACP system ID behavior of the multichassis aggregated Ethernet (MC-AE) interfaces changes, and understanding those changes is what separates a controlled failover from a split-brain outage. This article walks through the ICCP failure scenarios documented for QFX and EX series switches, explains how backup liveness detection changes the outcome, and lists the verification commands you need during an incident.

The Building Blocks: ICCP, ICL and MC-AE

MC-LAG peers exchange control state over ICCP (a TCP-based protocol), synchronize MAC/ARP and LACP state, and protect each other's links through the interchassis link (ICL). Each MC-AE interface uses a configured LACP system ID while ICCP is healthy, and one peer is designated active while the other holds its links in standby.

Three objects get conflated constantly during an outage, so keep them separate in your head:

• ICCP is the control channel. It is a TCP session between the two peer addresses, established over an IP route rather than over a particular cable, optionally authenticated with an MD5 key, and optionally protected by BFD liveness detection. Because it relies on IP reachability to the peer address, a routing problem between the loopbacks breaks ICCP even when every link is lit.
• ICL and ICL-PL are the forwarding paths between the chassis. The ICL carries traffic that must cross from one peer to a member link on the other peer; the protection link, ICL-PL, is the path the control traffic uses and the path that carries data when the primary interchassis path is unavailable.
• MC-AE is the multichassis aggregated Ethernet interface the downstream device actually sees. It carries a matching mc-ae-id on both peers, a shared lacp admin-key, a shared lacp system-id, a mode (active-active in most data centre designs), and a status-control statement that decides which chassis is active.

set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae mode active-active
set interfaces ae0 aggregated-ether-options mc-ae status-control active
set interfaces ae0 aggregated-ether-options mc-ae events iccp-peer-down prefer-status-control-active
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp admin-key 1
set interfaces ae0 aggregated-ether-options lacp system-id 00:00:5e:00:53:01

The system ID and the admin key must be identical on both chassis, but nothing in the commit process enforces that equality. A mismatch is therefore not a configuration error you catch at review time; it becomes a forwarding problem you debug at 3 a.m.

Why the LACP System ID Is the Whole Story

A downstream server or switch does not know that MC-LAG exists. It builds its aggregator purely from the LACP information it receives: the partner system identifier, the partner system priority and the partner operational key carried in each received LACPDU. When both chassis advertise the configured system ID and admin key, the host sees one partner and forms one aggregator with members spread across two physical chassis.

The moment the two chassis advertise different system identifiers, the host sees two partners. It then either forms two aggregators or selects the bundle it heard first and leaves the other links out of forwarding, depending on its LACP implementation and the order in which the PDUs arrive. Either outcome is a partial or complete outage for the half of the traffic that hashes to the abandoned links.

This is exactly why the documented ICCP failure behaviour changes the LACP system ID. Advertising the per-chassis default system ID makes the two peers look like two separate partners again, which deliberately prevents the downstream device from load-sharing across a pair of chassis that are no longer coordinating. The behaviour is not a bug; it is the mechanism that stops dual-active forwarding when the control plane has lost the ability to arbitrate.

Timers That Govern the Transition

Two timers decide how quickly the scenario plays out, and they are frequently left at defaults. The session establishment hold time is the window during which a peer tries to build the ICCP connection after IP reachability to the peer address comes up. On QFX and EX Series switches the default is 300 seconds, and the session establishment time must be at least 100 seconds higher than the init delay time; a common pair of tuned values is 340 seconds for the establishment hold time and 240 seconds for the init delay.

Configuring the hold time explicitly is a documented recommendation, because it speeds up ICCP reconnection after a peer reboot and removes a long window of degraded, default-system-ID operation. For a peer reboot specifically, Juniper recommends the ICL-down delay rather than the hold time to control convergence, since the delay defers bringing the MC-AE back up until the local protocols have converged.

set protocols iccp local-ip-addr 10.0.0.1
set protocols iccp peer 10.0.0.2 session-establishment-hold-time 340
set protocols iccp peer 10.0.0.2 redundancy-group-id-list 1
set protocols iccp peer 10.0.0.2 liveness-detection minimum-interval 8000
set protocols iccp peer 10.0.0.2 backup-liveness-detection backup-peer-ip 10.0.0.2

Note that ICCP liveness detection uses multihop BFD by default and runs in centralized mode; if ICCP is carried over an IRB interface, a liveness-detection interval of at least 8 seconds is recommended so that graceful Routing Engine switchover keeps working. Backup liveness detection is a separate mechanism: it exchanges keepalive messages over the management link between the peers, and it is only exercised when the ICCP TCP connection is down. It must be configured on both peers to have any effect.

ICCP Failure Scenario Matrix

Juniper documents the interaction between ICCP connection status, ICL status and backup liveness detection (BLD) in a compact table. The key combinations are:

• ICCP down, backup liveness not configured: the LACP system ID on the MC-AE interfaces is changed to the default value. Both peers advertise the default system ID, so the connected server keeps the first bundle that comes up and drops the other - this is intentional behavior that prevents dual-active forwarding while ICCP is down.
• ICCP down, BLD active: the LACP system ID is also changed to the default for both active and standby MC-AE interfaces. The liveness check tells each peer whether the other is still alive, which gates the failure action.
• ICCP down, BLD inactive: there is no change in the LACP system ID.
• ICCP up, ICL down: the standby peer's LACP state is set to standby and the MUX state moves to waiting; the active peer keeps forwarding so the server sees no disruption.

Restating the same four combinations as a decision table makes the trade-off explicit, because each row is a different balance between availability and loop prevention:

Scenario 1 and 2: ICCP Down With and Without Working Liveness

When ICCP goes down and BLD is not configured at all, the local chassis has no way to tell whether the peer is alive. The switch therefore takes the conservative route: it stops claiming the configured LACP identity and falls back to the default system ID. From the downstream device's perspective the pair is no longer a single partner, so it settles on one bundle and the other bundle's links drop out of the aggregator. Throughput is halved but nothing loops.

With BLD configured and answering, the peer state is known to both chassis. Backup liveness detection only runs while ICCP is down; while the ICCP TCP connection is established or BFD reports the peer up, peer status is treated as up. That knowledge allows a coordinated takeover instead of an indefinite standoff. Configure BLD on the management interface of both peers for sub-second traffic loss across a peer reboot.

Scenario 3: ICCP Down and BLD Not Answering

This is the dangerous row in the table. If BLD is configured but the liveness check fails - because the management link is down, the backup peer address is wrong, or the response window expired - the documented behaviour is that the LACP system ID is not changed. Each chassis keeps advertising the configured identity, and each chassis may conclude that the remote peer is gone. Both peers can then become active simultaneously for the same access device.

The result is a temporary packet loop toward the server: both chassis believe they own the MC-AE and both forward. Juniper explicitly classifies packets looping on the server during an ICCP failure as the expected behaviour inside that failure window. The lesson for design is not that the behaviour is wrong, but that BLD is a dependency in its own right: if you enable it, you must monitor the management path and the backup peer address, because a silent BLD failure converts a safe degradation into a loop.

Scenario 4 and 5: ICL Failure With ICCP Healthy

An ICL failure with ICCP still up is the benign case, and it is worth knowing why. ICCP rides on IP reachability between the peer addresses, so it can survive the loss of the primary interchassis path entirely. On the standby peer the MC-AE is moved to standby and the LACP multiplexer state moves to waiting, which tells the downstream device not to use those links. The active peer keeps forwarding for the group, so the host sees no disruption at the LACP level. If you want to observe this, watch the MUX state on the standby chassis while you shut the ICL.

The subtlety is that ICCP reachability and ICL forwarding are now decoupled: traffic that must reach a member link on the standby chassis crosses the ICL-PL instead. An ICL failure silently converts your protection path into a production path, so size ICL-PL accordingly.

Scenario 6: LACP System ID Mismatch Between Peers

A mismatch is not an ICCP failure at all, and that is precisely why it is so easy to miss. ICCP can be perfectly up, both peers can be exchanging MAC and ARP state, and the deployment can still be broken because the two chassis advertise different values in their LACPDUs. Common causes:

• The system ID was typed from the chassis MAC on one peer and copied as a literal on the other.
• One chassis was replaced and restored from a different template, leaving the site-standard system ID in a template that the replacement never loaded.
• The admin key differs even though the system ID matches, which produces the same two-partner symptom because the operational key is what distinguishes aggregators within a partner.

The symptom pattern is a specific one: link-state is up on both chassis, ICCP is up, but traffic randomly under-performs or a fraction of flows fail while the rest are healthy. Because the host's aggregator selection can change on the arrival order of PDUs, the fault can also appear to move between hosts after a reboot of the access device.

The fix is to standardise the identity as configuration data rather than as a per-device value, and then verify it from the downstream side. On the downstream device, compare the partner system ID reported for each member link; on the peers, compare the configured value and the value actually advertised.

! on the MC-LAG peers
show configuration interfaces ae0 | display set | match "lacp"
show lacp interfaces ae0
show lacp interfaces ae0 extensive | match "system-id"

! on the downstream server or switch: confirm one partner, not two
show lacp neighbor interface ae0
show etherchannel summary | include LACP

Scenario 7: ICCP Flapping

A session that goes down and comes back repeatedly is worse than one that stays down, because each transition rewrites the LACP system ID the downstream device sees and forces it to re-evaluate its aggregator. The usual causes are path-related rather than protocol-related: an unstable route to the peer loopback, an MTU mismatch that lets small control packets through but breaks larger sync traffic, a BFD interval so aggressive that a busy control plane exceeds it, or an MD5 authentication key mismatch that intermittently passes after a rekey. Look at the ICCP trace and the session history before touching LACP at all.

Behavioral Notes That Matter in the Field

• Packets loop on the server when ICCP fails: with backup liveness detection enabled, a temporary loss of BLD packets can leave both peers active, and both then send traffic to the server. Juniper classifies this as expected behavior for the failure window.
• Default system ID after reboot: after a reboot or a new ICCP configuration, until the ICCP connection becomes active the LACP messages on the MC-AE interfaces use the default system ID. The configured system ID only takes effect after the peers synchronize.
• Double failover: if ICCP goes down and then the MC-AE interface on the active peer also goes down, the standby peer does not detect the second event and blocks ICL-PL traffic - a scenario worth designing around rather than discovering at 3 a.m.
• Peer does not enter standby: this usually means the peer IP in the ICCP configuration differs from the peer IP in the multichassis protection configuration; they must match.
• Maximum VLAN scale changes the timers: the recommended MC-AE bring-up deferral is 240 seconds for a full VLAN configuration, and 420 seconds if IGMP snooping is enabled on all VLANs.

Step-by-Step Triage Runbook

1. Establish which layer failed. Run show iccp and read the connection state, liveness state and the list of client applications. If the session is not established, the problem is IP reachability, authentication or the hold time, and LACP is a symptom rather than a cause.
2. Check the peer configuration symmetry. Compare local-ip-addr, peer address and the multichassis protection peer address on both chassis. A mismatch here is the documented cause of a peer that never enters standby.
3. Read the LACP identity the host sees. Compare show lacp interfaces on both peers against the host's neighbour view. If both peers advertise the default system ID, ICCP is down and the switch is in its designed degraded mode.
4. Decide the failure window. If the peer is genuinely dead, the halved bandwidth of scenario 1 is the correct outcome. If the peer is alive but unreachable over ICCP, the correct action is to restore ICCP reachability rather than to fight the LACP state.
5. Only then touch data-plane configuration. Changing the system ID or admin key while ICCP is flapping adds a second variable to an unresolved problem.

Verification Commands

user@switch> show iccp
user@switch> show iccp peers
user@switch> show iccp trace
user@switch> show mc-ae status
user@switch> show lacp interfaces ae0
user@switch> show interfaces ae0 extensive | match "LACP|system-id"
user@switch> show configuration protocols iccp

During an incident, check the ICCP state first, then compare the LACP system ID in show lacp interfaces against the configured value. If the peers are advertising the default system ID, ICCP is down and the MC-LAG is operating in its degraded failure mode by design.

Related reading: Junos MC-LAG ICCP failure: LACP system ID behavior, Juniper MC-LAG design: ICCP liveness and ICL hold timers and Juniper ICCP backup liveness: preventing MC-LAG split-brain.

Original article: Troubleshooting Multichassis Link Aggregation (Junos OS) | Junos OS MC-LAG User Guide (PDF)