MC-LAG ICCP Failure Scenarios: Liveness Detection and LACP System ID Behavior

When the Inter-Chassis Control Protocol (ICCP) connection between two Juniper MC-LAG peers drops, the way each peer treats its multichassis aggregated Ethernet (MC-AE) interface decides whether the server loses its bundle or keeps forwarding. This article walks through the official ICCP failure scenarios for QFX and EX series switches: what backup liveness detection does, when the LACP system ID is changed, and which settings engineers should apply to avoid a split-brain state.

Why ICCP Failure Is Dangerous for MC-LAG

ICCP carries the control messages that keep both MC-LAG peers coordinated - configuration parameters, LACP system IDs and forwarding state. Because ICCP runs over TCP/IP between the peers, a link failure, an FPC failure, or a full peer reboot can bring the adjacency down while both switches stay powered on. If nothing else intervenes, both peers may remain active and both may forward frames to the connected server, which is exactly the split-brain condition described in Junos documentation.

It helps to separate the two links that hold an MC-LAG pair together. The interchassis link (ICL) is the data-plane path: it is the aggregated Ethernet bundle between the two chassis that carries traffic picked up on the standby peer's member links over to the active peer. ICCP is the control-plane session that runs between peer addresses (normally loopbacks or dedicated IRB interfaces) and synchronises state. Losing the ICL hurts throughput and redundancy; losing ICCP hurts correctness, because the peers can no longer agree on who owns the bundle.

The failure is asymmetric, which is what makes it so damaging. The server attached to the MC-AE has no idea that a control session between two switches failed - it only sees LACP PDUs. If both peers keep their member links collecting and distributing with the same LACP system ID, the server keeps a working bundle and never signals a problem, while the two chassis quietly disagree about which of them should be forwarding. That is how a control-plane fault becomes a silent forwarding loop or duplicated frames instead of a clean link-down event.

What Actually Runs over ICCP

ICCP is not a single protocol so much as a transport for three client applications. Junos reports them by name in the ICCP output, and knowing what each one synchronises explains why an ICCP failure has such wide blast radius:

• LACPD - the LACP daemon. It exchanges the LACP state of the MC-AE, including the aggregate key and the system ID that both peers present to the server.
• MCSNOOPD - the multichassis snooping daemon. It keeps snooping-derived state such as ARP and ND entries learned on the access side synchronised, so both peers know the same set of hosts.
• ESWD - the Ethernet switching daemon. It synchronises bridging state, including MAC entries and the forwarding state of the MC-AE member links.

When ICCP goes down, all three stop exchanging information at once. MAC and ARP entries learned by one chassis are no longer known by the other, and - most importantly - the LACP view of the bundle diverges. Everything that follows in the failure scenarios below is a consequence of that LACP divergence.

Backup Liveness Detection and the LACP System ID

Junos uses an out-of-band channel for backup liveness detection: the peers exchange keepalive messages over the management network so each can tell whether the other is really down. The key behavior table from the Junos MC-LAG documentation shows:

• ICCP down + backup liveness not configured: the LACP system ID changes to the default value on both active and standby MC-AE interfaces.
• ICCP down + backup liveness active: the LACP system ID also changes to the default - the standby peer confirms the other is dead and takes over.
• ICCP down + backup liveness inactive: no change in LACP system ID, because the peer is assumed alive.
• ICCP up but ICL down: the LACP state on standby links is set to standby and the MUX state moves to waiting.

Because only one LACP system ID can win on the downstream server, the system ID change is what breaks the tie after a real peer failure - the server keeps only the links from the surviving peer.

The subtlety engineers most often miss is that a system ID change is a deliberate action, not a symptom. While ICCP is healthy, both peers advertise the same multichassis system ID so the server builds a single aggregator across two chassis. When one peer believes the other is dead, it stops presenting the shared identity and falls back to its own default system ID. On the wire this looks like the partner device changed identity, which is precisely the signal LACP uses to tear down a bundle.

The ICCP Failure Matrix in Practice

The four rows of the documented behavior table translate into four very different operational outcomes. Reading them as a matrix makes the design decisions obvious:

The dangerous combination is the first row combined with a healthy ICL: two chassis that both think they are active, both advertising a default system ID, both forwarding for the same hosts. If the two chassis started from the same configuration, the default system IDs can even be identical, and the server happily keeps a bundle straddling both switches while the peers disagree.

How the LACP System ID Breaks the Tie at the Server

LACP does not elect a single winner for the whole port channel - it evaluates each physical link. Every link carries actor and partner information, and links are aggregated only when the actor and partner identifiers and keys match. The partner's system ID (a priority plus a MAC address) and aggregate key are part of that test.

So when one MC-LAG peer switches to its default system ID and the other does not, the server sees two groups of links with different partner system IDs. The LACP state machine marks the mismatching links as not aggregable; they drop out of the bundle and either go to standby or become individual links, depending on the implementation. That is the mechanism behind a sentence that otherwise reads like magic: the LACP system ID change is what protects the server link. It removes the ambiguity at the exact point where the server would otherwise have to choose between two contradictory switches.

There is a corollary worth remembering during an incident: if a server's port channel suddenly reports half of its members as standalone or standby, and the switch logs show MC-AE or ICCP activity at the same moment, suspect a system ID change rather than a cabling fault. Look at the LACP neighbour output on both chassis before you touch a patch panel.

Session Establishment Hold Time and Split-Brain Avoidance

Two timers decide how quickly a peer gives up on its partner and how gracefully it comes back. The ICCP session-establishment-hold-time is how long a restarted peer waits before it considers the ICCP session usable again; setting it high (Juniper's examples use 340 seconds) prevents a peer that has just rebooted from immediately fighting for the active role while its partner is still forwarding. The MC-AE init-delay-time similarly holds the access interfaces down for a configurable period after boot so the bundle does not come up before ICCP state is rebuilt.

Combine those with status-control and prefer-status-control-active and the pair has a predictable bias: one chassis is normally the active side for the MC-AE, and the other stays standby with its member links in standby state. When ICCP and the ICL are both healthy, this bias keeps traffic on one chassis and avoids the split-brain case entirely; when ICCP fails and liveness confirms the peer is gone, the standby side promotes itself deliberately instead of by accident.

Verifying ICCP and Liveness State

user@qfx> show iccp
Redundancy Group Information for peer 1
  TCP Connection:          Up
  Liveness Detection:      Up
  Client Application:      MCSNOOPD, LACPD, ESWD

Use show iccp detail to inspect the TCP connection, the BFD-based liveness detection state, and each client application (MCSNOOPD, LACPD, ESWD). When troubleshooting, confirm that backup-liveness-detection is configured on both peers, otherwise the check never runs. A one-sided configuration is the most common reason a lab reproduces the documented behavior and production does not.

user@qfx> show iccp detail
user@qfx> show iccp statistics
user@qfx> show interfaces ae0 extensive
user@qfx> show lacp interfaces ae0
user@qfx> show interfaces ae0 mc-ae
user@qfx> show configuration protocols iccp

Read the output in this order during an incident: first the ICCP session state and the liveness state, then the MC-AE interface block (which reports the peer, the chassis ID, the redundancy group and the current forwarding status), then the LACP neighbour view. If the ICCP session is down while the ICL is up, compare the LACP system IDs on both chassis - a mismatch there is the fingerprint of this failure mode.

Configuration Best Practices from Juniper

set protocols iccp peer 1 session-establishment-hold-time 340
set protocols iccp peer 1 liveness-detection minimum-interval 8000
set protocols iccp peer 1 backup-liveness-detection backup-peer-ip 192.0.2.2
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae0 aggregated-ether-options mc-ae status-control standby
set interfaces ae0 aggregated-ether-options mc-ae prefer-status-control-active

Juniper also recommends configuring ICCP over peer loopback addresses, using separate ports/FPCs for the ICL and ICCP links, and setting the ICCP BFD interval to at least 8 seconds when ICCP runs over an IRB interface so that GRES stays graceful.

Two further habits pay for themselves. First, keep the ICL and the ICCP path physically independent: if both ride the same line card, a single FPC failure removes the data path and the control path together, and the peers cannot even tell each other what happened. Second, size the ICL for the full bandwidth of the standby peer's member links. If the ICL is thinner than the MC-AE access bundle, an ICCP failure that leaves the pair degraded will also leave traffic hairpinning through an oversubscribed link.

A Troubleshooting Sequence That Works Under Pressure

1. Confirm the failure domain: show iccp on both peers. Note whether the TCP connection, liveness detection, or only a client application is down.
2. Check the ICL separately with show interfaces ae0 extensive. ICCP down plus ICL up is a degraded pair; ICCP down plus ICL down is a split brain.
3. Compare LACP system IDs and aggregate keys on both chassis with show lacp interfaces ae0.
4. Verify backup-liveness-detection exists on both peers. If only one side has it, the documented behavior table does not apply.
5. Read the log: show log messages | match ICCP and show log messages | match mc-ae give the sequence of events with timestamps, which is what you need for a post-incident write-up.
6. Only after the control plane is understood, look at the server side. Server LACP state that shows individual or standby memberships is usually the consequence, not the cause.

Reproducing the Failure in a Lab

Nothing about this behavior needs a production outage to learn. In a lab pair you can deactivate the ICCP session on one peer and watch the LACP system ID move to the default value on the other side. Run show iccp and show lacp interfaces ae0 before and after the change, then with backup liveness detection enabled and disabled, and the four rows of the failure matrix become visible in a few minutes.

user@qfx> configure
user@qfx# deactivate protocols iccp
user@qfx# commit
user@qfx# run show iccp
user@qfx# run show lacp interfaces ae0 | match "System ID"

Record the LACP system ID before and after. If the two peers show the same value while ICCP is up and different values after ICCP fails, your lab is behaving exactly as documented - and you now have the reference output to compare against the next time production surprises you.

Related Reading on This Site

See our earlier deep dives on MC-LAG ICCP failure scenarios: standby and split-brain and Juniper ICCP backup liveness configuration, plus the ICCP, BFD and LACP system ID deep dive and our MC-LAG design notes on ICCP liveness, ICL and hold timers.

原文链接：https://www.juniper.net/documentation/us/en/software/junos/mc-lag/topics/concept/best-practices-usage-notes.html