MC-LAG ICCP Failure Scenarios: Standby and Split-Brain

MC-LAG (Multichassis Link Aggregation) depends on a healthy Inter-Chassis Control Protocol (ICCP) session between two peers, and when that session fails the behavior of the LACP system ID, the standby link state and the data plane all change in very specific ways. This article walks through the documented Juniper MC-LAG ICCP failure scenarios, explains why a peer may not enter standby mode, how backup liveness detection prevents split-brain, and which Junos commands to run when the ICCP connection misbehaves.

What Happens When the ICCP Connection Fails

The ICCP session is the brain of the MC-LAG pair: it synchronizes MAC learning, ARP, and LACP state between the two multichassis aggregated Ethernet interfaces. When ICCP goes down, both peers must coordinate a fallback through backup liveness detection and the inter-chassis link (ICL). Juniper documents the exact matrix of ICCP status, ICL status, backup liveness status and the resulting action on the multichassis aggregated Ethernet interfaces with status control set to standby.

The critical point is that ICCP failure alone does not instantly mean traffic loss. The outcome depends on three independent inputs: whether the ICCP TCP session is up, whether the inter-chassis link is up, and whether backup liveness detection (a keepalive sent over the management network) is receiving replies. Only when those inputs are combined does each peer decide to stay active, move its MC-AE interfaces into standby, or disable them entirely.

The ICCP Architecture and Its Client Daemons

ICCP is a TCP-based protocol that runs between the two peers, typically over the inter-chassis link or a dedicated control path. It does not forward traffic itself; instead it carries state for three client daemons that each own a slice of the MC-LAG behavior:

• mcsnoopd — the multichassis snooping daemon, responsible for synchronizing MAC and ARP learning so that a frame learned on one peer is known to the other.
• lacpd — the LACP daemon, which shares the negotiated LACP state, actor/partner information and the system ID so that both peers present a consistent LAG to the attached server.
• eswd — the Ethernet switching daemon, which programs the forwarding state derived from the synchronized data.

Because ICCP is the transport for all three, a session flap is not just a single event: it is a coordinated reset of MAC, ARP and LACP state across both peers. That is why troubleshooting always starts with the ICCP session itself and only then moves down the stack.

MC-LAG Peer Does Not Go into Standby Mode

A classic failure: the MC-LAG peer never enters standby mode because the peer IP address configured under protocols iccp does not match the IP address configured under the multichassis protection configuration. The two addresses must be identical for the state machine to work.

user@switch# set interfaces ae1 unit 0 multi-chassis-protection 172.16.32.6 interface ae1
user@switch# set protocols iccp local-ip-addr 172.16.32.5
user@switch# set protocols iccp peer 172.16.32.6 session-establishment-hold-time 50
user@switch# set protocols iccp peer 172.16.32.6 redundancy-group-id-list 1

The address under multi-chassis-protection is the address of the remote peer that this MC-AE should be protected against, while the address under protocols iccp peer is the peer it will talk ICCP with. If these two differ, a peer can believe it is talking to one chassis while protecting against another, and the standby state transition never fires. Always verify them together with show iccp and show interfaces ae1 extensive.

Packets Loop on the Server When ICCP Fails

When backup liveness detection is enabled and its packets are lost during a temporary MC-LAG failure, both peers stay active and both send packets to the connected server. This is expected behavior, and it is exactly the split-brain condition that backup liveness detection is designed to recover from: the liveness check fails after the configured consecutive misses, and the failure action is implemented.

The window between the ICCP failure and the liveness detection timeout is the dangerous period. During that window both peers consider themselves active, so any flooded or unicast frame can reach the server twice, which can trigger MAC flapping, duplicate ARP replies and, in the worst case, broadcast storms. Shortening the liveness detection intervals narrows the window at the cost of sensitivity to transient loss on the management network.

Both Peers Use the Default LACP System ID After a Reboot

After a reboot or a new ICCP configuration commit, if the ICCP connection is not active, LACP messages on the multichassis aggregated Ethernet interfaces use the default system ID instead of the configured one. The server then accepts the first link that comes up and brings down the other links carrying a different LACP system ID, so traffic fails over to one peer only.

user@switch> show lacp interfaces ae0
user@switch> show lacp interfaces ae0 extensive

This is one of the most confusing symptoms in the field because the topology looks healthy: ICCP may recover a few seconds later, but the server has already pinned its LAG to a single peer and will only re-balance after LACP renegotiation. The fix is to ensure ICCP is fully established before the server-facing interfaces are brought up, or to explicitly configure the LACP system ID so the peer identity is deterministic.

ICCP Does Not Come Up After Adding and Deleting an Authentication Key

If an authentication key is added and then deleted at the global ICCP level, the ICCP connection may fail to establish even though authentication works correctly at the peer level. Juniper's documented solution is to delete the ICCP configuration and re-add it.

user@switch# delete protocols iccp
user@switch# commit
user@switch# set protocols iccp local-ip-addr 172.16.32.5
user@switch# set protocols iccp peer 172.16.32.6 authentication-key <key>
user@switch# commit

The root cause is stale internal state left behind by the configuration change: the session object retains an authentication attribute that no longer exists at the global level. A clean delete and re-add forces the daemon to rebuild the session from scratch, which is faster and more reliable than trying to toggle individual knobs.

The ICL and the Role of the Inter-Chassis Link

The inter-chassis link is the data path that carries traffic between the two peers when a server sends a frame to the peer that does not own the corresponding MAC entry. It is separate from the ICCP control session that carries the state information, although in many designs both ride on the same physical bundle. A healthy design keeps the two logically distinct so that a control-plane flap does not immediately kill the data-plane path.

user@switch# set interfaces ae0 aggregated-ether-options lacp active
user@switch# set interfaces ae0 multi-chassis-protection 172.16.32.6 interface ae1
user@switch# set interfaces ae1 aggregated-ether-options lacp active
user@switch# set interfaces ae1 multi-chassis-protection 172.16.32.5 interface ae0

If the ICL fails while ICCP is still up, the peers can still exchange state, but they can no longer forward each other's traffic. In that case the MC-AE interfaces either stay active (and traffic destined for the remote MAC is dropped) or move to standby depending on the status control setting. Designing the ICL as a multi-member LAG on different FPCs removes most single points of failure.

Liveness Detection and Split-Brain Prevention

Backup liveness detection exchanges keepalives over the management link between the ICCP peers. Configure it on both sides, using the peer's management (backup) IP address:

user@switch# set protocols iccp peer 10.50.1.2 backup-liveness-detection backup-peer-ip 10.1.1.2
user@switch# set protocols iccp peer 10.50.1.1 backup-liveness-detection backup-peer-ip 10.1.1.1
user@switch# set protocols iccp peer 10.50.1.1 liveness-detection minimum-receive-interval 1000

Note that when backup liveness detection is configured, peer status is always up if the ICCP TCP connection is established or BFD is up; the liveness check only runs while ICCP is down. As a further safeguard on QFX switches, enable arp-l2-validate on the IRB interface to resynchronize ARP and MAC table entries during flapping:

user@switch# set interfaces irb arp-l2-validate

The logic behind split-brain prevention is simple: if ICCP is down but the backup path is alive, the peer knows the remote chassis is still forwarding and therefore moves to standby to avoid duplicate delivery. If both ICCP and the backup path are down, the peer assumes the remote chassis is dead and takes over as active, accepting a brief risk of duplication in exchange for avoiding a full outage.

Verifying MC-LAG State Step by Step

When a report of "traffic is going to only one peer" arrives, walk the layers in order rather than guessing:

1. Check the ICCP session: show iccp should show Connected and list mcsnoopd, lacpd and eswd as registered clients.
2. Check the MC-AE status: show interfaces ae0 should show the multi-chassis-protection state and the standby flag for each peer.
3. Check the LACP view: show lacp interfaces ae0 should show a consistent system ID on both peers.
4. Check the forwarding table: show ethernet-switching table should contain the server MAC learned as a multichassis entry, not as a local entry on a single peer.
5. Check the logs: show log messages | match ICCP and the LACP logs reveal the timeline of the flap.

Key Troubleshooting Commands

• show iccp - ICCP session state and registered client daemons (mcsnoopd, lacpd, eswd)
• show lacp interfaces ae0 - LACP state and system ID in use
• show ethernet-switching table - MAC entries learned on multichassis aggregated Ethernet interfaces
• show log messages | match ICCP - ICCP state transitions
• show interfaces ae0 extensive - multi-chassis protection state and standby indication
• show redundancy-groups mc-lag - which peer is active for each MC-LAG redundancy group

Recovery Procedure After an ICCP Failure

Once the underlying cause is fixed, recovery is usually automatic, but a deliberate sequence avoids a second outage:

user@switch> show iccp
user@switch> clear lacp statistics interfaces ae0
user@switch> show interfaces ae0 extensive | match "Standby|Status"
user@switch> monitor traffic interface ae0

Bring the ICCP session up first and confirm it is stable, then allow LACP to renegotiate the server-facing links, and only then verify that both peers are forwarding. If the server pinned its LAG to one peer during the outage, a bounce of the server-facing interface (or a server-side LACP restart) may be required to restore the balanced state.

Best Practices to Avoid ICCP Failure Scenarios

• Use the peer loopback address for ICCP peering so a single link failure does not reset the session.
• Configure ICCP and the ICL on different interfaces and different FPCs.
• Keep session-establishment-hold-time at 50 seconds for faster ICCP bring-up.
• Configure alternative routes (or a two-member LAG) between the ICCP end points.
• Always configure backup liveness detection with a management path that never shares fate with the ICL.
• Match the peer IP in multi-chassis-protection and protocols iccp exactly, and document it.
• Tune the liveness detection intervals so the split-brain window is shorter than the application's tolerance for duplicate frames.

For related reading on the same topic, see our MC-LAG ICCP failure liveness and LACP system ID deep dive, the MC-LAG ICCP failure packet loop troubleshooting guide, and the ICCP, BFD and LACP system ID overview.

原文链接：https://www.juniper.net/documentation/us/en/software/junos/mc-lag/topics/task/troubleshooting-mc-lag-qfx-series-cli.html