Juniper MC-LAG ICCP Failure: Packet Loop and Troubleshooting Guide

MC-LAG (Multichassis Link Aggregation) on Junos makes two switches behave like one active-active bundle, but the moment the Inter-Chassis Control Protocol (ICCP) session between the peers breaks, a cascade of subtle failures can appear — from packet loops on the attached server to stale MAC entries and wrongly suppressed failover. Based on Juniper's official MC-LAG troubleshooting documentation for QFX and EX series, this article collects the most common ICCP failure scenarios, explains why each one happens, gives the exact solution or expected-behavior note for every case, and walks through the diagnostic order that separates a real forwarding loop from an ICCP control-plane problem.

How MC-LAG forwarding works while ICCP is healthy

An MC-LAG pair presents one LAG to the attached device. Each peer has local membership in the multichassis aggregated Ethernet (MC-AE) interface, the two peers are joined by an interchassis link (ICL), and ICCP runs between them, normally across the ICL or a dedicated path, to synchronize MAC and ARP information and coordinate forwarding state.

The ICL is not just a control path. When a frame arrives on a peer that does not have a local egress port for the destination, the frame is sent over the ICL to the peer that does. Broadcast, unknown-unicast and multicast traffic that arrives on one peer is normally handled by that peer's local ports, and the ICL carries the remainder. Both peers forward Layer 2 traffic at the same time, which is what makes the design active-active.

All of that coordination depends on ICCP. Remove ICCP and the peers no longer know what the other is doing, yet each one still has live MC-AE member ports and a complete forwarding state of its own.

Why ICCP failure causes packet loops

When you enable backup liveness detection for an MC-LAG and the liveness packets are lost due to a temporary failure, both peers can remain active at the same time. With two peers forwarding traffic for the same Multichassis Aggregated Ethernet (MC-AE) interface, the server sees frames arriving from two sources and packets can loop. The recommended fix is to configure interfaces aeX aggregated-ether-options mc-ae events iccp-peer-down force-icl-down, which forces the inter-chassis link (ICL) down when ICCP is lost so that the secondary peer stops forwarding.

The loop path is worth tracing explicitly, because it explains every symptom you will see on the wire:

1. The server transmits a broadcast or unknown-unicast frame, and the LAG hash sends it out of only one of its two member links — say, toward peer A.
2. Peer A has no local port for the destination, so it floods the frame to all local ports and across the ICL to peer B.
3. Peer B floods the frame out of its own MC-AE member ports, including the link back to the same server. The server now receives a copy of a frame it originally sent, and if it is itself a switch or bridge, it forwards it again.
4. Meanwhile the server's own return traffic for the same destination may hash toward peer B, producing a second copy that travels the mirror-image path. With both peers active and no ICCP-derived state, the two copies are no longer deduplicated.

The result is not a classic Layer 2 loop with a broadcast storm and 100 % CPU — that is why it is often missed. It is duplicate delivery, asymmetric forwarding and MAC movement, which shows up as packet loss, retransmissions, TCP resets and a server that "briefly disappears" at irregular intervals.

First five minutes: the diagnostic order

user@switch> show iccp
user@switch> show configuration protocols iccp
user@switch> show lacp interfaces ae0
user@switch> show ethernet-switching table
user@switch> show ethernet-switching mac-learning-log
user@switch> show interfaces ae0 extensive
user@switch> show log messages | match "iccp|MAC_MOVE|loop"

Work through them in this order:

• ICCP state — show iccp tells you whether the session is up, whether the peer's status is known, and which peer is status-control active. If ICCP is down, stop guessing about loops and establish which peer is supposed to be forwarding.
• Configuration intent — show configuration protocols iccp confirms the peer address, the redundancy group list, and whether backup liveness detection and force-icl-down are actually configured. Empty statements here explain most "unexplained" behaviour.
• LAG membership — show lacp interfaces ae0 shows which member links are collecting and distributing. A member that is receiving but not distributing on one side is a common cause of one-way traffic and apparent duplication.
• MAC location — show ethernet-switching table shows where a server MAC is learned. If the same MAC appears to move between the MC-AE port and the ICL, the peers are not agreeing about the topology.
• MAC movement history — show ethernet-switching mac-learning-log is the fastest way to prove that a MAC is flapping, and between which interfaces.
• Interface counters — show interfaces ae0 extensive exposes drops, queue congestion and error counters on the LAG and the ICL. A saturated ICL produces the same symptoms as a broken one.
• Logs — ICCP transitions, MAC move notifications and the removal of the ICL from the bundle all generate messages; correlating them with the incident start time usually identifies which event came first.

If you need to see the traffic rather than the state, capture on both peers simultaneously — a packet seen twice, once on the MC-AE and once on the ICL in the opposite direction, is the fingerprint of the duplicate-forwarding case:

user@switch> monitor traffic interface ae0 no-resolve
user@switch> monitor traffic interface ae1 no-resolve

Does the loop come from ICCP, or did ICCP fail because of the loop?

This distinction changes the fix, so make it explicitly. An ICCP-first failure shows up as ICCP session state changes in the log, with the LAG and its member links stable and error-free. A loop-first sequence typically leaves ICCP alone and shows MAC moves, output drops and queue congestion, and the ICCP session may still be up the entire time. If ICCP is up and the peers agree about state, force-icl-down and backup liveness detection will do nothing for you — the duplicate forwarding has a different cause, such as an incorrect LAG hash with a bridge in the path, or a downstream device doing something unexpected with the frames it receives.

Common Junos MC-LAG failure scenarios

1. MAC Address Age Resets to Zero on MC-AE

Activating and then deactivating an interchassis link-protection link (ICL-PL) resets the MAC address age learned on the multichassis aggregated Ethernet interface to zero. This is expected behavior — the next-hop interface change triggers MAC updates in hardware, which in turn trigger aging updates in the Packet Forwarding Engine. The show ethernet-switching table output showing age 0 is normal.

2. MAC Not Learned Remotely in Default VLAN

On a QFX3500 running Junos 12.3 or earlier, ICCP does not synchronize a MAC address learned in the default VLAN with the other MC-LAG peer. Upgrade to a newer release to resolve this known limitation.

3. Local Status Stays Standby

If the MC-AE interface is down when the state machine is in a synchronized state, the local status becomes standby. If the interface goes down after the state machine is active, the local status remains active. Check the interface state before concluding the peer has failed.

4. Double Failover Scenario

If ICCP goes down and then the active MC-AE interface also goes down, a double failover occurs: the standby peer has no ICCP information and cannot coordinate failover correctly. This is one of the most disruptive MC-LAG failure modes and is why backup liveness detection plus force-icl-down is strongly recommended.

5. Layer 3 Traffic Not Redirected to Standby

When ICCP is down, the status of the remote MC-LAG peer is unknown — even if it is configured as standby, Layer 3 traffic is not redirected to it because the switch assumes the peer is down.

6. ICCP Session Flaps Without a Link Failure

ICCP runs over a TCP session, so it can drop for reasons that have nothing to do with cabling: control-plane congestion, CoS marking that starves the ICCP path, an MTU mismatch on the path that carries it, or a routing change that moves the session to a different path. Every flap resets peer synchronization for a period, and if the flap happens while traffic is heavy, the symptom is a brief burst of duplicates rather than a full outage. Protect the session by carrying it over a loopback-to-loopback path with a dedicated, well-provisioned ICL or a separate interface, and monitor the transition count instead of waiting for a hard failure.

ARP and MAC out-of-sync fix: arp-l2-validate

To prevent ARP and MAC table entries drifting out of sync in an MC-LAG configuration, configure the arp-l2-validate option on the switch's IRB interface. This option turns on validation of ARP and MAC entries and automatically applies updates when they become out of sync. It is available on QFX Series, EX4300 (from Junos 15.1R4) and EX9200 (from 13.2R4) switches.

user@switch# set interfaces irb unit 0 family inet address 10.10.10.1/24 arp-l2-validate
user@switch# commit

Enable it as a workaround during incidents, but disable it during normal operation because it can impact performance in large-scale configurations. The performance cost is proportional to the number of ARP and MAC entries being validated, so a large Layer 2 domain with tens of thousands of entries is exactly the case where the workaround should be time-boxed and removed once the real fix lands.

Stopping the loop: force-icl-down and link protection

user@switch# set interfaces ae0 aggregated-ether-options mc-ae events iccp-peer-down force-icl-down
user@switch# commit

The statement tells the peer: if ICCP is gone, shut the ICL and stop pretending you can deliver traffic that arrives for the other chassis. Combined with backup liveness detection on both peers, it converts the two ambiguous failure modes — ICCP down with a live peer, and ICCP down with a dead peer — into predictable outcomes instead of duplicate forwarding.

Apply it to every MC-LAG interface in the design, not only the one that has caused an incident. An MC-LAG whose interfaces disagree about force-icl-down will behave differently depending on which pair of failure events occurs first.

Worked example: a server reporting duplicate frames

A hypervisor reports duplicate TCP acknowledgements and occasional RSTs while its LAG status stays up. Both peers show the MC-AE up, ICCP is down, and the log shows the ICCP session dropping when the ICL's aggregated Ethernet lost one of its two member links. Nothing looks broken at Layer 1 because a member link loss does not necessarily take down the LAG.

1. show iccp on both peers confirms the session is down and the peer status is unknown.
2. show configuration protocols iccp shows backup liveness detection configured, but no force-icl-down on the MC-AE.
3. The liveness probes answer, because both peers are alive and their management network is healthy — so no failure action is taken and both peers keep forwarding.
4. Adding force-icl-down and committing clears the duplicate frames within one probe interval.
5. The underlying ICL member link is then repaired; when ICCP re-establishes, the MC-AE returns to normal active-active forwarding.

The lesson: a live peer does not mean a healthy MC-LAG. Backup liveness detection and force-icl-down cover two different failure modes, and a design that only has the first one will keep forwarding duplicates every time the ICL alone breaks.

Preventing recurrence

• Configure ICCP over a loopback or IRB address, and carry it on a path that is as protected as the ICL itself.
• Use an aggregated Ethernet interface for the ICL, with members on different line cards, so a single card failure cannot remove the interchassis path.
• Enable backup liveness detection and force-icl-down on both peers, and audit the configuration after every change window — asymmetric statements are the most common latent defect in an MC-LAG design.
• Alert on ICCP session flaps, not just on session failure: a flapping session produces intermittent duplicates that never trip a threshold.
• Capacity-plan the ICL for the traffic that will cross it in the worst case, and monitor it for congestion rather than only for errors.
• Keep arp-l2-validate in the toolbox as an incident workaround, and track which switch models and releases support it before you need it at 03:00.

Related reading on this site: MC-LAG split-brain and backup liveness detection configuration, ICCP failure effects on ARP, PIM and the ICL, Juniper MC-LAG best practices and ICCP failure scenarios and standby split-brain. If you are new to the concept, start with our MC-LAG ICCP liveness and LACP system ID deep dive, or compare with Arista EOS MLAG configuration and Cisco Nexus vPC failover troubleshooting.

原文链接：https://www.juniper.net/documentation/us/en/software/junos/mc-lag/topics/task/troubleshooting-mc-lag-qfx-series-cli.html