Juniper ICCP Backup Liveness: Prevent MC-LAG Split-Brain

When the ICCP session between MC-LAG peers fails, both switches can end up active at the same time - a split-brain state where the server receives duplicate traffic. Juniper's answer is backup liveness detection: an out-of-band keepalive channel over the management network that lets each peer determine whether the other is still alive, and drive the LACP system ID change that forces the server to pick one side. This article explains how the feature works and how to configure it.

How Backup Liveness Detection Works

Backup liveness detection determines whether a peer is up or down by exchanging keepalive messages over the management link between the two ICCP peers. When an ICCP connection is operationally down, each peer sends liveness detection requests; if responses are not received within the specified time for a given number of consecutive attempts, the liveness check fails and the configured failure action is implemented. It must be configured on both peers.

Important detail: when backup liveness detection is configured, the peer status is always up if either the ICCP TCP connection is established or BFD is configured and up - the backup liveness check only runs while ICCP is down.

The statement was introduced in Junos OS 12.2, and it sits at the [edit protocols iccp peer ip-address backup-liveness-detection] hierarchy. One extension is worth knowing about: from Junos OS 24.2R2 the statement also accepts a routing-instance option, so backup-peer-ip can be reached through a management routing instance instead of the default one. That matters in designs where the out-of-band network lives in its own VRF for exactly the same reason the VSX keepalive does.

Where Backup Liveness Sits in the Control Plane

MC-LAG on Junos is built from four moving parts, and knowing which one detects which failure is most of the troubleshooting:

• ICCP (Inter-Chassis Control Protocol) - a TCP session between the two peers that replicates forwarding information, validates that both sides are configured identically, and propagates the operational state of the MC-LAG members. This is the primary control channel.
• ICL (Inter-Chassis Link) - the data and synchronisation path between the peers, equivalent to the peer link in other vendors' MLAG implementations. ICCP is usually carried over it.
• BFD liveness detection - an optional, faster failure detector for the ICCP session itself. When BFD is configured and up, Junos treats the peer as up regardless of what backup liveness detection thinks.
• Backup liveness detection - the out-of-band mechanism that answers the question the other three cannot when ICCP is gone: is the far switch still alive, or is it dead? It runs only when the ICCP connection is down, and it is a request-response process, not a continuous heartbeat.

That division explains the note above. Junos treats "peer is up" as the OR of the in-band signals: an established ICCP TCP session, or a configured-and-up BFD session, is enough. Backup liveness detection is the fallback used only in the gap where ICCP itself has failed, which is precisely the gap in which a naive design would conclude "peer is dead" and go active on both sides.

Configuration

! QFX1
user@switch# set protocols iccp peer 10.50.1.2 backup-liveness-detection backup-peer-ip 10.1.1.2

! QFX2
user@switch# set protocols iccp peer 10.50.1.1 backup-liveness-detection backup-peer-ip 10.1.1.1

backup-peer-ip is the management (out-of-band) IP address of the remote peer. Configure BFD for faster ICCP failure detection:

user@switch# set protocols iccp peer 10.50.1.1 liveness-detection minimum-receive-interval 1000
user@switch# set protocols iccp peer 10.50.1.1 liveness-detection minimum-transmit-interval 1000

In a real deployment the two statements above are the last lines of a larger MC-LAG configuration, and it is worth seeing them in place, because the ordering of local-IP, peer, protection and aggregated-interface statements is where most configuration mistakes happen:

! ---- ICCP control channel ----
set protocols iccp local-ip-address 10.50.1.1
set protocols iccp peer 10.50.1.2 local-ip-address 10.50.1.1
set protocols iccp peer 10.50.1.2 backup-liveness-detection backup-peer-ip 10.1.1.2
set protocols iccp peer 10.50.1.2 authentication-key "$9$..."   ; optional, both sides

! ---- ICL / multi-chassis protection ----
set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 unit 0 family ethernet-switching
set multi-chassis multi-chassis-protection 10.50.1.2 interface ae1

! ---- MC-LAG member interface ----
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:01
set interfaces ae0 aggregated-ether-options lacp admin-key 1
set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae0 aggregated-ether-options mc-ae mode active-active
set interfaces ae0 aggregated-ether-options mc-ae status-control active
set interfaces ae0 aggregated-ether-options mc-ae chassis-id 0
set interfaces ae0 aggregated-ether-options mc-ae init-delay-time 240
set multi-chassis mc-lag peer 10.50.1.2 interface ae0 iccp-lag-id 1
set multi-chassis mc-lag peer 10.50.1.2 interface ae0 lacp status-control active   ; per side

The two statements that must be symmetric across the pair are system-id and admin-key: MC-LAG peers share one LACP system ID and one administrative key, and Junos uses the shared values to verify that both sides agree. Everything else - chassis-id, status-control, and the interfaces themselves - is local to each switch.

Reading the State with show iccp

The primary verification command is show iccp, which reports the state of the TCP connection, the BFD-based liveness detection, and the backup liveness peer status, plus the client applications that have joined the redundancy group:

user@switch> show iccp
Redundancy Group Information for peer 10.50.1.2
  TCP Connection       : Established
  Liveliness Detection : Up
  Backup liveness peer status: Up

  Redundancy Group ID   Status
  1                     Up

Client Application: l2ald_iccpd_client
  Redundancy Group IDs Joined: 1
Client Application: lacpd
  Redundancy Group IDs Joined: 1

Read it top to bottom. TCP Connection is the ICCP session itself. Liveliness Detection is BFD - spelled with Junos's own spelling of the word. Backup liveness peer status is the out-of-band verdict and is the one people miss, because it is the only field that still means something when the first two are down. Client Application blocks confirm that the L2 address-learning daemon and LACP have both joined the redundancy group; if lacpd is missing, the LACP side of the split-brain prevention is not in play no matter what the top of the output says.

Useful companions are show multi-chassis mc-lag for the MC-LAG interfaces and their states, show lacp interfaces ae0 to see the system ID actually in use on the wire, and show log messages | match iccp to reconstruct when each transition happened.

Split-Brain Prevention and LACP System ID Behavior

During a split-brain, both active and standby peers change their LACP system IDs. The connected server accepts the LACP system ID of the first link that comes up and brings down the other links carrying a different system ID - so only one MC-LAG peer forwards traffic. When the ICCP connection is active again, both peers return to the configured LACP system ID.

Juniper documents the failure matrix: with ICCP down and ICL down or up, an active backup liveness detection changes the LACP system ID to the default value for both active and standby multichassis aggregated Ethernet interfaces; an inactive one leaves the system ID unchanged.

The mechanism is worth stating in plain language, because it is the elegant part of the design. Both peers normally present the same configured system ID to the server, which is what makes the dual-homed LAG look like a single switch to the server's LACP implementation. The moment ICCP fails, that agreement is suspect: the two switches can no longer verify each other's state, so continuing to claim a shared identity would let both of them be active. Backup liveness detection therefore triggers a deliberate identity split - each peer reverts to its own default system ID and LACP admin key. The server now sees two distinct systems offering it links, and LACP resolves the conflict the way it always does: the first system ID to bring a link up wins, and the links from the other system are aggregatable-but-not-selected, so they stay out of the forwarding path. Traffic flows through exactly one peer, and no duplicated frames reach the server.

The matrix, as documented, is short enough to memorise:

The last row is the one that answers the most common question about this feature: while ICCP is healthy, the backup liveness check does not run at all. It is not a heartbeat that runs constantly and is ignored; it is a fallback process that starts when it is needed.

Timer and Topology Recommendations

• Keep the ICCP liveness-detection interval at 8 seconds or more when ICCP runs over an IRB interface, so GRES works seamlessly; single-hop BFD is fine over a dedicated physical link.
• Use the peer loopback address for ICCP peering so a direct link failure does not reset the session.
• Configure the master-only statement on the management interface IP so the connection is not reset during GRES.
• Use different interfaces and different FPCs for the ICL and ICCP paths.

A few more rules that come out of how the feature is built:

• Put the backup liveness IP in a different failure domain from the ICL. The whole point is to survive the loss of the in-band path. If backup-peer-ip is reachable only across the ICL, the feature cannot do the one job it exists for.
• Leave headroom in the BFD multiplier. Detection time is the negotiated interval multiplied by the multiplier, so a 1000 ms interval with a multiplier of 3 declares failure after roughly 3 seconds of loss. Tightening that to 300 ms over a shared management network buys speed you do not need and buys it with false positives.
• Remember that the out-of-band path is now part of a redundancy design. Every ACL, NAT rule, firewall policy and MTU change on the management network is a change to the MC-LAG's split-brain protection.
• Test it before you need it. The only meaningful test is to disable ICCP in a maintenance window and confirm that the LACP system ID changes on both peers, that exactly one peer's links are selected by the server, and that traffic continues without duplication.

Operational Notes and Edge Cases

The status can go stale after a peer reboot. Juniper documents this explicitly: when an MC-LAG node is rebooted with ICCP disabled on it, the backup liveness process on the other node remains active even though the remote link goes down, because the CLI shows the last action the backup liveness process performed and the peer cannot read the rebooted node's current status. There is no traffic loss, and the process itself is not running continuously - it starts when initiated and stops when it is not needed. The practical lesson is that Backup liveness peer status is a record of the last known verdict, not a live heartbeat, so corroborate it with the LACP state before drawing conclusions.

Backup liveness detection is not a substitute for BFD. The two solve different problems. BFD detects ICCP session failure quickly and stays in the in-band path; backup liveness detection survives the loss of the in-band path entirely. A robust design uses both, and then verifies with the failure matrix that the combination produces exactly one active forwarding peer in every row.

A tunnel is not an out-of-band path. If the management network is carried over the data plane - for example a management VLAN that rides the ICL - then the "independent" path fails at the same moment as the thing it is supposed to protect against.

Deployment Checklist

1. Confirm MC-LAG is configured on both peers with identical system-id and admin-key.
2. Verify layer 3 reachability between the two management interfaces, in both directions, with the exact source IP you intend to use.
3. Configure backup-liveness-detection backup-peer-ip on both peers - a one-sided configuration is a silent failure, because the peer that is not configured simply will not run the check.
4. Optionally add authentication-key on the ICCP peer statement, matching on both sides.
5. Configure BFD for ICCP failure detection and confirm Liveliness Detection : Up in show iccp.
6. Confirm Backup liveness peer status reports Up, and that both lacpd and the L2 address-learning client have joined the redundancy group.
7. Record the configured LACP system ID and the default system ID for each peer, so a future incident review can tell instantly whether the failure matrix has been entered.
8. Rehearse the failure in a window: drop ICCP, observe the system ID change and the link selection on the server, restore, and confirm both peers return to the configured system ID.

Frequently Asked Questions

Does backup liveness detection run all the time? No. It runs only while the ICCP connection is down; if ICCP is established, or BFD is configured and up, the peer is considered up and the backup check is not performed.

What happens if I configure it on only one peer? The configured peer will run the check and, depending on the timers, may draw a conclusion the other peer never agreed to. Configure it on both - the documentation is unambiguous about this.

Why does the LACP system ID change at all? Because a shared system ID is an assertion that both switches are one logical device. When ICCP is down, that assertion cannot be verified, so each peer reverts to its own default identity and lets LACP on the server pick one.

Will the server lose traffic when the system ID changes? The server brings down the links that carry a different system ID and keeps the first one that came up, so the LAG loses members rather than the whole bundle. Some LACP renegotiation blip is expected; duplicated frames are not.

Do I still need BFD if backup liveness detection is configured? Yes, if you want fast detection of ICCP failures rather than only correct behaviour after them. Backup liveness answers "is the peer alive"; BFD answers "is the ICCP session still working" much faster.

Related articles: MC-LAG ICCP failure liveness and LACP system ID, ICCP, BFD and LACP system ID deep dive, MC-LAG ICCP failure packet loop troubleshooting and Junos BFD liveness detection timers and MC-LAG integration.

原文链接：https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/backup-liveness-detection-edit-protocols-iccp-peer-qfx-series.html