Junos MC-LAG Split-Brain: Backup Liveness Detection Config

A split-brain is the worst failure mode of an MC-LAG: both peers lose ICCP connectivity but keep forwarding, so the downstream server receives duplicate frames from two active switches. Junos OS mitigates this with backup liveness detection (BLD), an out-of-band keepalive channel between the peers over the management network that runs only when the ICCP connection is down.

This article explains how BLD works, how to configure it on both peers, which supporting statements — session establishment hold time, master-only, init-delay-time and prefer-status-control-active — make the MC-LAG converge cleanly instead of looping, and what the mechanism does not protect against. That last part matters more than the configuration: backup liveness detection answers the question "is my peer still alive?", and a surprising number of MC-LAG incidents are caused by a peer that is perfectly alive but no longer reachable across the interchassis link.

MC-LAG building blocks: what has to fail for a split-brain

Four components make up a Junos MC-LAG, and each one fails differently:

• MC-AE (multichassis aggregated Ethernet) — the bundle that terminates on both peers and connects to a dual-homed server or downstream switch. Both peers have membership in the same LAG, which is what makes the pair look like one switch to the attached device.
• ICL (interchassis link) — the link between the two peers that carries traffic when a frame ingresses on one peer and must egress on a port belonging to the other.
• ICCP (Inter-Chassis Control Protocol) — the control session between the peers, usually carried over the ICL or a dedicated path. ICCP synchronizes MAC and ARP information and coordinates which peer is forwarding.
• Redundancy groups and status control — an MC-LAG is typically configured with one peer as status-control active and the other as standby, so that Layer 3 and multicast behavior is predictable.

In normal operation both peers forward Layer 2 traffic and ICCP keeps their databases in sync. The split-brain appears when the peers can no longer talk to each other over ICCP but have no reason to stop forwarding locally. From the server's point of view, the LAG has two live partners with the same system identity. Broadcast and unknown-unicast frames are replicated in both directions, MAC address learning alternates between two chassis, and the duplicate forwarding can be severe enough to be mistaken for a loop in the access layer.

What Backup Liveness Detection does

When the ICCP connection is operationally down, each MC-LAG peer sends liveness detection requests to the other over the management link. If a peer does not respond within the configured time for a given number of consecutive attempts, the liveness check fails and the failure action is implemented — the surviving peer takes over cleanly instead of both peers staying active. Backup liveness detection must be configured on both peers hosting the MC-LAG, and the failure action only applies while ICCP itself is down.

Because the probes travel over the out-of-band management network rather than over the ICL, the mechanism survives the exact failure it is designed for: a broken, congested or badly cabled interchassis path. The backup-liveness-detection statement was introduced in Junos OS Release 12.2, and the option to point it at a non-default management routing instance was added later (Junos OS Release 24.2R2), which matters in environments where the out-of-band network lives in its own VRF.

When the liveness check runs — and when it does not

Note that when BLD is configured, peer status is always considered up while the ICCP TCP connection is established, or while BFD is configured and the peer is up. The backup liveness check only runs when the ICCP connection is down. In other words, BLD is a fallback, not a parallel supervisor: as long as ICCP is healthy it has nothing to say, and its verdict is only consulted during an ICCP outage.

There is one documented edge case worth knowing before you interpret show output during an incident. When an MC-LAG node reboots with ICCP disabled on it, backup liveness on the other node continues to display the last state it recorded. The process is not running continuously on the rebooted node, so the other node cannot read a fresh status. There is no traffic loss, and the displayed state is stale rather than wrong.

Configuration on both peers

user@switch-1# set protocols iccp local-ip-addr 172.16.32.5
user@switch-1# set protocols iccp peer 172.16.32.6 session-establishment-hold-time 50
user@switch-1# set protocols iccp peer 172.16.32.6 redundancy-group-id-list 1
user@switch-1# set protocols iccp peer 172.16.32.6 backup-liveness-detection backup-peer-ip 10.92.76.4
user@switch-1# commit

On switch-2, mirror the configuration with the local and remote addresses reversed. The backup-peer-ip is the address of the peer's management interface — the out-of-band path — and it must be reachable when the in-band ICCP path has failed. If that address is only reachable inside a dedicated management routing instance, name it explicitly so the probes are sent in the right table:

user@switch-1# set protocols iccp peer 172.16.32.6 backup-liveness-detection backup-peer-ip 10.92.76.4
user@switch-1# set protocols iccp peer 172.16.32.6 backup-liveness-detection routing-instance mgmt-vrf
user@switch-1# commit
user@switch-1> show configuration protocols iccp | display set

Two rules apply to every deployment: configure BLD on both peers, and make sure the two management addresses are reachable from each other without traversing the MC-LAG itself. A management path that depends on the LAG you are protecting will fail at the same moment as the LAG.

Supporting statements that make failover clean

Backup liveness detection is one piece of a set of statements that together decide how the pair behaves when things break.

• session-establishment-hold-time (Juniper recommends 50 seconds) controls how long a chassis waits before taking over as primary at the ICCP session level, giving the peer time to come back. A value that is too short turns a transient ICCP flap into a full failover.
• master-only is configured on the IP address of the management interface used for backup liveness detection, on both the primary and backup Routing Engines. This prevents the BLD connection from being reset during a graceful Routing Engine switchover (GRES) on the remote peer.
• init-delay-time on the MC-AE delays bringing the interface up after a peer reboot, letting protocols converge first. A typical value is 240 seconds, which is long enough for routing protocols and the LAG to settle before the bundle starts forwarding.
• force-icl-down under the MC-AE events configuration forces the interchassis link down when ICCP is lost, which stops the secondary peer from forwarding traffic it can no longer deliver. Without it, an ICCP-only failure leaves both peers active.
• prefer-status-control-active states the intent that the configured active peer should be preferred whenever ICCP information is incomplete, so the pair does not flip roles after a partial failure.

user@switch-1# set interfaces ae0 aggregated-ether-options mc-ae init-delay-time 240
user@switch-1# set interfaces ae0 aggregated-ether-options mc-ae events iccp-peer-down force-icl-down
user@switch-1# set interfaces ae0 aggregated-ether-options mc-ae status-control active
user@switch-1# commit

Guidelines for a resilient setup

• Configure ICCP peering to a loopback or IRB address rather than a physical interface, so individual link failures do not tear down the ICCP session.
• Use an aggregated Ethernet interface for the ICCP/ICL links where possible, with member links spread across different FPCs, to reduce the chance of a single-card failure isolating the peers.
• Configure the master-only statement on the IP address of the management interface used for backup liveness detection, on both the primary and backup Routing Engines, so a GRES event on one peer does not reset the liveness session.
• Separate the ICL and ICCP interfaces — best practice is to use different ports and different FPCs for the interchassis link and the ICCP connection.
• Keep the management path independent — different switches, different power feeds, and no dependence on the MC-LAG or on the same line cards that carry the ICL.
• Document the failure action you expect. BLD triggers a takeover when the peer stops responding; force-icl-down stops forwarding when ICCP is lost. Both should be in the runbook with the expected show output.

Walkthrough: following a split in real time

1. The ICL fails — a fiber strand, an optic, or the aggregated Ethernet that carries it. Traffic that used to cross between peers now has nowhere to go, but each peer's MC-AE membership is still up.
2. ICCP loses its TCP session. Local status on both peers may remain active, and the attached server keeps sending and receiving on both halves of the LAG.
3. Backup liveness detection starts probing the peer's management address over the out-of-band network, because ICCP is down.
4. If the peer answers, the failure is an ICCP/ICL failure with a healthy peer. BLD will not declare the peer down — the peer is alive. This is precisely the case where force-icl-down is required, because otherwise nothing stops the peer from forwarding traffic it can no longer hand over. This is the most commonly misunderstood MC-LAG failure mode.
5. If the peer does not answer, the failure is a peer failure: the peer is gone, its management path is broken, or its control plane is stuck. BLD fails, the failure action is implemented, and the surviving peer takes over cleanly instead of duplicating forwarding.
6. When the failed component returns, the init-delay-time holds the MC-AE down for the configured interval so protocols converge before traffic resumes, and session-establishment-hold-time gives the ICCP session a chance to re-establish without a second role flip.

Verifying liveness configuration and state

user@switch> show configuration protocols iccp
user@switch> show iccp
user@switch> show iccp peers
user@switch> show lacp interfaces ae0
user@switch> show log messages | match "iccp|liveness"

Read the output in this order: confirm the configuration exists on both peers (show configuration protocols iccp), confirm the ICCP session state and the peer status (show iccp, show iccp peers), confirm the LAG is bundling the ports you expect (show lacp interfaces ae0), then look for liveness and ICCP transitions in the log. A common finding during audits is that BLD is configured on one peer only — the failure action then depends on which side of the pair survives, which is exactly the kind of asymmetry that makes an incident unpredictable.

If you configure prefer-status-control-active on both nodes, pair it with ICCP peering over loopback addresses and backup liveness detection on both peers; otherwise the design loses its protection against the exact failure it is trying to survive.

Failure modes backup liveness detection does not cover

• ICL failure with a live peer. BLD stays quiet because the peer responds; only force-icl-down or ICL-PL protected links prevent the wrong peer from forwarding.
• A management path that shares fate with the ICL. If the out-of-band network rides the same fiber bundle, the same line card or the same switch as the ICL, both fail together and BLD never gets a chance to answer.
• Double failover. ICCP down plus the active MC-AE down leaves the standby peer with no ICCP information and no way to coordinate correctly.
• Asymmetric configuration. BLD, force-icl-down or master-only present on one peer only.
• Stale state after a reboot with ICCP disabled, as described above — the CLI shows the last recorded state, not a live check.

Related reading: Juniper MC-LAG design and ICCP hold timers, MC-LAG ICCP failure: packet loop troubleshooting, backup liveness detection on MC-LAG, BFD liveness timers with MC-LAG and ICCP failure and LACP system ID behavior.

Original article: backup-liveness-detection (Junos OS CLI Reference) | Junos OS MC-LAG User Guide (PDF)