Dell OS10 VLT Troubleshooting: show vlt Commands & Failures

Virtual Link Trunking on Dell OS10 looks simple until a port-channel comes up as up but inactive. VLT is not stacking: the two peers keep independent control planes, synchronise MAC and ARP data over the VLT interconnect (VLTi), and rely on a separate backup heartbeat to decide whether the peer is actually alive. Almost every VLT incident traces back to one of those three paths. This guide lists the verification commands, the failure modes they expose, and the behaviors that surprise people during failover testing.

How VLT works: the three paths that fail

Before reading a single line of command output it is worth being precise about what VLT is and what it is not. Two OS10 switches in a VLT domain do not merge into one switch. Each keeps its own control plane, its own forwarding table and its own management address; they simply agree to present themselves to attached devices as one logical LAG endpoint. Three separate paths carry that agreement, and each of them fails in a different way.

• VLT interconnect (VLTi) — a port-channel, usually two to four links, that carries ordinary data traffic between the peers plus internal signalling: MAC address synchronisation, ARP synchronisation, VLT port state and, when it is enabled, peer-routing reachability. A VLTi that is up but congested produces symptoms that look like data-plane bugs — intermittent loss, MAC-flap alarms, servers that drop one path for a few seconds.
• Backup heartbeat — a small out-of-band path, in most designs the management network, over which the peers exchange keepalives. Its only job is to answer one question: is my peer still alive? It is the path administrators most often forget to make redundant.
• Peer-routing path — when hosts use the domain's shared gateway address and a packet arrives on the peer that has no route for it, that peer must be able to hand the packet across the VLTi to the peer that does. Peer-routing rides on the VLTi like everything else, which means a VLTi problem is also a routing problem.

Keep those three paths in mind and show vlt output stops looking like a wall of text: every field maps back to exactly one of them.

Establish the baseline before touching anything

Never troubleshoot a VLT port-channel before proving the VLT domain itself is healthy:

show vlt 1
show vlt 1 mismatch
show vlt 1 backup-link
show vlt 1 vlt-port-detail
show vlt 1 peer-routing

show vlt 1 must show a role (primary/secondary), an active VLTi and a peer that is up. show vlt 1 mismatch compares the peers and reports VLT-MAC, peer-routing, VLAN and VLT-VLAN mismatches — these usually come from a half-applied change or a software-version difference. show vlt 1 vlt-port-detail is the money command: local and peer status must both be up for every VLT port-channel.

Reading show vlt output field by field

Two habits make this faster. First, capture the healthy output to a file the day the domain is built, so a broken output has something to be compared against. Second, always collect from both peers — a large share of VLT faults are a disagreement between the two switches, and a disagreement is only visible from two vantage points.

Case 1: port-channel is up but carries no traffic

show lacp interface ethernet 1/1/1
show lacp port-channel interface port-channel 10
show interfaces port-channel 10 status

Symptoms are OUT_OF_SYNC on the member link, collecting/distributing false, or PDUs sent but none received. Work through it in this order:

• LACP mode and operational key must match the partner device (active/active is the safe default).
• The VLTi must be up before any VLT port-channel can synchronise — if VLT is in down, fix that first.
• VLAN and switchport settings must be identical on both peers for the same VLT port-channel.
• If PDUs leave one side but never arrive on the partner, the fault is on the attached device or the cabling, not in VLT.

Case 2: peer-link (VLTi) failure

This is documented behaviour and it catches people out: when the VLTi goes down and the peer is still reachable over the backup heartbeat, the secondary peer shuts its VLT port-channels down so that the two halves cannot both forward for the same MAC addresses. In a lab failover test where the administrator shuts every interface except management, the result is not a failover but a full outage — the heartbeat still says the peer is alive, so nothing takes over.

VLT-Secondary# show vlt br
Domain ID: 1   Role: Secondary
ICL Link Status: Down
HeartBeat Status: Up
VLT Peer Status: Link Down

Only when the heartbeat itself stops does the secondary declare the peer dead and bring its VLT port-channel up. If you need deterministic single-sided failover, test by shutting the VLTi and the backup destination, or configure the delay-restore timer to match your expectation.

Case 3: configuration drift on the VLTi

Do not add VLANs manually to the VLT interconnect. OS10 manages tagged and untagged traffic on the VLTi automatically; adding VLAN configuration to the interconnect port-channel has been observed to disrupt traffic. VLAN membership belongs on the VLT port-channel, not on the peer link. Use show vlt 1 mismatch and show running-configuration vlt to confirm what is actually programmed:

OS10# show running-configuration vlt
vlt domain 1
 discovery-interface ethernet 1/1/1
 discovery-interface ethernet 1/1/2
 primary-priority 4096
 vlt-mac 00:11:22:33:44:55
 backup destination 1.1.1.2
!
interface port-channel 5
 vlt-port-channel 5

Case 2b: peers healthy, backup heartbeat silently broken

The heartbeat is the least glamorous part of VLT and the one that bites hardest, because its failure stays invisible until the moment it is needed. By default OS10 runs the heartbeat in the management VRF. If backup destination points at an address that is only reachable across the data path, the heartbeat rides the same fibre as the VLTi and protects nothing at all.

show vlt 1 backup-link
show vlt 1 backup-link statistics
show ip interface brief | grep -i management

A more dangerous variant appears on two-node designs where the heartbeat is routed across the very fabric the VLT exists to protect. A single fabric failure then removes the VLTi and the heartbeat at the same instant. Both peers conclude that the other is dead, both bring their VLT port-channels up, and the attached servers suddenly see two active paths to two independent routers — a duplicate-IP or spanning-tree event rather than a clean failover. Keep the heartbeat on a physically separate path (a dedicated pair of ports, or the out-of-band management switch) and write down which cable it uses, because the next person to unplug it will not remember.

Case 4: VLT port-channel stays down long after a peer reload

After a peer reloads, VLT port-channels are deliberately held down for a short period so that the returning peer finishes learning MAC addresses and routes before it starts forwarding. Two timers control how long that takes, and both are worth understanding before declaring a fault.

show vlt 1 vlt-port-detail
show running-configuration vlt
show vlt 1 peer-routing

The delay-restore timer holds the port-channels down and then releases them. If it has been set far above the platform default, a reload that should recover in well under a minute looks like a permanent fault, and the engineer who set it has usually left the company. The peer-routing timeout carries the opposite risk: set it too short and routes are withdrawn during an ordinary reload, so traffic is dropped even though the peer was only rebooting; set it too long and traffic is black-holed while the domain waits for a peer that is never coming back. Match both values to the real reboot time of the platform, then prove it by reloading one peer inside a maintenance window and timing the recovery yourself.

Peer-routing and default gateway pitfalls

If hosts on both peers must route through either switch, peer-routing has to be enabled and consistent on both sides; a Peer-routing mismatch in the mismatch output is the usual reason why traffic dies when one peer is reloaded. The same applies to the peer-routing timeout during a peer restart: too short and routes are withdrawn during a normal reload, too long and traffic blackholes while waiting for a peer that is not coming back.

Case 5: duplicate MAC addresses and MAC-table churn

When both halves of the domain forward for the same MAC address, hosts lose connectivity in bursts rather than completely, and the fault is usually reported as "the network is slow" rather than "the network is down". Confirm it by looking up the same address on both peers and then at the VLT port state.

show mac address-table address 00:11:22:33:44:66
show vlt 1 vlt-port-detail
show interfaces port-channel 10 status
show vlt 1 | grep -i icl

Three causes cover most cases: a VLT port-channel that is up on one peer and down on the other, so only half the domain forwards and asymmetric paths create churn; a server dual-homed outside VLT with one un-bundled NIC on each switch; and a VLTi failure combined with a dead heartbeat, which lets both peers go active as described above. In all three the fix belongs at the VLT layer, not in the MAC table.

Layer 2 or Layer 3: pick one design and be consistent

VLT supports two broad designs. In the Layer 2 design, VLT port-channels carry VLANs between the two peers and all routing happens on core devices; the domain shares a system MAC and behaves like a single switch to spanning-tree. In the Layer 3 design, the peers route for directly attached hosts and use peer-routing plus a shared virtual gateway address so that either peer can forward. Both designs work; mixing them is what breaks.

The classic failure is a host whose default gateway is a VRRP or anycast address held only by one peer, while its port-channel load-balances across both. Return traffic arrives on the peer that does not own the gateway address, and if peer-routing is disabled that traffic is dropped silently. Enable peer-routing on both peers, confirm the shared gateway address exists on both, and verify with:

show vlt 1 peer-routing
show ip route vrf all
show ip interface brief

If you also run OSPF or BGP on the pair, the VLT domain changes how the two control planes advertise. See Dell OS10 OSPF configuration and Dell OS10 BGP configuration for the routing side.

A failover test that does not take the network down

VLT is often tested exactly once, in a change window, by shutting interfaces — which is the worst possible test, because it creates the double failure described in Case 2. A safer sequence, run one step per maintenance window, is:

• Confirm the baseline on both peers: VLTi up, heartbeat up, every VLT port-channel up locally and remotely.
• Fail one member link of a VLT port-channel and time the convergence. No traffic should be lost if the remaining members have capacity.
• Reload the secondary peer, timing recovery while watching VLT port-channel state on the primary.
• Only then test a VLTi failure — and only with the heartbeat also failed if deterministic single-sided takeover is the behaviour you want. Document the result, because this behaviour surprises auditors and new engineers equally.

For the same reason, spanning-tree settings on the VLT port-channels matter more than they appear to: a mismatched edge or portfast configuration in front of the domain produces exactly the intermittent loss that VLT gets blamed for. The checklist in Dell OS10 spanning-tree: RSTP and Rapid-PVST is worth running through before calling a VLT outage.

Collecting evidence for a support case

If the fault survives everything above, capture a fixed set of output before you open a case — ideally from both peers, within seconds of each other. Support engineers ask for the same list every time:

show version
show vlt 1
show vlt 1 mismatch
show vlt 1 backup-link
show vlt 1 vlt-port-detail
show running-configuration vlt
show lacp port-channel interface port-channel 10
show logging | grep -i vlt

Attach the outputs from before and after the incident if you have them, plus the software version from both peers. Most VLT cases that stall are stalled on a version mismatch or a missing show vlt 1 mismatch output, not on anything exotic. Dell OS10 firmware upgrade: ONIE versus in-place image install explains how to bring the two halves onto the same release, which resolves a surprising number of "VLT mismatch" tickets.

Related reading: Dell OS10 VLT peer-routing configuration, the VLT setup walkthrough, OS10 port-channel and LACP configuration and, for the multi-vendor equivalent, EVPN multihoming versus MLAG.

原文链接：https://www.dell.com/support/kbdoc/en-us/000102901/dell-emc-networking-os10-how-to-set-up-virtual-link-trunking-vlt