Unexpected MAC Learning on Catalyst 9000 Switches

When the gateway MAC address suddenly shows up on a user-facing access port instead of the uplink, the entire VLAN can lose connectivity — and on a network running 802.1x with MAC authentication bypass, the problem can stick until someone manually recovers the port. This article reconstructs a real Cisco TAC case on Catalyst 9300 switches where an endpoint was silently reflecting traffic sourced from the gateway back into the switch, and shows exactly how the engineers proved it using embedded packet capture (EPC) and SPAN, and why the security feature set made the MAC 'stick' to the wrong port. It is a masterclass in MAC learning fundamentals and endpoint-driven anomalies.

The reason this case is worth studying is that it inverts the instinct most engineers develop after a few years on the job. When something breaks in Layer 2, we suspect the switch. We reload it, we clear the MAC table, we blame a software bug. In this case the Catalyst hardware, IOS-XE software and the configured feature set all behaved exactly as designed; the anomaly was injected by an endpoint with a NIC driver defect. Learning to tell the difference is the real skill.

How Catalyst Switches Learn MAC Addresses

Catalyst switches learn MAC addresses on ingress based on the source MAC address (SMAC) of the received frame. The MAC address table is normally a trustworthy map of where an address lives — if a MAC is learned on an unexpected interface, the switch received a frame with that SMAC on that port. In very rare cases internal forwarding-plane reflection can also cause this, but the endpoint explanation should always be ruled out first.

The control plane detail that matters here is the source of that frame. A switch never invents an SMAC; it reads the 12 hex digits in the source field of an Ethernet header that physically arrived on a port. So when the gateway MAC appears on an access port, one of exactly two things happened: either something on that port transmitted a frame claiming to be the gateway, or the switch internally reflected a frame it received from the gateway back onto that port's forwarding path. Both are testable, and the capture below distinguishes them.

It is also worth remembering how the CAM table ages. A dynamic entry ages out after the MAC aging time — typically 300 seconds on IOS-XE — and is re-learned the next time the switch sees that SMAC on a port. Learning therefore follows the most recently received frame, which is why a single misbehaving endpoint can hijack a MAC entry even on a switch that is otherwise healthy.

The Problem: Gateway MAC Learned on Random Access Ports

In the reported case, endpoints in the data VLAN lost connectivity to hosts outside their subnet. The VLAN 2 gateway MAC was learned on a user interface (Gi1/0/2) instead of the expected uplink (Te1/1/1). The symptom appeared randomly across a multi-campus network, but a trend emerged: the same endpoint model was involved in every occurrence.

Because the access ports ran 802.1x with MAB fallback, the reflected gateway MAC triggered an authentication session and was programmed as a static entry. The security implementation then blocked MAC movement, so the switch could neither age out the MAC on the user port nor re-learn it on the uplink.

The blast radius is worth spelling out. The gateway MAC is the destination of every inter-subnet frame sent by every host in the VLAN. When the switch believes that MAC lives on Gi1/0/2, it delivers all off-subnet traffic to a single desktop port. Hosts cannot reach the router, DHCP renewals fail if the server is off-subnet, and the symptom presents as a total VLAN outage — even though the uplink to the router is perfectly healthy and Spanning Tree is stable.

Sequence of Events

1. MACs are learned on the expected interfaces — normal state.
2. The endpoint reflects traffic sourced from the gateway back into its switch port.
3. Port security treats the reflected MAC as a new host: it authenticates and the MAC is programmed as STATIC.
4. When the correct entry ages out on the uplink, security prevents re-learning.
5. Traffic for the whole local VLAN is impacted; the port needs shut/unshut to recover.

Step 3 is the crux of the case and the reason the outage persisted. In a network without 802.1x or port security, the reflection would cause a brief MAC flap: the entry would move to the user port, then move back to the uplink the next time the real gateway transmitted, and the VLAN would self-heal within one aging cycle. Adding a security feature that converts the provisional entry into a static one removes that self-healing path and turns a transient flap into a hard outage.

Proving It with Packet Capture

Use the Embedded Packet Capture (EPC) on Catalyst to catch inbound frames from the suspect endpoint:

Switch# monitor capture TAC interface gi1/0/2 in match mac host aaaa.bbbb.cccc any
Switch# monitor capture TAC start
Switch# monitor capture TAC stop
Switch# show monitor capture TAC buffer brief

Physical SPAN with a MAC filter is equally reliable:

Switch(config)# monitor session 1 source gi1/0/2 rx
Switch(config)# monitor session 1 filter mac access-group MACL
Switch(config)# monitor session 1 destination gig1/0/48

The EPC match clause is doing the heavy lifting: it selects only frames whose source is the gateway MAC, so a busy port that carries thousands of frames per second produces a buffer containing only the incriminating ones. In the TAC case, the capture showed inbound frames on Gi1/0/2 with a source MAC equal to the gateway — arriving from the endpoint, not from the network. That single result moved the investigation from "the switch is broken" to "the endpoint is reflecting."

To make the evidence airtight, capture the same filter on the uplink at the same time. The uplink will show the gateway transmitting legitimately, and the access port will show the same source MAC arriving from the wrong direction. Two captures, one conclusion.

For deeper work, SPAN is often the better tool when you want the whole frame rather than a summary, and you can combine it with a match-any filter to keep the mirror stream manageable. The trade-off between local SPAN, RSPAN and ERSPAN transports is covered in SPAN, RSPAN and ERSPAN: choosing a mirror transport.

Why the Security Features Made It Worse

This is the counter-intuitive part that surprises engineers in post-incident reviews. Features that exist to stop MAC spoofing behaved as configured and, in doing so, removed the natural self-recovery the network would otherwise have had.

• Port security learns a limited set of MACs per port and, in most configurations, prevents a learned MAC from moving to another port. That is exactly what you want against an attacker moving a MAC — and exactly what traps a reflected gateway MAC.
• 802.1x with MAB authenticates by MAC address when supplicant software is absent. A reflected frame is a valid MAB attempt, so the gateway MAC is admitted as an authenticated host and pinned to the port.
• Sticky learning converts dynamic entries into static ones. Static entries do not age out, so the bad entry survives until an administrator intervenes.

The lesson is not that these features are wrong — they are mandatory on access ports — but that they change the failure signature of a MAC-learning anomaly from a self-healing flap into a persistent outage. The verification workflow for sticky MAC and the different violation modes is in Cisco port security and sticky MAC: violation modes explained, and the full 802.1x/MAB configuration path is in Cisco 802.1X and MAB configuration: step-by-step CLI guide.

Confirming the Anomaly from the MAC Table

Before reaching for a capture, the MAC address table itself is a strong signal. Compare the offending entry against the expected uplink:

Switch# show mac address-table interface gi1/0/2
Switch# show mac address-table dynamic address aaaa.bbbb.cccc
Switch# show mac address-table count
Switch# show interfaces gi1/0/2 status
Switch# show authentication sessions interface gi1/0/2

An entry that shows the gateway MAC on an access port, combined with an authentication session bound to the same port, is the fingerprint of this failure mode. If the entry is type STATIC where dynamic entries dominate the table, security has pinned it — clear the port (shut / no shut) to force re-learning, then fix the endpoint.

Preventing a Repeat of the Reflection

Knowing the cause, the operational hardening is straightforward. First, inventory endpoints: when a specific model is implicated, a firmware baseline becomes a hard requirement, not a nice-to-have, and the model should be flagged in the CMDB so the next occurrence is recognised instantly. Second, decide what the network should do when a MAC appears where it should not. Two schools of thought exist — let the switch self-heal by keeping entries dynamic and allowing the move, or pin the entry and accept that a human must intervene. Security usually wins the argument, which is why MAB and sticky learning stay enabled, but that choice means you need monitoring that fires on the anomaly rather than waiting for a user to call.

A simple syslog alert is enough. IOS-XE can raise a message when a MAC moves between ports, and an EEM applet can turn that message into an automated action — a log email, a trap to the NMS, or even an automatic interface reset. The EEM pattern is covered in our guide to Cisco EEM applets with event syslog triggers. Third, keep the gateway MAC under observation: since it is the single address whose misplacement takes down a whole VLAN, a periodic check that it resolves to an uplink or port-channel and never to an edge port is cheap insurance and catches the next bad endpoint before a support ticket is filed.

Resolution and Key Takeaways

The ultimate fix was an endpoint firmware update — the reflection behavior was already known to the vendor. The Catalyst hardware, software and configuration behaved entirely as expected. The takeaway: if a MAC appears on an unexpected interface, the switch is almost certainly telling the truth about what it received.

A practical checklist for the next time you meet this symptom:

1. Confirm the offending entry in show mac address-table and note whether it is dynamic or static.
2. Check for an authentication session on the same port; security features change the failure signature.
3. Capture inbound frames on the port filtered by the suspect MAC — use EPC for a quick look, SPAN for full frames.
4. Capture the same filter on the expected uplink to prove the direction of the anomaly.
5. Rule the endpoint in or out before escalating to TAC or opening a hardware case.
6. Clear the stuck entry with a shut / no shut once the endpoint is fixed.

Also review our EVPN MAC-VRF validation on ACX7000 and the Arista EOS MLAG run book for related MAC/forwarding troubleshooting on other platforms. For the classic switch-side failure, where MAC flapping is caused by a real Layer 2 loop rather than an endpoint, see our guide to troubleshooting MAC flaps and Layer 2 loops on Catalyst switches.

原文链接：https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-switch/222771-understand-unexpected-mac-learning-on-ca.html