VXLAN with MP-BGP EVPN Control Plane: Design and Configuration

Original RFC 7348 VXLAN relies on multicast-based flood-and-learn: VTEPs discover each other from data traffic, and unknown hosts are flooded across the overlay. That works at small scale but struggles with scalability, and it forces multicast into the underlay. MP-BGP EVPN replaces this model with a standards-based control plane that distributes host MAC and IP reachability like any other BGP route. This article explains the architecture shift and walks through the Nexus 9000 configuration steps from Cisco's design guide.

The core problem with flood-and-learn is that the network has no idea where anything lives until traffic tells it. A VTEP learns a remote MAC only after it has received a frame from that MAC, and until then every unknown unicast, broadcast and multicast frame is replicated to every other VTEP in the VNI. In a two-rack lab that is invisible. In a fabric with 200 leaf switches and 5,000 endpoints per VNI, it is the difference between a network that converges predictably and one that melts down after a single Top-of-Rack reboot. EVPN turns reachability into a routing problem: the control plane knows the answer before the first packet ever needs to be forwarded, and the data plane simply follows the routes.

What MP-BGP EVPN Changes

• Control-plane learning — host MAC/IP routes are advertised over BGP instead of learned by flooding, minimizing overlay broadcast.
• VTEP discovery and authentication — VTEPs learn each other through BGP and only accept VXLAN traffic from learned peers, blocking rogue VTEPs.
• Integrated bridging and routing — the EVPN address family carries both L2 (MAC) and L3 (IP) reachability, enabling symmetric IRB.
• ARP suppression — local VTEPs answer ARP on behalf of remote hosts from a suppression cache, cutting ARP flooding dramatically.

The practical consequences are what sell EVPN in production. Unknown unicast is no longer flooded once a host has been seen anywhere in the fabric, because every leaf already holds a Type-2 route for it. Broadcast domains stop being a fabric-wide liability, because BUM traffic is confined to the VNI and can be replicated via ingress replication rather than multicast. And because the same address family carries IP prefixes as well as MAC addresses, you stop running a separate routing protocol inside every tenant VRF — BGP carries both the Layer 2 and Layer 3 reachability in one session.

Why Flood-and-Learn Does Not Scale

Flood-and-learn has three structural weaknesses that EVPN eliminates. First, it has no mechanism for suppressing BUM traffic in the underlay — you either enable multicast (which means PIM, RPs and multicast state in the spine) or you accept head-end replication to every VTEP, which turns every ARP into N copies. Second, it depends on data-plane learning, so convergence after a failure is tied to traffic arrival rather than to a control-plane event. Third, it offers no clean way to enforce VTEP membership: a rogue or misconfigured VTEP that joins the multicast group can inject traffic into the overlay.

EVPN fixes all three. BUM replication can still be ingress replication, but the destination list is derived from the control plane rather than from the multicast group. Convergence is driven by BGP withdrawal and route update, so a failed leaf stops receiving traffic as soon as its routes disappear. And because VTEPs peer with each other over BGP, an unauthenticated device cannot simply appear in the overlay.

EVPN Route Types in Practice

EVPN defines several route types, but a data center fabric only leans on a handful of them. The table below maps each type to what it actually does on a Nexus 9000 leaf.

• Type-2 (MAC/IP Advertisement Route) — the workhorse. Advertises a host MAC, and optionally its IP and the L3 VNI, which is what makes ARP suppression and symmetric IRB possible.
• Type-3 (Inclusive Multicast Ethernet Tag Route) — signals which VTEPs are members of a given L2 VNI so that BUM traffic has a destination list.
• Type-5 (IP Prefix Route) — advertises an IP prefix (rather than a host) between VRFs, used for external connectivity and for summarizing tenant subnets.
• Type-1 (Ethernet Auto-Discovery) — used for multihoming and fast convergence in ESI-based designs.
• Type-4 (Ethernet Segment Route) — elects the designated forwarder among the peers that share an Ethernet segment.

If you only remember one thing about route types, remember this: Type-2 with an IP address in it is what lets a remote leaf route to a host without ARP. Without the IP field, you fall back to flooding to resolve every destination.

Key Constructs: VNI, RD, RT

EVPN uses route distinguishers (RDs) to keep identical routes unique across VRFs and route targets (RTs) to control import/export. On all-Cisco Nexus fabrics, NX-OS can auto-generate RD and RT values; in multi-vendor deployments, configure them manually for consistency.

Three identifiers show up again and again, and confusing them is the single most common source of EVPN misconfiguration:

• VNI — the VXLAN Network Identifier, the Layer 2 or Layer 3 segment identity carried in the VXLAN header. A VLAN is mapped to a VNI with vni <id> l2 or vni <id> l3.
• RD — the Route Distinguisher, an 8-byte value (typically router-id:number) that makes otherwise identical prefixes distinguishable in the BGP table. It is not used for import/export decisions.
• RT — the Route Target, a BGP extended community that decides which VRFs import which routes. This is your tenant isolation boundary, and it must match on both ends.

A useful mental model: the VNI is what the data plane puts in the packet, the RD is what makes the route unique in the BGP table, and the RT is what draws the lines between tenants. Get the RT wrong and tenants leak into each other; get the RD wrong and legitimate routes get rejected as duplicates.

Underlay Design for the VXLAN Overlay

Everything in the overlay rides on the underlay, and the overlay will inherit every underlay flaw. The underlay needs three things: loopback-to-loopback reachability between every VTEP, enough equal-cost paths that a single leaf or spine failure costs an acceptable fraction of bandwidth, and an MTU that accommodates the VXLAN encap.

VXLAN adds 50 bytes on top of the inner frame (8 bytes for the VXLAN header, 8 for UDP, 20 for the outer IP, 14 for the outer Ethernet) plus the original outer Ethernet header. The practical answer is to set the underlay MTU to at least 9000 on every leaf-spine link and every SVI or routed port in the path, then set the overlay's system mtu appropriately. A fabric that is fine for pings will fail for large TCP segments if one link in the path is left at 1500.

interface Ethernet1/1
  mtu 9216
  no switchport
  ip address 10.1.1.1/31
  no shutdown

RFC 7938-style eBGP as the underlay routing protocol is the default recommendation for a reason: it scales to thousands of nodes without tuning, it gives you ECMP at the leaf for free, and it keeps the failure domain tiny because each leaf only needs to know its directly connected spines.

Distributed Anycast Gateway

The distributed anycast gateway solves a problem that plagued older designs: in a hairpinned architecture, all inter-subnet traffic for a VLAN had to route through one designated switch, which both wasted bandwidth and created a choke point. With a distributed anycast gateway, every leaf hosting the VLAN configures the same virtual IP and the same virtual MAC on its SVI.

The result is that a host's default gateway is always local — one hop away — regardless of which leaf the host is attached to. Traffic is routed at the ingress leaf, then carried over the L3 VNI to the egress leaf, which does the final Layer 2 delivery. This is the foundation of symmetric IRB, and it is what makes the fabric behave like a single logical router with one hop per leaf.

fabric forwarding anycast-gateway-mac 0000.2222.3333
interface Vlan101
  no shutdown
  vrf member Tenant-A
  ip address 192.168.10.1/24
  fabric forwarding mode anycast-gateway

Because the anycast MAC is identical everywhere, a host that moves between leaves keeps its ARP cache intact and simply follows the EVPN Type-2 route to its new location. That host mobility — with no re-ARP, no gateway change and no manual intervention — is one of the strongest arguments for EVPN over a traditional three-tier design.

ARP and ND Suppression

ARP suppression is where the control plane stops being an optimization and starts being a capacity win. Because leaves know remote hosts' IP-to-MAC bindings from Type-2 routes, a leaf can answer an ARP request locally instead of flooding it across the VNI. The request never leaves the leaf, and the ARP broadcast domain stops being a fabric-wide event.

The same logic applies to IPv6 Neighbor Discovery, which is materially worse without suppression because of its multicast behavior. Enable both:

interface Vlan101
  ip arp suppression
  ipv6 nd suppression

Suppression is not free — it depends on the Type-2 routes carrying IP information, which in turn depends on correct L3 VNI mapping. If suppression is enabled but the IP field is missing from the advertisements, hosts silently fall back to flooding, and you lose the benefit without noticing unless you are watching counters.

Configuration Steps on Nexus 9000

1. Configure the tenant VRF and a Layer-3 VNI per VRF.
2. Map VLANs to Layer-2 VNIs and define their EVPN parameters.
3. Configure SVIs with the distributed anycast gateway (same virtual MAC + IP on every VTEP in the VNI).
4. Create the VXLAN tunnel interface and associate L2/L3 VNIs:

interface nve1
  source-interface loopback0
  member vni 20010
    ingress-replication protocol bgp
  member vni 20000
    mcast-group 239.1.1.1

5. Configure BGP with the L2VPN EVPN address family:

router bgp 65001
  router-id 10.0.0.1
  neighbor 10.0.0.2 remote-as 65001
    address-family l2vpn evpn
      send-community extended
  address-family l2vpn evpn

The two NVE members above illustrate the two replication options side by side: VNI 20010 uses BGP-based ingress replication, where the leaf builds its own copy list from the control plane, while VNI 20000 uses a multicast group in the underlay. Most modern fabrics standardize on ingress replication to keep the underlay free of multicast state, accepting the extra CPU from local replication in exchange for operational simplicity.

Tenant VRF and L3 VNI Mapping

The L3 VNI is what makes symmetric IRB work, and it is created per tenant VRF rather than per VLAN. Every VLAN in the same tenant VRF shares one L3 VNI, one RD and one set of route targets.

vrf context Tenant-A
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

vlan 101
  vn-segment 20101

vlan 501
  vn-segment 50001

The distinction between VLAN 101 and VLAN 501 in the snippet above matters: 20101 is the Layer 2 VNI that carries the tenant subnet, and 50001 is the Layer 3 VNI that carries routed traffic between leaves. Both live in the same VRF, and the leaf maps between them automatically once vni 50001 is configured under the VRF.

Design Options

MP-iBGP EVPN commonly uses route reflectors on the spine; MP-eBGP EVPN peers each leaf in its own AS with the spine preserving next-hop and RT attributes. Border leaves connect the fabric to external networks, redistributing tenant routes and advertising public subnets out — usually as summarized prefixes rather than /32 host routes.

The choice between iBGP with route reflectors and eBGP between tiers is mostly an operational one. Route reflectors keep every leaf in the same AS, which simplifies route-target policy and avoids inter-AS attribute manipulation, but it introduces a control-plane scaling concern as the fabric grows and requires careful redundancy planning for the reflectors themselves. eBGP between leaf and spine eliminates the reflector dependency, gives you natural next-hop-unchanged behavior with next-hop-unchanged on the spine, and lets each leaf be its own BGP domain — at the cost of more per-neighbor configuration.

Verification and Troubleshooting Commands

Verification should always start with the underlay and work upward. If the loopback-to-loopback BGP sessions are not Established, no amount of overlay debugging will help.

show bgp l2vpn evpn summary
show bgp l2vpn evpn
show nve peers
show nve vni
show l2route evpn mac all
show l2route evpn mac-ip all
show ip arp suppression-cache detail
show vrf

Read the output in order. show nve peers tells you which VTEPs are up and which VNIs they participate in. show l2route evpn mac-ip all shows the Type-2 routes actually installed in the forwarding table — if a host appears in show bgp l2vpn evpn but not here, the problem is local mapping rather than remote advertisement. If the ARP suppression cache is empty for a host that clearly exists, the Type-2 route is missing its IP field, which almost always traces back to an L3 VNI not being mapped to the VRF.

A quick isolation technique: ping the remote VTEP loopback, then ping across the L3 VNI, then ping the remote host. Whichever step fails first is the layer to debug, and it saves you from chasing a forwarding problem that is actually a routing one.

Common Design Pitfalls

• MTU undersizing — the underlay is at 1500, and large frames are dropped silently. Symptom: pings succeed, file transfers stall.
• Mismatched route targets — routes are advertised but not imported, so nothing crosses the fabric. Symptom: partial reachability that looks like a routing loop.
• L3 VNI not mapped to the VRF — host routes lack IP information, ARP suppression does nothing, and inter-subnet traffic still floods.
• Reusing the same L2 VNI across tenants — the VLAN-to-VNI mapping must be unique per tenant, or isolation collapses.
• Anycast gateway MAC mismatch — a leaf with a different virtual MAC breaks host mobility and produces intermittent return-path failures.

Scaling Considerations

Two numbers drive fabric sizing: the number of VNIs the platform can support, and the number of MAC/IP routes the control plane can carry. A leaf that holds 100,000 Type-2 routes is doing real work, and the convergence time after a spine failure scales with that table size. This is why summarization matters — advertise tenant subnet prefixes as Type-5 routes at the border rather than leaking individual /32 host routes into every leaf.

Ingress replication has a similar profile. Every BUM frame that arrives at a leaf must be copied to every remote VTEP in the VNI, so the CPU and replication engine become the constraint long before the links do. Keeping broadcast domains small and leaning on suppression is the standard mitigation.

Continue with Cisco EVPN/VXLAN Multisite 排障 for EVPN/VXLAN fault isolation and EVPN 数据中心 BGP 设计 for BGP design choices in EVPN fabrics.

For adjacent reading on this site, see the VXLAN EVPN multi-site border gateway design guide, which picks up where this article's border-leaf discussion leaves off, and the VXLAN MTU and fragmentation analysis for a deeper treatment of the 50-byte tax referenced above.

原文链接：https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/guide-c07-734107.html