BGP in EVPN Data Center Fabrics: Design Choices

2026年8月26日 LOG admin

In an EVPN data center fabric, BGP is both the underlay routing protocol and the overlay control plane, so the choice you make at design time decides how far the fabric scales and how much operational complexity you sign up for. This article works through the three mainstream designs described in the ipSpace.net design guidance - IBGP with an IGP underlay, pure EBGP in the RFC 7938 style, and IBGP EVPN running over an EBGP underlay - and explains why next-hop-unchanged is the constraint that decides whether the EBGP options are even available on your hardware.

EVPN and BGP: The Basic Relationship

EVPN uses an additional BGP address family (l2vpn evpn) to carry MAC and IP reachability;
the next hop in an EVPN advertisement points at the egress LSR or VTEP, not at the device that passed the route along;
the underlay routing protocol is responsible for computing the best path to that BGP next hop;
a router that modifies the EVPN next hop must also perform data-plane decapsulation and re-encapsulation, because it has just promised to be the tunnel endpoint.

That last bullet is the whole story. In a classic MPLS L3VPN, only the PE at the edge of the tunnel changes the next hop, and it is a real data-plane device. In a VXLAN EVPN fabric, the spine is not a VTEP: it forwards IP packets between leaf loopbacks and nothing else. If a spine rewrites the EVPN next hop while reflecting or re-advertising a route, it tells every other leaf "send the VXLAN-encapsulated frame to me", and the frame dies there. This is why an EVPN fabric cannot simply inherit the next-hop behaviour of a generic BGP design.

Design Option 1: IBGP with an IGP Underlay

Run OSPF or IS-IS as the underlay and exchange EVPN routes between leaves with IBGP. This is the easiest option to learn, the easiest to troubleshoot with familiar tools, and it works on essentially every platform that speaks EVPN at all. The IGP advertises loopbacks and point-to-point link addresses, the leaves build IBGP sessions to route reflectors, and the EVPN next hop - a loopback address - is already in the IGP, so no next-hop manipulation is required.

IBGP needs either a full mesh or route reflectors; at fabric scale, use route reflectors. Spines are the natural place to put them because they already have a session to every leaf, but you can also place RRs on bare-metal servers or virtual machines. If you do move them off the switches, make sure the infrastructure hosting the RRs does not itself depend on the fabric being up, or you have built a fabric whose control plane cannot recover from its own failure.

The costs are real but familiar: two protocols to operate, an IGP that must be tuned consistently across every leaf and spine, and a topology where the underlay and overlay failure domains are coupled. In exchange you get vendor-neutral behaviour, mature tooling, and no dependency on a vendor-specific next-hop knob.

Design Option 2: Pure EBGP with No IGP

Here every device runs BGP and nothing else. Each leaf peers with each spine, the loopback addresses are advertised in BGP itself, and there is no IGP to tune. Because the spines do not participate in the data plane, EVPN advertisements crossing an EBGP session inside the fabric must not have their BGP next hop changed - the design needs something equivalent to inter-AS Option C next-hop preservation.

Before committing to this design, confirm what your vendor actually implements. Some platforms support EVPN only over IBGP, some do not support next-hop-unchanged for the EVPN address family over EBGP, and some support it but only through convoluted per-neighbour configuration that is easy to break during a maintenance window. Any of those cases means falling back to IBGP with an IGP underlay.

The appeal of the pure-EBGP model is a single protocol, no IGP scaling limits, and a natural scaling path - but it also puts every MAC and IP route of the fabric into the spines' BGP tables, so the spine control plane becomes the sizing question. Check the platform's EVPN route scale before you size the fabric, not after.

Design Option 3: IBGP EVPN over an EBGP Underlay

This hybrid uses EBGP for the underlay in the RFC 7938 style and IBGP for the overlay EVPN routes. It is attractive when the underlay must not run an IGP but the vendor's EVPN implementation is happiest over IBGP - which is the common combination on platforms where EVPN next-hop-unchanged over EBGP is not available.

Two common variants exist. In the first, every leaf uses the same AS number, which requires allowas-in so that a leaf accepts a route whose AS path already contains its own AS. In the second, each leaf uses a distinct ASN but a local-as statement makes it appear to be in the route reflector's AS, so the IBGP overlay sessions come up without allowas-in. The local-as approach is tidier to operate because the AS-path manipulation is explicit and confined to the overlay session.

Route Reflectors: Placement and Redundancy

Once you commit to IBGP for the overlay, route reflector placement becomes an architectural decision, not a detail. The rules that keep it safe:

Use at least two RRs, and give them distinct cluster IDs so a leaf can tell the reflected copies apart.
Peer every leaf with every RR; do not chain RRs unless you have a hierarchy you genuinely need.
Keep RR sessions on the underlay addresses so that a data-plane failure does not take the control plane with it.
Do not run the RR role on a device whose failure also removes the only path to a VTEP loopback; the overlay and underlay planes should fail independently.
If you move RRs off the switches, size their CPU for BGP update churn, not for average load - the worst case is a leaf reboot storm, and that is when the fabric most needs the RR to stay up.
Set cluster-id explicitly instead of relying on the router ID, so that replacing an RR does not silently change the cluster identity.

AS Numbering, Autonomy and Path Selection

In the RFC 7938 model every device gets a unique ASN drawn from the private 4-byte range, rather than every leaf sharing one AS. The benefit is that the AS path length then becomes a meaningful distance metric: a leaf-to-leaf route crosses two ASes, and a directly attached spine shows one, which keeps best-path selection behaving sensibly without any extra configuration.

As soon as the AS paths of equal-cost paths differ, ECMP stops working unless you tell BGP to ignore the AS path when comparing paths. bgp bestpath as-path multipath-relax is the standard fix, and maximum-paths has to be raised far above the default before a leaf will install every spine as a next hop for a given prefix. Verify with the vendor's equivalent of show ip bgp <prefix> that all expected next hops appear, then confirm that the hardware forwarding table actually contains the ECMP entries - a route that looks multipath in the RIB and single-path in hardware is a classic EVPN underlay mistake.

Convergence and Failure Domains

Nothing in the choice of BGP design removes the need for fast failure detection. BFD on the underlay sessions detects a failed spine in milliseconds where BGP hold timers take seconds; on the overlay, EVPN route withdrawal plus ARP suppression determines how quickly hosts stop sending into a black hole. Whichever design you choose, decide explicitly how the fabric reacts to a spine failure (underlay reconvergence), a leaf failure (EVPN withdrawal and remote VTEP removal), and a link failure that leaves the session up but the path unusable (BFD or an IGP/EBGP combination that does not rely on the physical layer).

Keep the two planes honest with each other: in an IBGP-over-IGP design, an IGP failure removes next-hop reachability and the BGP routes go away without any BGP event. In a pure eBGP design there is no IGP to lean on, so the BGP session on the affected pair must be the detection mechanism. Both behave well - but only if you know which one you built.

Design Decision Tree

Can the fabric run an IGP underlay? → IBGP with an IGP is the simplest, most portable answer;
Want pure EBGP with no IGP? → confirm vendor support for next-hop-unchanged in the EVPN address family before you promise a date;
Prefix scale beyond what the switches' control plane can handle? → move route reflectors onto VMs; IBGP EVPN over an EBGP underlay may be the only workable option;
Vendor supports EVPN only over IBGP and the underlay must be EBGP? → use the hybrid with local-as;
Unsure? → start with IBGP over an IGP. It is the design you can debug at 03:00 with tools you already know, and moving up to EBGP later is a project, not a rewrite.

Frequently Asked Questions

Do spines have to be VTEPs? No. In all three designs the spine can be pure IP, which is what makes the next-hop question so important - the spine never terminates VXLAN, so it must never claim to.

Can I mix designs in one fabric? You can, but every boundary becomes a translation point where next-hop behaviour differs. Keep one design per fabric and treat a second design as a separate fabric joined by a border gateway.

Is ARP suppression required? No, but it reduces BUM flooding and BGP update churn, and it is close to mandatory once the number of hosts per leaf grows into the thousands.

How do I test the next-hop behaviour? Build a two-leaf, one-spine lab, advertise a prefix from one leaf, and inspect the EVPN route and the VXLAN outer header on the other leaf. If the outer IP is the spine, the design is broken regardless of what the configuration guide says.