Cisco MPLS VPN Troubleshooting: VRF, BGP and Labels

MPLS VPN faults can hide in four different places — the VRF definition, the MP-BGP control plane, the PE-CE routing protocol or the label forwarding plane — and guessing which one wastes hours. This article adapts the official Cisco troubleshooting methodology for the classic three-route-reflector MPLS VPN topology into a step-by-step command sequence that walks you from VRF verification to an end-to-end VRF-aware ping.

Background: Why MPLS VPN Troubleshooting Is Different

In an ordinary IP network, a broken route means the packet has nowhere to go, and you see it immediately as a "network unreachable" or a black hole at one hop. In an MPLS L3VPN, the provider edge (PE) router holds the customer's route in a per-VRF routing table, exports it into a VPNv4 address family with a route distinguisher prepended, and forwards the customer packet wrapped in a two-label stack: an outer transport label that gets it across the backbone and an inner VPN label that identifies the egress VRF on the far PE. Every one of those stages is a possible failure point, and the failure usually looks like silence rather than an error.

The four planes to check, in the order that resolves problems fastest, are:

• Configuration plane — does the VRF exist, with the right route distinguisher (RD) and route targets (RT)?
• Control plane — did the customer prefix reach the remote PE as a VPNv4 route with a label?
• Edge routing plane — did the PE learn the prefix from the CE, and is the next hop for that VRF correct?
• Forwarding plane — is there a label-to-next-hop entry, and does the label stack swap correctly hop by hop?

Single-namespace IOS commands show you only the default VRF and will happily report a healthy network while the customer is down. Every command in this guide is therefore VRF-aware or VPNv4-specific.

Topology and Assumptions

The reference topology is the one Cisco uses in its MPLS VPN troubleshooting document: two customer sites connected through two PEs, a core of P routers that only run IGP and LDP, and a route reflector providing iBGP VPNv4 reachability between the PEs. The RD and RT in the examples are 65000:1, and the customer VRF is named Customer_A. Host 10.1.1.14 is a CE-side address, and 10.1.1.1 is the PE's VRF interface facing the customer. If your design differs — inter-AS option B, carrier-supporting-carrier, or a VRF-to-VRF (option A) handoff — the same planes apply, only the exact show commands change.

Step 1: Validate the VRF Configuration

show ip vrf [vrf-name]
show ip vrf detail vrf-name

Confirm the VRF exists with the correct route distinguisher and import/export route targets. Remember that VRF names are case sensitive: Customer_A is not customer_a. A missing VRF or wrong RT here breaks everything downstream.

The first command gives you the interfaces assigned to the VRF and the RD. The second one is where the real information lives: the RD, the export RTs, the import RTs, and the list of interfaces in the VRF. Three checks matter most.

The RD must be unique per VRF per router. Two VRFs sharing an RD on the same PE collapse into one VPNv4 table entry and produce route leaking that is extremely hard to diagnose from the CLI. Use show ip vrf on every PE and confirm the RDs are distinct.

The route targets must match in a mirror image. The export RT of the local VRF must be present in the import list of the remote VRF, and vice versa. If the customer is asymmetric — for example, a hub that imports everything but exports a narrower RT set — the RT design will look "wrong" but be intentional; check the design document before changing it.

At least one interface must be in the VRF. A VRF with no interfaces is advertised into the VPNv4 table but cannot forward; show ip vrf detail shows the interface list, and an empty list is a dead giveaway. Remember also that the interface must not have ip address dhcp or be shut; a shut interface still appears in the VRF but carries no routes.

Finally, confirm the VRF is actually defined on both the ingress and the egress PE. Provisioning tools that push configuration per-router are a common source of one-sided VRFs, and a one-sided VRF produces a perfectly healthy-looking control plane with nothing usable in the forwarding plane. For a refresher on how VRFs behave outside the MPLS context, see the Cisco IOS VRF Lite configuration guide.

Step 2: Check the VPNv4 BGP Control Plane

show ip bgp vpnv4 all
show ip bgp vpnv4 vrf Customer_A
show ip bgp vpnv4 all tags

Verify the VPNv4 table contains the customer routes and that labels are assigned. If the VPNv4 table is empty, the problem is MP-BGP: check show ip bgp neighbors for session state and address-family activation between PEs and the route reflectors.

Read the output in three passes. First, does the prefix appear at all? Second, does it have a valid next hop and a best-path marker (the > at the start of the line)? Third, does it carry a label? The tags variant of the command shows the VPN label assigned to each prefix — if the label column is empty, the PE has the route but cannot forward it.

If a prefix is missing on the remote PE but present locally, the fault is in BGP advertisement. Walk the path:

• show ip bgp vpnv4 all summary — is the VPNv4 session to the route reflector Established, and does it show prefixes received?
• show ip bgp neighbors <rr-address> — is the address-family vpnv4 unicast negotiated? A session can be Established in the IPv4 unicast family while VPNv4 was never activated, which is the classic "BGP is up, VPN is down" case.
• show ip bgp vpnv4 all neighbors <rr-address> advertised-routes — did the local PE actually send the prefix? Output filters such as an outbound route-map on the RR, a missing send-community extended, or a next-hop-self problem will stop it here.
• On the route reflector, show ip bgp vpnv4 all route-reflector or a check of the cluster-id confirms reflections are happening. A duplicated cluster-id across two RRs silently drops routes — see the BGP route reflector cluster-id guide.

Extended communities are the mechanism that carries route targets, which is why send-community extended is mandatory on every VPNv4 session. If RTs are stripped, the receiving PE cannot import the prefix into any VRF and the route appears in the VPNv4 table but never in show ip route vrf Customer_A. That specific symptom — present in VPNv4, absent from the VRF table — is almost always an RT or extended-community problem, not a BGP session problem.

When the VPNv4 session itself refuses to come up, the causes converge with ordinary BGP: TCP reachability to the peer's update-source, an ACL blocking TCP 179, mismatched authentication keys, or a flapping session caused by hold-timer expiry on a congested link. See BGP session stuck in Idle or Active and BGP neighbor flapping root causes for the systematic version.

Step 3: Validate PE-CE Routing

If the customer runs a protocol other than BGP at the PE-CE edge, apply the traditional show commands within the VRF context — for example show ip rip database vrf [vrf-name], or OSPF/EIGRP equivalents scoped to the VRF. Missing routes here indicate redistribution or filtering issues on the PE.

Concretely, the VRF-scoped commands you need are:

show ip route vrf Customer_A
show ip protocols vrf Customer_A
show ip ospf neighbor vrf Customer_A
show ip eigrp neighbors vrf Customer_A
show ip bgp vpnv4 vrf Customer_A

The routing table for the VRF is the arbiter. If the CE's prefix is in the local VRF table but not in the neighbour's, the fault is on the advertisement side; if it is in the VPNv4 table but not in the VRF table, the fault is RT/import side. If it is nowhere, the PE-CE adjacency is the problem.

Two subtleties cause most of the PE-CE failures in practice. First, redistribution between the PE-CE protocol and BGP requires an explicit redistribute statement inside router bgp with address-family ipv4 vrf Customer_A; a redistribution that is configured in the global address family does nothing for the VRF. Second, when the PE-CE protocol is OSPF you must watch the OSPF domain-id and the capability vrf-lite behaviour, because the same prefix learned from two different customer sites is otherwise treated as an intra-area route and the remote site's route is not installed.

Also verify the CE's default route. In hub-and-spoke VPNs the spoke CE usually receives a default from the hub via the PE; if that default is filtered by a distribute-list or an outbound route-map, the customer's symptom is "some sites reachable, internet not working", which looks like a backbone problem but is a policy problem at the PE.

Step 4: Test with VRF-Aware Tools

ping vrf Customer_A 10.1.1.14
traceroute vrf Customer_A 10.1.1.14 source 10.1.1.1

Use VRF-aware ping/traceroute to prove connectivity from the PE into the customer site. When tracerouting across the MPLS backbone, the absence of core router hops is normal if the backbone does not propagate IP TTL — configure mpls ip propagate-ttl if you need hop-by-hop visibility. For related control-plane issues, see BGP neighbor flapping root causes and the VXLAN MP-BGP EVPN design guide.

Run the ping from the PE closest to the source, then repeat it from the far PE toward the same host to isolate direction. A ping that succeeds in one direction and fails in the other is the signature of an asymmetric routing table or a one-way RT mismatch, not of a broken label path.

Extended ping gives you source-address control, which matters because the PE may have several addresses in the same VRF:

ping vrf Customer_A
Protocol [ip]:
Target IP address: 10.1.1.14
Extended commands [n]: y
Source address or interface: 10.1.1.1
Repeat count [5]: 20
Size [100]: 1500
Timeout in seconds [2]: 2

Two useful variants: a large-size ping to test whether the customer MTU survives the label stack, and a repeated ping with a low interval to detect intermittent drops that a five-packet test averages away.

Step 5: Verify the Label Forwarding Plane

Once control plane and PE-CE routing are clean, prove the data plane on the P routers and the PEs:

show mpls forwarding-table
show mpls forwarding-table vrf Customer_A
show mpls ldp neighbor
show mpls interfaces

On the ingress PE the forwarding entry for the customer prefix shows an outgoing label (the transport label) and no label operation on the inner VPN label — that is the imposition of the label stack. On the egress PE, show mpls forwarding-table shows the VPN label with a "Pop tag" or "Aggregate" operation pointing at the VRF. If the transport label's next hop is unreachable, the entry will read "no route" or be missing entirely, which means the IGP or LDP has a hole.

LDP-specific checks are worth knowing because LDP failures are silent. show mpls ldp neighbor should list every core link; a missing neighbour means the LDP session never formed, usually because the interface's IGP is down, the LDP router-id is not reachable, or an ACL blocks UDP/TCP 646. A label-switched path that exists but never receives traffic points back at the routing table, not at LDP.

For a broken path in the middle of the backbone, traceroute with MPLS extensions on the P routers (or show mpls forwarding-table <prefix> detail) walks the stack swap hop by hop. Note that the 15-label limit in the IP header is a non-issue for L3VPN — a two-label stack has enormous headroom.

Step 6: MTU, PMTUD and Fragmentation

Small pings works, large transfers stall — that pattern is the classic MPLS MTU symptom. The label stack adds 4 bytes per label to the frame, so a 1500-byte customer packet becomes 1504 or 1508 bytes on the wire. If the provider core is exactly 1500 bytes end to end, oversized packets are dropped silently because the DF bit prevents fragmentation.

Check the interface MTU and the MPLS MTU on the core links:

show interface serial0/0 | include MTU
show mpls interfaces detail
ping vrf Customer_A 10.1.1.14 size 1500 df-bit repeat 10

The usual fixes are to raise the core interface MTU (and keep the customer-facing MTU at 1500), or to ensure the provider core allows fragmentation of labelled packets with mpls mtu per interface. Verify after the change with a DF-bit ping at the customer's maximum payload size and confirm the PMTUD ICMP messages are not being filtered by an ACL anywhere between the PE and the CE — filtering ICMP type 3 code 4 breaks PMTUD and produces a hard-to-explain "web pages load but large downloads stop" report.

Step 7: Common Failure Matrix

• VRF present, no routes anywhere: interface not in the VRF, or the VRF exists only on one PE.
• Routes in VRF table locally, missing remotely: redistribution into address-family ipv4 vrf, outbound route-map, or a PE-CE adjacency that is down.
• Route in VPNv4 table but not in the remote VRF table: RT mismatch or missing send-community extended.
• Route in remote VRF table, no forwarding: VPN label missing, or show mpls forwarding-table shows an unreachable transport next hop — an IGP/LDP problem.
• Ping works, TCP sessions hang: MTU/PMTUD, as described in Step 6.
• Flapping only under load: hold-timer expiry or a congested link dropping BGP keepalives; check interface error counters and QoS policy on the egress.

Work the list top to bottom. Each row corresponds to a plane you have already verified with the commands above, so by the time you reach the bottom you know which layer to change instead of guessing.

FAQ

Do I need a separate loopback per VRF? No. The BGP update source for VPNv4 is the global loopback, and the VRF's own addresses are local to the customer. Adding a loopback to a VRF only confuses the next-hop resolution.

Why does show ip route vrf show the route as "B" with a next hop of 0.0.0.0? That is the VPN label route pushed by MP-BGP — normal for a locally originated or directly connected customer prefix advertised into VPNv4.

Can I have the same customer prefix in two VRFs? Yes, that is the point of the RD. The RD makes the two prefixes distinct in the VPNv4 table; the RT decides which VRFs import them.

Is mpls ip needed on the customer-facing interface? No, only on the provider core interfaces that carry labelled packets. Enabling it toward the CE leaks labels into the customer network and is a common misconfiguration.

How do I check the whole path for one prefix fastest? Start from the egress PE: show ip route vrf <vrf> <prefix> then show mpls forwarding-table <prefix>, and work backwards to the ingress PE. Following the packet is faster than following the configuration.

原文链接：https://www.cisco.com/c/en/us/support/docs/multiprotocol-label-switching-mpls/mpls/13734-mpls-vpn-tsh.html