TCAM and FIB Scaling: Why Routes Don't Install

A prefix can be present in the routing table, selected as best, and still never be programmed into the forwarding hardware — the control plane says the route exists, the data plane has no entry, and traffic takes the default route instead. The cause is almost always finite hardware resources: TCAM and FIB tables are fixed-size, divided into regions with per-region limits, and when a region fills up, additional entries are silently rejected. This article covers how to detect the condition, how to interpret the counters, and the design decisions that prevent it.

RIB versus FIB versus hardware

Three layers are involved in getting a route to a forwarding decision. The routing information base (RIB) is the protocol-level table where best path selection happens. The forwarding information base (FIB) is the derived table with resolved next hops and adjacency information. The hardware table — TCAM or an equivalent — is where the actual lookup happens per packet. Resource exhaustion affects the third layer, and the platforms differ in architecture: some families use a unified forwarding ASIC derived from the UADP line, others use a newer generation of silicon where the resource model and diagnostic commands differ.

Detecting the problem

Start by comparing the number of entries the software believes exist with the count installed in hardware. On Catalyst 9000 platforms the route summary gives exactly that comparison, per mask length.

switch# show platform software fed switch active ip route summary
Total number of v4 fib entries = 11348
Total number succeeded in hardware = 8186
Mask-Len 24 :- Total-count 10002 hw-installed count 8171
Mask-Len 30 :- Total-count 1331  hw-installed count 0

A gap between total and hardware-installed counts is the signature: routes exist in software and are not in the data plane. Note the mask-length breakdown — a specific prefix length failing while others succeed points at how the region is programmed, and /30 host subnets or /32 host routes are frequent offenders because they consume an entry each and multiply quickly.

switch# show platform hardware fed switch active fwd-asic resource tcam utilization
Codes: EM - Exact_Match, I - Input, O - Output, IO - Input & Output, NA - Not Applicable
                                              CAM Utilization for ASIC [0]
Table                Subtype      Dir   Max    Used   %Used  V4  V6  MPLS Other
-------------------------------------------------------------------------------
Mac Address Table    EM           I     32768  97     0.30%   0   0   0     97
IP Route Table       EM           I     24576  12     0.05%   11  0   1     0
IP Route Table       TCAM         I     8192   8190   99.98%  8177 10 2     0

That output is the definitive answer. A region at 99.98% utilisation with routes being rejected is a capacity limitation, not a bug. Each platform name and syntax differs slightly, so check the equivalent command for your hardware generation before concluding — on some families TCAM utilisation is not exposed at all and you must infer from the FIB comparison.

What consumes the table

• Internet-scale tables on access hardware. Platforms intended for campus access cannot hold a full BGP table alongside other features; full tables need higher-end modules with larger TCAM.
• Host route explosion. DHCP pools with short leases, mobile clients, or virtualised workloads can generate thousands of /32 entries. Some platforms can be configured to treat /32s differently, but the durable fix is to reduce the number of addresses routed individually.
• Multicast and MPLS sharing regions. Enabling a second address family or MPLS in a region reallocates capacity from another; adding features reduces how many routes fit.
• ACL and policy growth. TCAM is shared, so a large ACL expansion can consume the space an address family needed.
• Shared resources across contexts. Where virtual contexts exist, the hardware table is shared, so one context's growth affects the others.

Fixes that hold

1. Aggregate and summarise. Fewer, larger prefixes are the only fix that scales indefinitely. Summarisation at area or domain boundaries has a bigger effect than any counter tuning.
2. Match hardware to the role. Access switches should hold a default route and the local subnets; they should never be asked to hold a full table. Where a full table truly is required, choose the platform accordingly.
3. Cap the blast radius. Maximum prefix limits with discard or extra-path options, as described in IOS XR maximum prefix and discard-extra-paths, stop a peering incident from filling the table.
4. Do not fight the limit. There is no configuration that adds TCAM entries. When the counters say a region is full, the options are aggregation, moving the function to different hardware, or accepting that the route will not be installed.
5. Alert before it is full. Poll TCAM utilisation and route-installation gaps continuously; crossing 80% is an engineering task, crossing 99% is an incident waiting for a convergence event.

Design work at this layer interacts with everything above it: fabric choices in Nexus 9000 VXLAN BGP EVPN design and path selection behaviour in BGP best path selection both change how many prefixes a device must hold. Capacity planning for tables is as real as capacity planning for bandwidth, and it fails more quietly.

原文链接：https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2025/pdf/BRKTRS-3090.pdf