Linux Policy Routing with ip rule: Multiple Routing Tables

Standard Linux routing consults one table and sends every packet the same way, but multi-homed hosts and VPN gateways need policy routing: different source addresses, marks or applications take different paths. Linux implements this with ip rule — an ordered list of rules that tells the kernel which routing table to consult for a packet. This guide sets up classic source-based policy routing: a custom table, a rule that selects it, and the routes that fill it.

The feature has been part of the kernel since the late 1990s and is the foundation of everything from ISP gateways to Kubernetes nodes, yet it stays invisible to administrators who never look past ip route show. That single command only prints the main table. Once rules exist that point elsewhere, the output of ip route stops describing where traffic actually goes — which is why policy routing earns a reputation for being mysterious. It is not mysterious; it is simply documented in a different place.

The Pieces: Tables, Rules, Routes

Linux ships three default tables — local (untouchable), main and default. Routes added without a table name go to main. To use policy routing you need three things: a custom table, rules that point traffic at it, and routes inside it.

The lookup order is the key mental model. When a packet needs a route, the kernel walks the rule list in ascending priority order and stops at the first rule that matches; that rule names a table, and the kernel looks up the route there. If no route matches inside that table, the lookup does not silently fall through to the next rule — the packet is dropped and you get an unreachable error. A table named by a matching rule must therefore contain a usable route, including a default route when the destination is not otherwise covered. This is the single most common cause of "I added the rule and the host went dark".

Default Rules and Priorities

ip rule show
# 0:      from all lookup local
# 32766:  from all lookup main
# 32767:  from all lookup default

• Priority 0 — the local table, which is managed by the kernel and holds local and broadcast addresses. Removing it breaks the host.
• Priority 32766 — the main table, where ordinary ip route add entries live.
• Priority 32767 — the default table, which is normally empty and serves as a catch-all that some routing daemons populate.

Lower numbers run first, so a rule at priority 100 is consulted long before the main table at 32766. Leaving gaps in the numbering — 100, 200, 300 rather than 1, 2, 3 — lets you insert rules later without renumbering everything. Rules added with ip rule add and no explicit pref get the next free priority just below 32765, which is why hand-added source rules often show up at 32765.

Step 1: Create a Custom Routing Table

Tables are numbered 1–255 and can be named in /etc/iproute2/rt_tables (persistent across reboots):

echo 200 custom >> /etc/iproute2/rt_tables

The file maps numbers to names so commands stay readable; the kernel itself only knows the number. Names such as isp1 or vpn beat numbers in an audit six months later. On distributions that have migrated to /etc/iproute2/rt_tables.d/ drop-ins, a snippet file with the same one-line mapping works just as well and is easier to manage with configuration management.

Step 2: Add a Policy Rule

The classic use is source routing — selecting the table by the packet's source address. Route all traffic originating from 192.168.30.200 through the custom table:

ip rule add from 192.168.30.200 lookup custom
ip rule show

Rules are evaluated top-down; ip rule show prints the priority, match criteria and target table. Lower priority numbers run first — the built-in local rule (priority 0) always wins for directly connected addresses, so source-address rules usually sit around priority 32765. Pin the priority explicitly when it matters:

ip rule add pref 100 from 192.168.30.200 lookup custom

Step 3: Populate the Table with Routes

ip route add default via 192.168.40.1 dev eth1 table custom

Now traffic from 192.168.30.200 follows the custom table (default via eth1), while everything else keeps using the main table. Without the table keyword the route would land in main and the rule would find nothing.

Two refinements are usually needed. A local route for the subnet behind eth1 keeps directly attached destinations working without a trip through the gateway, and a connected route lets the kernel answer ARP correctly:

ip route add 192.168.40.0/24 dev eth1 table custom
ip route add default via 192.168.40.1 dev eth1 table custom

When routes must also be removed later, the same commands with del work, and ip route flush table custom empties a table in one shot — useful in scripts that rebuild policy routing on every boot.

Practical Pattern: Symmetric Replies on a Multi-Homed Host

A common requirement is that replies to packets received on interface A leave through interface A. Create one table per uplink and add rules keyed on the incoming interface's source ranges:

echo 100 isp1 >> /etc/iproute2/rt_tables
echo 200 isp2 >> /etc/iproute2/rt_tables
ip route add default via 203.0.113.1 dev eth0 table isp1
ip route add default via 198.51.100.1 dev eth1 table isp2
ip rule add from 203.0.113.0/24 lookup isp1
ip rule add from 198.51.100.0/24 lookup isp2

For persistent configuration on modern distributions, NetworkManager or systemd-networkd can express the same rules; the raw ip commands above are the ground truth that any tool eventually programs.

Why source rules rather than destination rules? Because the router's job on a multi-homed host is usually to keep a session intact, not to steer new flows. Once a connection is established with source address 203.0.113.10, every reply leaving with a different source would be dropped by the far end's reverse-path check and by most stateful firewalls in between. Selecting the table by source address guarantees the reply set leaves through the interface that owns the address — the definition of symmetric routing.

The Full Rule Grammar

Source address is only one of several selectors. The complete set of match criteria covers most real requirements:

ip rule add from 10.0.0.0/8 lookup vpn        # source prefix or a single host
ip rule add to 198.51.100.0/24 lookup isp2   # destination prefix
ip rule add iif eth2 lookup dmz              # incoming interface
ip rule add oif eth3 lookup vpn              # outgoing interface
ip rule add fwmark 0x1 lookup vpn            # packet mark set by firewall
ip rule add tos 0x10 lookup ef                # IP TOS / DSCP byte
ip rule add from 10.1.0.0/16 to 10.2.0.0/16 lookup interco

Conditions combine: several selectors in one rule are ANDed, and each rule must resolve to a table. The iif and oif selectors are especially useful with VRF-style designs and with policy that only applies to traffic arriving on a particular VLAN sub-interface. Note that oif is only meaningful for locally generated traffic, since forwarded packets do not yet have an outgoing interface when the lookup runs.

The fwmark Pattern: Routing Specific Applications or Users

When the discriminator is not an address but a workload — one container, one user, one port — the firewall marks the packet and the rule matches the mark. With nftables:

nft add table ip mangle
nft add chain ip mangle output '{ type route hook output priority mangle; }'
nft add rule ip mangle output ip daddr 10.9.0.0/16 meta mark set 0x1

ip rule add fwmark 0x1 lookup vpn

For historical iptables setups the equivalent is iptables -t mangle -A OUTPUT -d 10.9.0.0/16 -j MARK --set-mark 0x1. Mark-based policy is the standard way to send a VPN client's traffic down the tunnel while leaving everything else on the physical uplink, and it is how most split-tunnel designs are implemented in practice. Remember that marks are a 32-bit field shared with other subsystems, so document the values you allocate.

Verifying Policy Routing

ip rule show
ip route show table custom
ip route get 8.8.8.8 from 192.168.30.200

ip route get ... from ... is the killer diagnostic — it replays the full rule+table lookup and shows exactly which interface and next hop the kernel would pick. See ss, netstat and tcpdump troubleshooting when reply traffic still goes the wrong way, and compare with Junos behavior in Junos static routes and qualified next hops.

Two more checks belong in the routine. ip route get 8.8.8.8 from 192.168.30.200 iif eth2 adds an incoming interface to the simulation, which is the only way to reproduce an iif rule. And ip -s rule show on newer iproute2 builds prints per-rule packet and byte counters, turning an invisible rule into something you can watch:

ip -s rule show
# lookups  packets  bytes  action  from  to  table

Making Policy Routing Persistent

ip rule and ip route changes vanish at reboot. Several mechanisms work, and picking one keeps the configuration reviewable:

• systemd-networkd — RoutingPolicyRule= and Route= entries in a .network file express rules and table routes declaratively.
• NetworkManager — nmcli connection modify ... +ipv4.routing-rules "priority 100 from 192.168.30.200 table 200" together with ipv4.route-table.
• Dedicated init script — a small systemd oneshot unit running the exact ip commands is the least magic option and the easiest to diff.
• Inside a network namespace — rules are per-namespace, so containers and labs need their own rules even when the host already has the ones you expected.

If policy routing is derived from firewall marks and the firewall is managed by another tool, keep both halves in the same repository. The classic outage is a firewall reload that clears the marks while the rules that depend on them stay in place.

Troubleshooting

• Host unreachable after adding a rule — the matched table has no route for the destination. Always add a default route to the custom table.
• Rule exists but has no effect — a rule with a lower priority number matched first. Print the list and check the order.
• Traffic arrives but replies leave the wrong interface — that is an oif or source-selection problem; verify with ip route get <peer> from <local>.
• Forwarded traffic ignores the rule — oif rules do not apply to forwarded packets; use iif or a firewall mark instead.
• Works until reboot — persistence is not configured. Pick one of the mechanisms above and commit it.
• Marks seem to disappear — a firewall reload flushed the mangle chain. Recheck nft list ruleset or iptables-save -t mangle.

Policy routing is also the natural partner of traffic shaping and of namespaced lab topologies: pair the rules with tc HTB rate limiting when different uplinks need different service levels, and see FRRouting BGP on Linux when the tables should be filled by a dynamic routing protocol rather than by hand.

原文链接：https://blog.scottlowe.org/2013/05/29/a-quick-introduction-to-linux-policy-routing/