Linux tc Traffic Shaping with HTB: Rate, Ceil and Filters

The Linux tc (traffic control) framework shapes outbound bandwidth at the kernel level, which makes it perfect for enforcing per-host or per-destination rate limits on a gateway without buying QoS hardware. HTB (Hierarchical Token Bucket) is the classful qdisc that implements rate guarantees plus borrowing: each class gets a guaranteed rate and may burst up to ceil when the parent has spare bandwidth. This guide builds a working HTB shaper and verifies it with iperf3.

Why Shape Bandwidth in the Kernel

Shaping means delaying packets so that the send rate matches a policy. Policing means dropping or re-marking packets that exceed the policy. On a Linux router you almost always want shaping on egress, because egress is the one direction the box genuinely controls: it decides when a frame is handed to the NIC. Inbound traffic arrives whether you like it or not, so inbound limitation must be done either by dropping (ingress policing) or by slowing your own TCP acknowledgements so remote senders back off — the trick consumer routers use to keep downstream rates under a contract.

The usual reasons to reach for HTB:

• Guarantee a floor for latency-sensitive traffic such as VoIP, DNS or an interactive SSH session while a nightly backup saturates the link.
• Cap a tenant, customer or VM to an agreed rate without touching the hypervisor or the switch.
• Stop one host from filling an uplink and starving every other host behind it.
• Reproduce a slow, jittery WAN link in a lab so an application's poor behaviour surfaces before production does.

tc Components in One Minute

• qdisc — the queueing discipline attached to the interface (default pfifo_fast has no shaping effect).
• class — a child bucket inside a classful qdisc; HTB classes carry rate/ceil.
• filter — classifies packets into classes, e.g. by destination IP with u32.

Classless vs Classful qdiscs

Classless qdiscs — pfifo_fast, fq_codel, sfq, tbf, netem — attach straight to an interface and do their job with no children. They are simple and cheap, but they cannot express "5 mbit for this host, 80 mbit for that one". Classful qdiscs — htb, hfsc, cbq, prio — own a class tree, and filters decide which class each packet joins. HTB is the pragmatic default: it is well documented, it supports borrowing, and its rate/ceil vocabulary maps cleanly onto how service contracts are actually written.

Understanding rate, ceil, burst and cburst

Every HTB class is a token bucket. A packet may leave the class only when enough tokens exist; otherwise it waits in the class queue. Four numbers define that behaviour.

• rate — the guaranteed long-term throughput. HTB honours this even when the parent is congested, which is precisely the promise a classless shaper cannot make.
• ceil — the ceiling the class may reach by borrowing idle bandwidth from its parent. Set ceil equal to the parent's rate when you want a class to be able to consume the whole link if nobody else needs it.
• burst — how many bytes may be released at full interface speed before the class is throttled down to rate. Too small a burst cripples TCP on fast links, because an ACK-clocked send window cannot be released in one go.
• cburst — the same idea for the ceil bucket.
• prio — the tie-breaker used when two classes both have queued packets and the parent has tokens to spend. Lower numbers are served first.

A burst of roughly 10 ms of line-rate traffic is a sane starting point. On gigabit interfaces the kernel computes acceptable defaults when you say nothing, but be explicit about every value you intend to keep in production.

HTB Shape Script: Parent, Children and Filters

Model: the uplink is capped at 100 mbit. Two destinations — 192.168.5.0/32 and 192.168.5.1/32 — each start at 5 mbit and may borrow up to 80 mbit when the other is idle:

#!/bin/bash
TC=/sbin/tc
IF=ens160
LIMIT=100mbit
START_RATE=5mbit
CHILD_LIMIT=80mbit

# root qdisc: unclassified traffic goes to class 1:30
$TC qdisc add dev $IF root handle 1:0 htb default 30

# parent class (the 100mbit pool children borrow from)
$TC class add dev $IF parent 1:0 classid 1:1 htb rate $LIMIT

# children
$TC class add dev $IF parent 1:1 classid 1:10 htb rate $START_RATE ceil $CHILD_LIMIT
$TC class add dev $IF parent 1:1 classid 1:30 htb rate $START_RATE ceil $CHILD_LIMIT

# filters: send matching dst IPs into the right child
$TC filter add dev $IF protocol ip parent 1:0 prio 1 u32   match ip dst 192.168.5.0/32 flowid 1:10
$TC filter add dev $IF protocol ip parent 1:0 prio 1 u32   match ip dst 192.168.5.1/32 flowid 1:30

Read the parameters correctly: rate is the guaranteed bandwidth, ceil the maximum after borrowing, burst/cburst token-bucket sizes, and prio decides which class is served first under contention. Always pair each leaf class with an sfq or fq_codel qdisc to fair-queue the flows inside it:

$TC qdisc add dev $IF parent 1:10 handle 10: sfq perturb 10
$TC qdisc add dev $IF parent 1:30 handle 30: sfq perturb 10

Without a child qdisc under each leaf, HTB's own FIFO handles all flows in that class, and a single aggressive TCP stream can push every other flow in the same class into the back of a very deep queue. sfq with perturb hashes flows into 128 buckets and rotates the hash periodically so no flow can camp in the best slot. Modern kernels make fq_codel an equally good, sometimes better, choice because it also combats bufferbloat.

Verifying the Shaper

$TC qdisc show dev ens160
$TC class show dev ens160

You should see the root htb qdisc with its default class, the parent class, both children with their rate and ceil, and the two leaf qdiscs. A typical excerpt looks like this:

qdisc htb 1: root refcnt 2 r2q 10 default 30 direct_packets_stat 0
qdisc sfq 10: parent 1:10 limit 127p quantum 1514b perturb 10sec
qdisc sfq 30: parent 1:30 limit 127p quantum 1514b perturb 10sec
class htb 1:1 root rate 100Mbit ceil 100Mbit burst 1600b cburst 1600b
class htb 1:10 parent 1:1 prio 0 rate 5Mbit ceil 80Mbit burst 15Kb cburst 1599b
class htb 1:30 parent 1:1 prio 0 rate 5Mbit ceil 80Mbit burst 15Kb cburst 1599b

If a class is missing, the class add line failed — usually because you referenced a classid that does not exist yet, or forgot the parent handle. If a filter is missing, run tc filter show dev ens160; a filter pointing at a non-existent flowid is silently accepted by some kernel versions but never matches.

Measuring Real Throughput With iperf3

Run iperf3 on the shaped gateway or on a host that routes through it, and generate traffic toward each shaped destination so both classes are active:

# on the receiving host
iperf3 -s

# toward 192.168.5.0, then toward 192.168.5.1
iperf3 -c 192.168.5.0 -t 60 -P 4
iperf3 -c 192.168.5.1 -t 60 -P 4

With both destinations receiving, each is capped near its 5 mbit share. Stop one flow and the survivor climbs toward ceil as it borrows the now-idle bandwidth. That borrowing behaviour is the entire point of HTB versus a flat token bucket, and it is the quickest way to prove the shaper is wired correctly. Watch the counters on the shaped interface while you test:

tc -s class show dev ens160

The Sent, rate, overlimits and dropped counters tell you how much traffic the class released, whether it is currently hitting its ceiling, and how often it had to wait. A class with a growing dropped counter is starving traffic you probably meant to let through; a class with a huge backlog is holding packets longer than intended, which shows up as latency on the wire.

Common Mistakes

• Shaping the wrong interface. HTB limits what leaves a device. Shaping eth0 does nothing for traffic arriving on eth0.
• Setting ceil below rate. child cannot exceed its own guaranteed rate, so the class will never borrow. HTB accepts this, and you will spend an afternoon wondering why borrowing never happens.
• Summing rate above the parent. If two children each guarantee 80 mbit under a 100 mbit parent, the guarantees are impossible; HTB does not validate this for you and behaviour becomes unintuitive.
• Forgetting the leaf qdisc. A shaped but un-fair-queued class serialises flows and punishes everyone for one bad stream.
• Filters that never match. Verify with tc -s filter show dev ens160 that packets are actually counted against a filter before trusting it.
• No burst tuning on fast links. On 10G interfaces, ignoring burst/cburst costs more throughput than the shape itself.

HTB vs TBF, HFSC and policers

HTB is not the only tool in the toolbox, and knowing when it is the wrong one saves a lot of debugging:

• TBF (Token Bucket Filter) is a classless single-rate shaper. It is an excellent fit when a whole interface needs one cap and nothing else — one tc qdisc add dev eth0 root tbf rate 50mbit burst 32kbit latency 400ms line and you are done. It cannot express per-host guarantees.
• HFSC offers true service curves: you can specify separate real-time and link-share bandwidth with different latency targets, which is the right answer when you must bound jitter for voice traffic rather than only bound average rate. The cost is a steeper learning curve and subtler misconfiguration symptoms.
• Policers (tc ingress, or a switch port ACL) drop rather than delay. They are cheaper on CPU, but dropped packets make TCP back off hard and interact badly with retransmission bursts, so a shape beats a police whenever you control the sender's path.
• Per-flow fair queuing (fq_codel, cake) alone will not hold a host to a contract, but as a leaf under HTB it removes bufferbloat inside each class, which is why the combination of HTB for the contract and fq_codel for the flows is the configuration most network engineers settle on.

The practical rule: use TBF for a single blunt rate limit, HTB when you need per-class guarantees with borrowing, and HFSC only when latency, not bandwidth, is the thing you are contractually promising.

Choosing and troubleshooting filters

Filters decide which class a packet enters, so a shaper is only as good as its classification. The u32 matcher used above is exact, cheap and works on any kernel, but it is positional: match ip dst 192.168.5.0/32 compiles to a byte-offset check in the IP header. Two alternatives are worth knowing. fw filters read a mark previously stamped by iptables -j MARK or nft, which lets you reuse an existing firewall rule set for classification instead of writing the same match twice. flower filters classify with a friendlier syntax and can also steer matching traffic to a hardware offload path on supported NICs, which matters when the CPU cost of shaper becomes the bottleneck.

To confirm classification is actually happening, reset the counters, push traffic, and read them back:

tc -s filter show dev ens160
tc -s qdisc show dev ens160

If a filter shows zero matches while you know the traffic is flowing, check the direction: a parent 1:0 filter on the root qdisc only sees egress packets leaving that interface. Traffic to the shaped destination that arrives from another interface never touches this qdisc at all, so make sure the shaper is attached to the interface that actually carries the traffic toward those hosts.

Persisting the Shape Across Reboots

tc rules vanish on reboot. The portable approach is to keep the script you just built in /usr/local/sbin/shape.sh, make it executable, and start it from a systemd unit so ordering is explicit rather than dependent on somebody's rc.local:

[Unit]
Description=HTB egress shaping
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/shape.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Make the script idempotent by deleting the root qdisc first (tc qdisc del dev $IF root 2>/dev/null || true) so a re-run never stacks two shapers on the same interface. On distributions that ship one, tuned or a config-management module for tc can also own the rules; the systemd unit is simply the least surprising option when you are hand-rolling.

Cleanup and Notes

$TC qdisc del dev ens160 root

HTB only shapes egress on the interface it is attached to; shaping inbound requires ingress policing (e.g. tc police) or shaping on the peer. To simulate latency or loss instead of limiting bandwidth, see Linux tc netem; for interface-level diagnosis pair the shaper with ethtool diagnostics. If throughput after shaping looks lower than the configured rate, review TCP buffer and backlog tuning, and if large transfers stall once the shaper is in place check MTU, MSS and PMTUD, because a badly sized burst plus an MTU black hole looks exactly like a shaper that is too aggressive.

原文链接：https://joshrosso.com/c/tc/