Linux conntrack: 'table full, dropping packet' Fix

nf_conntrack: table full, dropping packet in the kernel log means the connection-tracking table is at its limit, and every new connection beyond that is dropped while existing sessions keep working. It is the classic symptom on NAT gateways, load balancers and Kubernetes nodes. This guide explains how to confirm it, how to size the table and hash properly (not just crank a single number), and how to stop the table filling up again.

Confirm it first

dmesg | grep -i conntrack
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_buckets
conntrack -S                     # per-CPU insert_failed, drop, early_drop counters
ss -tan | wc -l                  # sockets the host itself is using

If count sits at max, the diagnosis is confirmed. Rising insert_failed in conntrack -S is the cleaner signal than parsing dmesg, and it is what you should alert on.

Size the table and the hash together

Two different knobs are involved: the maximum number of entries and the size of the hash table that indexes them. Raising only nf_conntrack_max on a small hash produces long hash chains and expensive lookups under load.

# runtime
sysctl -w net.netfilter.nf_conntrack_max=1048576

# persistent
cat >/etc/sysctl.d/99-conntrack.conf <<'EOF'
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
EOF
sysctl --system

# hash table (module parameter, needs reload -> schedule a window)
modprobe -r nf_conntrack
modprobe nf_conntrack hashsize=262144

Rule of thumb: each tracked connection costs on the order of a few hundred bytes of kernel memory, so a one-million-entry table is a few hundred megabytes that must actually exist on the box. On systems with a lot of memory the default bucket count is derived from RAM, which is why a small VM hits the limit long before a large one does.

The timeout that fills the table

The default nf_conntrack_tcp_timeout_established is measured in days on many kernels. Every long-lived idle TCP session occupies an entry for that entire time, so a busy proxy with thousands of mostly-idle connections can exhaust the table with connections that nobody is using. Dropping it to 3600 seconds (one hour) is the single most effective change, and in practice systems with an hour timeout still never recycle an active flow.

Stop tracking what you do not need

# do not track high-volume internal traffic that needs no NAT or stateful filtering
nft add rule inet raw prerouting ip saddr 10.10.30.0/24 tcp dport 443 notrack
iptables -t raw -A PREROUTING -s 10.10.30.0/24 -p tcp --dport 443 -j NOTRACK

Raw-table exemptions, per-CPU flow tables (nf_flowtable) for established flows, and moving to a stateless design for internal traffic can remove most of the pressure. If you run Kubernetes with iptables-based kube-proxy, most of the table is Service DNAT state – which is another argument for the nftables datapath on new clusters.

Monitoring

• Export nf_conntrack_count and nf_conntrack_max as metrics (node_exporter provides them) and alert on the ratio, not on a static number.
• Cap growth potential by checking what the top sources are: conntrack -L | awk '{print $5}' | sort | uniq -c | sort -rn | head.
• Watch for asymmetric routing, which doubles entries per conversation on NAT devices.
• After raising limits, verify memory headroom – the fix should not turn a drop problem into an OOM kill.

Related: Linux network tuning with sysctl, switch CPU troubleshooting, and nftables migration.

原文链接：https://access.redhat.com/solutions/8721