ZFS ARC Size Tuning on Linux

The ZFS ARC is the most effective cache in a Linux storage stack and the most common source of "where did my memory go" complaints. Its default ceiling is derived from installed RAM rather than configured, which is exactly right for a dedicated storage server and exactly wrong for a hypervisor that also hosts virtual machines. This article explains how the default is computed, how to set a limit deliberately, and how to verify the change actually took effect.

How the default is calculated

With zfs_arc_max at its default of 0, OpenZFS uses the larger of all_system_memory - 1 GiB and 5/8 x all_system_memory. On a 64 GiB host with no parameter set, the ARC may grow to roughly 40 GiB. That is a feature on a storage server and a problem on a machine whose primary job is something else.

The value must be at least 67108864 bytes (64 MiB). It can be changed at runtime, with two important caveats: it cannot be set back to 0 while the module is loaded, and lowering it below the current ARC size does not cause the ARC to shrink without memory pressure to induce eviction. If you need memory back immediately, drop caches or accept that the reduction takes effect over time.

Setting a limit

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=17179869184     # 16 GiB
options zfs zfs_arc_min=4294967296      # 4 GiB

# runtime change on a running system
echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max
echo 4294967296  > /sys/module/zfs/parameters/zfs_arc_min

# verify
grep -E "^(size|c_max|c_min|size)" /proc/spl/kstat/zfs/arcstats

Note the asymmetry in the defaults: zfs_arc_min at 0 resolves to consuming the larger of 32 MiB and all_system_memory / 32, so on a large host the floor is already substantial. On a system where ZFS is the primary workload, raising the floor ensures the cache is not evicted by transient pressure from an unrelated process.

Verify with real numbers

arc_summary | head -30
awk '/^size/ || /^c_max/ || /^c_min/ {print}' /proc/spl/kstat/zfs/arcstats
awk '/^hits/ || /^misses/ {print}' /proc/spl/kstat/zfs/arcstats

Compare size against c_max: if the ARC is sitting well below the ceiling, the limit is not your constraint and changing it will do nothing for performance. The hit ratio is the number that tells you whether the cache is doing its job. If hits are high and the workload is still slow, the bottleneck is elsewhere — look at pool layout and device latency, not at RAM.

Practical guidance

• Dedicated storage server — leave the default. The ARC will use most of RAM and that is the correct allocation.
• Hypervisor also running VMs — set zfs_arc_max explicitly to leave headroom for guest memory, and account for it in your capacity planning rather than letting the two subsystems compete.
• Container host with many tenants — a hard ceiling plus monitoring on arcstats is better than discovering the contention during an incident.
• Database server on ZFS — tune the database's own cache first; two large caches fighting for the same pages is worse than one sized properly.
• Do not disable the ARC — treating it as wasted memory and clamping it to a few hundred megabytes is the single most common way to make ZFS slower than ext4 on the same hardware.

Related parameters worth knowing about

Metadata handling has its own levers: zfs_arc_meta_limit_percent and zfs_arc_dnode_limit_percent control how much of the ARC may be consumed by metadata and dnode entries. On filesystems with very large directory trees, metadata pressure can evict useful data buffers, and increasing the metadata allowance is the targeted fix. L2ARC and special vdevs change the calculus entirely and should be sized before touching the ARC ceiling.

Storage tuning is one layer of a system: the sweep of parameters documented here is only useful once the pool layout, recordsize and device health are correct. Where the storage is reached over a network rather than locally, the transport considerations in Fibre Channel vs iSCSI vs NVMe-oF compared apply before any cache tuning will show measurable benefit.

For how local storage compares with networked storage transports, see Fibre Channel versus iSCSI versus NVMe-oF.

原文链接：https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html