ZFS Send and Receive: Snapshot Replication Guide

ZFS replication is the rare backup technology that is both cheap and simple: take a snapshot, pipe zfs send into zfs receive on the other side, and you have an exact, block-level copy of the dataset transferred over ordinary SSH. The same mechanism gives you hourly incrementals that only move changed blocks, and a destination you can browse or mount for a single-file restore. This guide covers the full workflow — snapshots, full and incremental sends, resume tokens, and the verification steps that keep replication honest.

Snapshots Are the Unit of Replication

A snapshot is instant and initially consumes no space; it costs only the blocks that change after it is taken. Recursive snapshots cover a whole dataset tree, which is what you want for application data with child datasets:

zfs snapshot -r tank/app@2026-09-14
zfs list -t snapshot -r tank/app -o name,creation,used
zfs list -H -t snapshot -o name tank/app          # names only, easy to script

Full Replication Over SSH

Send a full recursive stream and receive it on the target host. Mark the destination read-only so nobody can accidentally fork the replica away from the source:

# destination: immutable replica
zfs set readonly=on backup
zfs set atime=off backup

# source: full recursive replication stream
zfs send -R tank/app@2026-09-14 | ssh backup-host zfs receive -Fduv backup/app

Incremental Replication: Only What Changed

Once a full replica exists, hourly (or 15-minute) jobs only ship the delta between the last common snapshot and the new one:

# new snapshot, then send only the delta from the previous snapshot
zfs snapshot -r tank/app@2026-09-14-02
zfs send -vi tank/app@2026-09-14 tank/app@2026-09-14-02 | \
     ssh backup-host zfs receive -Fduv backup/app

The two snapshot names must both exist on the source and share history with the destination. Break that chain — for example by deleting the intermediate snapshot on the source — and the next incremental send fails with "incremental source … is not present"; you then need a new full send.

Resume Interrupted Transfers

Large first-time sends over a slow WAN link rarely finish on the first attempt. ZFS supports resumable streams, which store receiver state so a failed transfer continues instead of restarting:

# source
zfs send -R -s tank/app@2026-09-14 | ssh backup-host zfs receive -Fduv backup/app
# if it fails, the receiver token is listed with:
ssh backup-host zfs get receive_resume_token backup/app
# resume:
zfs send -t <token> | ssh backup-host zfs receive -Fduv backup/app

Bookmarks are the lighter alternative when you want to keep the "previous" reference without the snapshot retention cost: zfs bookmark tank/app@2026-09-14 tank/app#lastrep.

Verification and Rotation

# both sides should report the same snapshot and similar used space
zfs list -t snapshot -o name,used,refer backup/app
zfs list -o name,used,avail,refer backup/app

# confirm the replica really is read-only
zfs get readonly backup/app

# retention: keep hourly for 48h, daily for 30d, monthly for a year
zfs destroy tank/app@2026-09-14-02      # only after the next snapshot exists downstream

Never delete a snapshot on the source before the destination has the following one — that is the single most common way people accidentally force a full resend of terabytes.

Practical Habits That Save Days

• Log the snapshot name, byte count and duration of every replication run; growing incrementals are the earliest sign of a filesystem churning or a retention bug.
• Keep the destination read-only, and use -F so reruns cleanly roll back instead of failing.
• Snapshot before application-consistent operations (database dumps, VM quiesce) rather than relying on crash-consistent ZFS snapshots alone.
• Test a restore quarter by quarter: mount the replica read-only and pull a real file, do not just check that the job exits 0.

What a ZFS Stream Actually Contains

A zfs send stream is not a tar archive and not a filesystem image in the usual sense. It is an ordered sequence of block-level records — data blocks, indirect blocks, the dataset's property set and, depending on the flags, the snapshots themselves — written to stdout as a single binary stream. Three consequences follow, and they explain almost every surprise people hit in production.

First, the stream is self-describing and the receiver is not a byte copy: zfs receive replays records into a freshly allocated dataset, so the replica is compressed, checksummed and deduplicated according to the destination pool's own settings rather than inheriting the source's physical layout. Second, properties travel with the stream only when you ask them to: -R (recursive plus properties) carries mountpoint, compression, quota, acltype and user properties, while a plain zfs send ships data alone. Third, streams are not idempotent by themselves — the receiver needs -F to roll back, otherwise re-sending a snapshot the destination already has fails outright with "destination already exists".

# inspect a stream without receiving it
zfs send tank/app@2026-09-14 | zstreamdump | head -40

# estimate a full stream's size before committing to the WAN transfer
zfs send -nP tank/app@2026-09-14

# estimate an incremental exactly, still without sending
zfs send -nvP tank/app@2026-09-14 tank/app@2026-09-14-02

zstreamdump shows the record types and the begin/end-of-stream records; -n with -P prints the size without writing anything. Put that dry run in your monitoring script. A full stream that suddenly reports 40 TB instead of 400 GB means the incremental chain broke and you are about to push a full send across a link that cannot carry it.

Compression, mbuffer and Transfer Tuning

SSH is convenient but a poor transport for multi-terabyte streams: its cipher runs on one core, it has no meaningful buffer of its own, and it stalls whenever the far end pauses for disk work. The three levers, in order of impact, are compression, buffering and a cheaper cipher.

# compress the pipe itself rather than relying on SSH
zfs send tank/app@2026-09-14 | zstd -3 | ssh backup-host zfs receive -Fduv backup/app

# mbuffer: absorb the bursty writes at both ends
zfs send -i tank/app@2026-09-14 tank/app@2026-09-14-02 \
  | mbuffer -q -m 2G -s 1M \
  | ssh backup-host mbuffer -q -m 2G \
  | zfs receive -Fduv backup/app

# swap the SSH cipher when the link is already private or the CPU is the bottleneck
ssh -c aes128-gcm@openssh.com backup-host ...

Do not pipe zfs send through gzip blindly: if the dataset already stores compressed blocks, ZFS transmits them compressed by default and a second pass burns CPU for a few percent of gain. Check zfs get compressratio,compression first. When you genuinely need to compress the wire, zstd -3 or lz4 give far better throughput per CPU cycle than gzip. mbuffer -m 2G -s 1M is the single change that most often turns a stalling 200 Mbit/s link into a saturated one, because it decouples the sender's write rate from the receiver's flush rate.

Encrypted Datasets and Raw Sends

Native OpenZFS encryption changes the replication rules in a way that is easy to get wrong. A normal send of an encrypted dataset is decrypted in transit and re-encrypted on the destination, which means the sending host must hold the key and the replica ends up under different keys. A raw send (-w) keeps the ciphertext and the key material inside the stream, so the destination stores exactly the same encrypted blocks and can only be read by someone holding the original key.

# raw recursive send: the destination stores ciphertext
zfs send -Rw tank/secure@2026-09-14 | ssh backup-host zfs receive -Fduv backup/secure

# the destination cannot mount it without the key — this is the point
ssh backup-host zfs get encryption,keystatus backup/secure

# attempt a local mount to confirm the replica is genuinely sealed
ssh backup-host zfs mount backup/secure   # expected to fail without the key

Raw sends unlock the property that matters for offsite backup: the backup host never sees plaintext. Keep two rules. Never mix raw and non-raw streams in the same incremental chain — the receiver will reject the mismatch. And escrow the key file separately, because a raw replica is useless during a disaster if the only copy of the key lived on the dead source.

Automating Replication Without a Framework

You do not need a commercial product for a reliable 15-minute replica job. A systemd timer plus a strict shell script is enough, provided the script refuses to run twice and fails loudly.

#!/bin/bash
set -euo pipefail
SNAP="tank/app@auto-$(date +%Y-%m-%d-%H%M)"
PREV=$(zfs list -H -t snapshot -o name -s creation tank/app | tail -2 | head -1)
zfs snapshot -r "$SNAP"
case "$PREV" in
  *@auto-*)  zfs send -Ri "$PREV" "$SNAP" | ssh backup-host zfs receive -Fduv backup/app ;;
  *)         zfs send  -R  "$SNAP"        | ssh backup-host zfs receive -Fduv backup/app ;;
esac
logger -t zfs-repl "sent $SNAP ($PREV)"

Two details make the script trustworthy. The tail -2 | head -1 idiom selects the previous snapshot — the second-newest — because zfs send -i needs a base that the destination already holds. And the trim step must never delete a source snapshot the destination has not yet received: the safe pattern is to list snapshots on the destination, subtract the retention window, and destroy only source snapshots older than the oldest surviving replica.

# enforcement: the destination is the source of truth for what may be destroyed
ssh backup-host zfs list -H -t snapshot -o name -s creation backup/app | tail -1

Failure Modes and Their Exact Error Messages

Log the exit code and the byte count of every run. A job that "succeeds" but transfers zero bytes usually means the snapshot already existed downstream — not that the data is protected.

Sizing and Maintaining the Destination

A replica pool needs room for the retention window plus snapshots, and it must satisfy the same reservations the source relies on: a dataset with reservation=1T cannot be received into a pool with 400 GB free. Check before you commit, and check again when retention is extended.

zfs get -r reservation,quota,refreservation tank/app
zfs list -o name,used,usedbydataset,usedbysnapshots,refer backup/app
zpool list -o name,size,alloc,free,capacity backup

Keep an eye on usedbysnapshots on the destination compared with the source. If the destination's snapshot space grows much faster, someone has been rolling back and re-sending full streams, and you are burning capacity on redundant data instead of protecting new writes.

相关阅读：LVM 精简置备与精简快照、RAID 级别与写洞（write hole）、Ceph OSD 掉线与 PG 排障 以及 NetApp ONTAP SnapMirror 配置。

原文链接：ZFS Quickstart