Ceph Public vs Cluster Network Separation

A single-network Ceph cluster works until recovery traffic competes with client I/O, and then it degrades in a way that is hard to diagnose: latency rises for everyone, but bandwidth counters look healthy because the same interfaces carry both flows. Separating the public network (clients, monitors, gateways) from the cluster network (OSD replication and recovery) isolates that interference and is the single most effective structural change available to a Ceph deployment. This article covers the addressing model, the configuration, and how to verify which interface each OSD is actually using.

What the two networks carry

The public network carries client I/O — RADOS, RBD, CephFS, RGW — plus monitor communication and cluster metadata. It must be reachable by every client.

The cluster network carries OSD-to-OSD replication, backfill and recovery. It only needs to be reachable between OSD hosts, and the documentation is explicit that it should not be reachable from the public network. Keeping it unroutable from clients also removes it from the blast radius of a client-side incident.

Every cluster needs a public network. The cluster network is optional, and on 25GbE and faster a single-network design is genuinely fine for many deployments — but once you plan recovery windows or run latency-sensitive workloads over RBD, the split pays for itself.

Configuring the split

[global]
    public_network = 10.10.10.0/24
    cluster_network = 10.10.20.0/24

[osd.0]
    public_addr = 10.10.10.11
    cluster_addr = 10.10.20.11

[osd.1]
    public_addr = 10.10.10.12
    cluster_addr = 10.10.20.12

Write the per-OSD addresses explicitly. Relying on public_network and cluster_network alone means each OSD picks an address from the matching subnet, which works until a host has several interfaces in the same subnet or the daemon restarts while an interface is down and binds the wrong address.

Apply without a full outage

ceph config set global public_network 10.10.10.0/24
ceph config set global cluster_network 10.10.20.0/24
# restart OSDs one host at a time
systemctl restart ceph-osd.target

Moving an existing cluster from one network to two requires the OSDs to re-bind, so it is a rolling restart, not a reload. Plan it around a healthy cluster state: ceph -s should show HEALTH_OK and no active backfill before you start. A restart during recovery turns a maintenance window into an incident.

Verifying which network is in use

ceph osd dump | grep -E "public_addr|cluster_addr"
ceph config dump | grep -E "public_network|cluster_network"
# Confirm the cluster actually uses the intended path
ip -s link show eno2
ss -tnp | grep ceph-osd

ceph osd dump is the authoritative check: it shows the public and cluster address recorded for each OSD. If cluster_addr is missing, that OSD is doing replication on the public network, and the split is not in effect for it.

Sizing and MTU notes

• Bandwidth — the cluster network carries roughly the replication factor times the client write rate, plus recovery. Recovery after a large disk failure can saturate a 10GbE link for hours; 25GbE is a reasonable modern floor.
• Latency — replication is synchronous for writes, so RTT on the cluster network is directly in the client write path. Keep it flat Layer 2 or a low-hop-count routed design.
• MTU — jumbo frames help recovery throughput, but only if every device in the path agrees. Verify end to end with ping -M do -s 8972 before enabling; the failure mode of a partial MTU change is silent fragmentation and worse performance, the same trap described in VXLAN MTU fragmentation and jumbo frames.
• Security — the cluster network should be unreachable from clients. Where the two subnets must touch, control it with ACLs on the router rather than trusting the addressing plan.

Storage-layer tuning complements this split: throttling recovery so it cannot starve clients, as covered in Ceph OSD recovery and backfill throttling, is the natural companion setting once the networks are separated.

原文链接：https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/