Ceph Placement Groups and the PG Autoscaler

Placement groups are the unit of data distribution in Ceph, and the number of them per pool drives almost everything that can go wrong at the RADOS layer. Too few PGs and a handful of OSDs carry a disproportionate share of the data; too many and the cluster spends memory and CPU on PG state instead of serving I/O. Since Nautilus, Ceph can work this out for you — the PG autoscaler adjusts pg_num based on actual usage. This guide explains the arithmetic, how to enable the autoscaler safely, and how to read the tools that show whether distribution is actually balanced.

The Arithmetic Behind pg_num

The only formula worth memorising is the rule of thumb: aim for roughly 100 PGs per OSD across the cluster, and remember that pg_num should be a power of two. With a replication factor of 3, a pool of 1024 PGs occupies 3072 PG replicas distributed across the OSDs, which is roughly 102 PGs per OSD in a 30-OSD cluster — right where you want to be.

The consequences of getting it wrong are asymmetric. Under-shooting means uneven data distribution, because CRUSH computes a PG for each object but has no idea how much data is stored there; a single hot PG can overload the OSDs that host it. Over-shooting means more memory for PG metadata, more peering work, and slower recovery, since every PG maintains its own state machine.

Enable the PG Autoscaler

Check the current cluster-wide setting and per-pool mode, then turn it on:

ceph osd pool autoscale-status
ceph config get global osd_pool_default_pg_autoscale_mode

Autoscaler modes: on, warn and off

Three modes exist per pool: on (adjust automatically), warn (log a recommendation without acting), and off. On a production cluster with existing data, start in warn mode so you can see what the autoscaler would do before it does it:

ceph osd pool set <pool-name> pg_autoscale_mode warn
ceph osd pool autoscale-status

When you are satisfied with the recommendations, switch to on:

ceph osd pool set <pool-name> pg_autoscale_mode on
ceph osd pool autoscale-status

To set the mode for all new pools, fix the default and, if you have very specific sizing needs for one pool, set that pool's target_size_ratio so the autoscaler knows how much of the cluster it should occupy:

ceph config set global osd_pool_default_pg_autoscale_mode on
ceph osd pool set <pool-name> target_size_ratio 0.25

Adjust pg_num Manually When Needed

Manual changes are still appropriate for a pool with a known access pattern, or immediately after creating it:

ceph osd pool create mypool 128
ceph osd pool set mypool pg_num 256
ceph osd pool set mypool pg_num_min 128

Since Nautilus, Ceph steps pgp_num incrementally whenever pg_num changes, whether the change came from the autoscaler or from you. That slow stepping is deliberate: increasing pg_num splits PGs, and data only migrates once pgp_num follows, so amortising the move keeps client impact low. You generally should not set pgp_num by hand — let it track pg_num — but you should watch the migration complete before making another change.

CRUSH Rules Determine Where PGs Land

How many PGs you have is only half the story; where they land is decided by the CRUSH rule. A pool bound to a rule that only targets one device class will place all its PGs on that class regardless of pg_num:

ceph osd crush rule ls
ceph osd crush rule dump replicated_ssd
ceph osd pool get <pool-name> crush_rule
ceph osd pool set <pool-name> crush_rule replicated_ssd

If a pool was created before device classes were assigned, or the rule targets a subtree that no longer matches the topology, distribution will look wrong no matter how the PG count is tuned. Fix the rule, not the PG count.

Verify Distribution and Health

ceph status
ceph osd pool autoscale-status
ceph osd df tree
ceph pg stat
ceph pg dump_stuck inactive
ceph pg dump_stuck unclean
ceph pg ls-by-pool <pool-name> | head -20

Balanced versus unbalanced clusters

ceph osd df tree is the command to read for balance: it shows utilisation per OSD, sorted, and the percentages at the top of the list should be within a few points of each other. A cluster where some OSDs sit at 80 percent and others at 40 percent is unbalanced, and the cause is usually pg_num, a misplaced CRUSH rule, or a large number of PGs in an undersized pool rather than a hardware issue.

ceph status tells you whether PGs are active and clean. Watch the misplaced and degraded columns specifically: misplaced means data is on the wrong OSD relative to the CRUSH rule and is being moved, while degraded means copies are missing because an OSD is down. When a pool's PG count changes, expect a period of misplaced PGs while backfill runs — do not compound it with further changes until the numbers settle.

Operational Notes

• Target roughly 100 PGs per OSD and keep pg_num a power of two.
• Turn the autoscaler on in warn mode first on any cluster with existing data.
• Keep small pools' PG counts out of the autoscaler's control with pg_num_min when the pool is intentionally small.
• Let pgp_num follow pg_num automatically; intervene only when migration has stalled.
• Check the CRUSH rule before assuming a PG count problem — a rule targeting the wrong device class produces the same symptom.

Related reading: our Ceph CRUSH map device class rules guide, the Ceph OSD down and placement group troubleshooting article, and the DRBD 9 resource configuration and quorum guide.

原文链接：https://docs.ceph.com/en/reef/rados/operations/placement-groups/