Ceph CRUSH Map: Device Classes and Custom Rules

CRUSH decides where every object lives, and by default it treats all your OSDs as interchangeable. That is fine until you mix NVMe with 7.2k SATA in the same cluster: a "fast" pool accidentally lands on spinning disks, or metadata writes contend with bulk storage. Device classes and CRUSH rules fix this without hand-editing the CRUSH map. Here is how the pieces fit and how to apply them safely on a live cluster.

What CRUSH Is Responsible For

The CRUSH map contains the OSD list, a hierarchy of buckets (host, rack, row, room, datacenter) and rules that tell CRUSH how to place replicas for each pool. Because the hierarchy mirrors your physical layout, CRUSH can spread copies across failure domains instead of putting all three replicas on one host or one PDU feed.

How an Object Finds Its Home: The Placement Walk

CRUSH is not a lookup table; it is a deterministic function that every client runs for itself. The walk from an object name to a physical disk happens in four steps, and once you can name them, the output of ceph osd crush rule dump stops looking like magic:

1. Object name to placement group. The object name is hashed, the hash is masked to the pool's pg_num, and the result becomes the PG ID. 2.7f means pool 2, placement group 127 — the pool number and the PG number are joined by a dot.
2. PG to an OSD set. CRUSH feeds the PG ID into the rule bound to that pool and walks the bucket hierarchy, returning an ordered list of OSDs. The first entry is the primary — the OSD that services client reads and writes — and the rest are replicas or erasure-coding chunks.
3. The rule language. Every CRUSH rule is a tiny program built from the same three verbs: take selects a starting bucket, choose/chooseleaf picks a number of items beneath it at a given bucket type, and emit returns the result. What looks like an opaque script is those three steps repeated.
4. Determinism is the whole point. Any client holding a copy of the map computes the same answer without asking a monitor for help, which is why the map is distributed to every client and why changing it is a cluster-wide data-moving event rather than a quick metadata tweak.

Device Classes: Automatic, Then Explicit

Every OSD gets a class at startup, detected from the underlying device: hdd, ssd or nvme. Detection is usually right, but you should set classes explicitly so a script or OSD restart cannot silently change them:

# assign classes explicitly
ceph osd crush set-device-class ssd osd.2 osd.3 osd.6 osd.7
ceph osd crush set-device-class hdd osd.0 osd.1 osd.4 osd.5

# inspect the shadow hierarchies Ceph builds for each class
ceph osd crush tree --show-shadow

# list classes in use
ceph osd crush class ls

Two behaviours to remember: once a class is set, it cannot be changed to a different class until you unset it with ceph osd crush rm-device-class <osd>; and class names are free-form strings — you can name a class after a workload (bucket-index) rather than a disk type. Internally Ceph implements classes as "shadow" CRUSH roots, which is why you will see entries such as root default~ssd in ceph osd crush tree --show-shadow. Do not edit those shadow roots by hand — change classes and let Ceph rebuild them.

Reading the Compiled Rule: take, chooseleaf, emit

Before you trust a rule, dump it. The fields worth reading are type (replicated or erasure), min_size, step_take (where the walk starts), step_chooseleaf_firstn (how many items), step_chooseleaf_type (the failure domain) and the final emit step:

ceph osd crush rule dump fast-pool

A healthy replicated rule reads top to bottom as: take the root you named, choose n leaf buckets of the stated type underneath it, emit. If step_chooseleaf_type is osd, replicas may land on the same host — almost always a mistake. If the rule references a bucket that does not exist, PGs sit in creating and health reports PG_AVAILABILITY, with nothing in the message naming the rule that caused it.

Create a Rule That Targets a Class

# replicated rule: name, root, failure domain, optional class
ceph osd crush rule create-replicated fast-pool default host ssd

# erasure-coded pools use a different rule type
ceph osd crush rule create-erasure ec-hdd default hdd

# see what exists and what a rule contains
ceph osd crush rule ls
ceph osd crush rule dump fast-pool
ceph osd crush rule ls-by-class ssd

The failure domain is the bucket type whose members must differ across replicas. host is the right default for most clusters; rack or chassis when hosts share power or a top-of-rack switch.

When a Custom Class Beats a Disk Type

The default three classes describe hardware. Nothing stops you describing intent instead. A class called bucket-index on the four OSDs reserved for RGW index shards keeps that pool on its own disks even after a hardware refresh makes the others faster. A class called wal on the small NVMe devices used for BlueStore write-ahead logs lets you target them explicitly instead of hoping they stay on the right tier. Two habits keep custom classes sane: name them for what the pool needs rather than the model number of the disk, and remember that a class is only a label — Ceph implements each class as a shadow root, so removing the class removes the shadow hierarchy with it.

Bind Pools to Rules

ceph osd pool set rbd-fast crush_rule fast-pool
ceph osd pool get rbd-fast crush_rule
ceph osd pool set rbd-fast size 3
ceph osd pool set rbd-fast min_size 2

Changing the rule on an existing pool moves data — slowly and continuously — as placement groups remap. Check progress instead of assuming:

ceph -s | head -20                 # look at misplaced / recovering / degraded
ceph osd df tree                   # per-OSD utilisation after the remap
ceph pg dump_stuck inactive        # any PG that cannot find a home

Erasure-Coded Pools and Device Classes

Erasure coding changes the arithmetic but not the idea. An EC profile describes k data chunks and m coding chunks; the rule must guarantee no two chunks land in the same failure domain, and the device class keeps the whole set on the hardware tier you chose:

ceph osd erasure-code-profile set ec-4-2 k=4 m=2 crush-device-class=hdd crush-failure-domain=host
ceph osd erasure-code-profile get ec-4-2
ceph osd pool create ec-bulk erasure ec-4-2
ceph osd pool set ec-bulk crush_rule ec-hdd

Two constraints bite here. First, an EC pool needs at least k+m OSDs in the target class, and after losing m of them it is read-only — so size the class against that, not against raw capacity alone. Second, when the class is baked into the profile with crush-device-class, moving the pool to different hardware later means a new profile and a new pool plus a data migration, not a simple crush_rule change.

PG Count, Autoscaling and the Balancer

Device classes decide which disks; PG count and the balancer decide how evenly the data lands on them:

ceph osd pool autoscale-status
ceph osd pool set rbd-fast pg_autoscale_mode on
ceph osd pool set rbd-fast target_size_ratio 0.3
ceph balancer mode upmap
ceph balancer on
ceph balancer status

Autoscaling reads the pool's target ratio and adjusts pg_num to keep roughly 100 PGs per OSD. The upmap balancer then moves individual PGs to correct skew CRUSH alone cannot remove — after a rule change, for example, when one host in the target class is larger and absorbs more PGs than its share. Keep the two jobs separate in your head: autoscaling changes PG counts (cheap metadata work), upmap moves data (real IO work). Do not enable either for the first time during a busy quarter.

When Placement Fails

Change Safely: The Order That Matters

1. Set device classes first, and verify with ceph osd crush tree --show-shadow.
2. Create the new rule and rule dump it to confirm the take/chooseleaf steps.
3. Apply it to one non-critical pool, watch ceph -s until misplaced objects clear.
4. Only then move production pools, one at a time, ideally with nobackfill/norebalance windows if the cluster is busy.
5. Keep a copy of ceph osd getcrushmap -o crushmap.bin before any manual CRUSH edit, so rollback is possible.

Verification Cookbook

After any class or rule change, run this set in order and read the highlighted columns:

ceph osd crush class ls
ceph osd crush tree --show-shadow
ceph osd crush rule ls-by-class ssd
ceph osd pool get rbd-fast crush_rule
ceph pg ls-by-pool rbd-fast | head
ceph pg dump_stuck inactive
ceph pg map 2.7f
ceph osd df tree
ceph balancer status

Ready means ceph -s reports HEALTH_OK, or at worst HEALTH_WARN with recovering objects falling; misplaced objects trending to zero; every PG in the pool mapped to OSDs of the intended class (confirm one PG with ceph pg map); and ceph osd df tree showing utilisation in a narrow band rather than a single OSD near full. For the failure mode where an OSD never comes up and its PGs stall, the triage steps in our Ceph OSD down and stuck-PG runbook pick up where this leaves off, and if the root cause is a dying disk, the SMART patterns in our smartctl and smartd guide will catch it before placement ever becomes the problem.

相关阅读：Ceph OSD 掉线与 PG 排障、ZFS send/recv 快照复制 以及 RAID 级别与写洞解析。

原文链接：Ceph Documentation - CRUSH Maps