Ceph failure domains: design for recovery, not just replicas

Define the failure you intend to survive

Three copies of an object do not automatically mean three independent copies. A storage design has to connect replica placement to the physical things that can disappear together: a host, a rack, a power feed or a network path.

This article uses an illustrative replicated pool with size=3 and min_size=2. It assumes four equally sized racks, suitable OSDs in each rack and an otherwise healthy control plane. It is a reasoning exercise, not a description of a particular production cluster. It also assumes familiarity with pools, OSDs and placement groups.

Start the design review with a precise statement: one whole rack may be unavailable while clients continue working, and the remaining racks must have room to restore the intended replica count. That is a stronger requirement than surviving one disk failure. Write the recovery requirement alongside the availability requirement so the first does not get lost in the second.

Inspect the rule, not just the hardware inventory

CRUSH uses a hierarchy and placement rules to choose storage locations. A rule that separates replicas across hosts can still place those hosts in one rack. Rack separation requires a matching hierarchy and rule; it is not inferred from how the equipment is physically installed. See the CRUSH map documentation.

ceph osd tree
ceph osd pool get example-pool crush_rule
ceph osd crush rule dump example-rule
ceph osd pool get example-pool size
ceph osd pool get example-pool min_size

These are read-only inspection commands. Substitute an existing pool and the rule name reported for it. Check the selected root, any device-class restriction and the failure-domain type. A rack containing no eligible devices for that rule does not supply recovery space for that pool, even if it has empty disks.

In the example, a rack-level rule places an object’s three replicas across three eligible racks. The fourth rack provides another possible placement domain after a loss. This does not mean every object has a fixed “spare rack”; placement varies across objects and placement groups.

Separate desired protection from the I/O threshold

For a replicated pool, size is the desired replica count; min_size sets the minimum replica threshold for I/O. Satisfying that threshold is necessary but not sufficient for service: peering, reachability and the rest of the cluster still matter. Ceph documents these settings in pool operations.

This is a per-object model. A second rack failure does not imply identical copy counts for all objects. Avoid describing the entire cluster with a single comforting number when different placement groups can have different conditions.

Do not treat lowering min_size as a routine recovery strategy. It changes the protection accepted while serving I/O rather than restoring a missing copy. The pool documentation explicitly warns about the data-loss risk of low replica settings.

Calculate the post-failure capacity pressure

Suppose total raw capacity is C, evenly divided across four eligible racks, and used raw storage is U. After losing one rack, restoring the same total replica data into the remaining capacity gives the simplified utilization estimate below:

remaining capacity = 0.75 × C
post-recovery utilization ≈ U / (0.75 × C)
                         = current utilization / 0.75

At 60% initially: 0.60 / 0.75 = 80%
At 75% initially: 0.75 / 0.75 = 100%

This is an illustrative calculation, not a sizing recommendation. It assumes a balanced workload and eligible capacity everywhere that recovery needs it. It excludes overhead, uneven OSD utilization, growth and competing pools. Real placement and configured fullness thresholds can prevent recovery before the aggregate number reaches 100%.

The review question is therefore not simply “how much space is free?” Ask whether the constrained placement has sufficient usable headroom after the specified loss. A percentage reported for the whole cluster can hide pressure in the particular device class or subtree used by a pool.

Also budget for recovery time. Remaining hardware must serve clients while rebuilding copies. Measure that contention with a representative workload rather than using an idle-cluster recovery result as a service guarantee.

Define what successful recovery looks like

Ceph tracks state at the placement-group level. States such as degraded, undersized or peering describe different situations; a client symptom should be connected to the affected PGs and OSDs. See monitoring OSDs and PGs for those distinctions.

For an agreed lab exercise, capture the normal topology and client latency before simulating a failure. Observe affected PGs, recovery progress, capacity distribution and client errors throughout the test. Restoring the physical component is only one milestone. The completion condition includes the intended protection level, acceptable client behavior and no unexplained residual health checks.

• Can the surviving placement satisfy the original rule?
• Does the client workload remain within its agreed latency and error targets?
• How long is the reduced-protection window?
• What limits recovery first: eligible space, storage throughput or another dependency?

The final artifact should explain both the successful case and the next failure the system cannot tolerate. That boundary is more useful than claiming that replication makes the platform highly available.