Read the Ceph health signal before changing the cluster

A health status starts the investigation

Storage symptoms often appear somewhere else first: an application becomes slow, a volume operation takes longer, or several services report errors together. That makes it tempting to change the storage cluster as soon as an alert appears.

Start by collecting a consistent picture instead. Ceph reports a summary health state and named checks that describe specific conditions. The short status is an entry point; the detail tells you which condition needs attention. The monitoring guide describes these views. Use documentation for your installed Ceph release when interpreting the output.

Before investigating, record the time, the affected applications and what users are experiencing. “The cluster is warning” and “a user cannot read data” describe different observations. Keep both without assuming one completely explains the other.

Collect a read-only snapshot

From a configured Ceph administration environment with appropriate read permissions, these commands provide an initial snapshot. They do not change cluster configuration.

ceph -s
ceph health detail
ceph osd stat
ceph df

The first two commands connect the summary to the named health checks. OSD status and capacity information provide additional context. Capture timestamps with the output and compare observations over time; a single sample cannot tell you whether a condition is improving.

Keep the snapshot in the incident record. It should help the next person answer what changed and when, without requiring them to reconstruct the initial state from memory.

Separate capacity, redundancy and latency

Capacity pressure, degraded redundancy and slow requests are different kinds of signals. A health check may identify more than one at once. Read the exact identifiers and supporting details rather than translating every warning into “a disk failed.” The health-check reference explains the individual conditions and their context.

Use three parallel questions: can clients complete their operations, is data protection below its intended level, and is there enough operating headroom? Record what is known for each. A performance complaint deserves application-side timing as well as a cluster snapshot.

Look for a shared timeline: a maintenance action, host event or workload change near the beginning of the symptoms. Timing is a useful lead, but it is not proof of causation.

Choose the smallest justified response

Do not turn an alert into an automatic instruction to restart daemons or change recovery settings. First identify the affected component, the intended result of an action and the observation that will show whether it helped. Follow the operational procedure for the installed release and deployment.

When the condition clears, check the application path again. Record which signal improved first and whether user-visible performance recovered with it. This closes the loop between infrastructure health and the reason the system exists.

• Preserve the initial health detail and timestamps.
• Separate observations from working hypotheses.
• Document each action and its expected effect.
• Verify recovery from a client’s perspective.

The most useful incident note is not a long command transcript. It is a short explanation of the evidence, the decision and the result.