Why a Kubernetes disruption budget can block your drain

A budget governs eviction, not every outage

A node drain stalls. The PodDisruptionBudget reports no allowed disruptions. The immediate temptation is to remove the budget, finish maintenance and restore it afterward. Before doing that, determine which promise the budget is preventing you from breaking.

A PDB constrains voluntary disruptions that use the Eviction API. It does not stop hardware failure or directly deleting a Pod. Involuntary disruptions count against the budget. Deployment and StatefulSet rolling updates are controlled by their own workload settings, rather than limited by the PDB. These boundaries are described in the disruptions documentation.

Assume a stateless API with three replicas, a meaningful readiness probe and a requirement that two instances remain healthy during planned maintenance. This lab example uses an integer budget to make the arithmetic explicit. It is not suitable for every workload, especially one whose application-level quorum or session behavior requires a different model.

Make the budget match the application

The following object selects an existing Deployment’s Pods. The namespace must exist, and its Pod labels must actually include app: api. Check the API against your cluster version before using it.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-maintenance
  namespace: example-app
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

With three healthy matching replicas and no other disruption in progress, the simple expectation is one permitted eviction. With only two healthy replicas, the budget has no room for another healthy Pod to be evicted. Read the actual status rather than assuming the controller has already reconciled your latest change.

Specify either minAvailable or maxUnavailable, not both. Percentage budgets round upward, which is especially significant at small replica counts. An integer is easier to reason about in this fixed three-replica exercise. The PDB configuration guide explains the calculation and selector behavior.

Replica count is not placement

Three replicas can share a failure domain. A PDB does not choose nodes, so replica placement needs a separate scheduling decision. Consider this fragment inside an existing Deployment’s spec.template.spec:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api

The hard constraint limits imbalance across eligible hostname domains; it is not a declaration that three hosts must exist. A different eligible-domain set changes what can be scheduled. Verify node labels, affinity, taints and resource availability together. The topology spread documentation defines the skew calculation and eligible-domain behavior.

For the lab, start with three eligible worker nodes and one replica per node. Keep enough resources for the replacement to become ready somewhere permitted by the placement policy. If a replacement remains Pending, the PDB cannot create capacity or weaken the scheduler’s constraints for you.

That is the central design tension: strict placement reduces correlated failure, but can restrict recovery when capacity disappears. Treat the tension as an explicit choice rather than discovering it during an upgrade.

Find what is consuming the disruption allowance

Inspect the following read-only views from the intended cluster context:

kubectl get pdb api-maintenance -n example-app -o yaml
kubectl get pods -n example-app -l app=api -o wide
kubectl get deployment api -n example-app -o yaml
kubectl get events -n example-app --sort-by=.metadata.creationTimestamp

Compare currentHealthy, desiredHealthy, expectedPods and disruptionsAllowed. Check that the budget’s selector describes the intended population. Then inspect unready or Pending Pods and their events before changing the policy.

Readiness itself must correspond to useful application service. A probe that returns success while every request fails makes the count misleading, even when the PDB is working exactly as configured.

Decide how unhealthy Pods should be evicted

On versions that support it, unhealthyPodEvictionPolicy: AlwaysAllow lets running but unhealthy Pods be evicted without satisfying the normal budget criteria for healthy Pods. This can prevent an unhealthy instance from blocking maintenance. Review the policy semantics for your version before choosing it.

The application-specific question is whether replacing that unhealthy process is acceptable now. A process that fails its readiness check may still be holding work, debugging evidence or local state. Write down the intended behavior and test it with the workload owner. Do not present the setting as a general cure for drain failures.

Test maintenance as a complete sequence

In a disposable cluster, establish a successful request baseline and verify the initial placement. Drain one selected worker through the normal maintenance workflow, then observe eviction, replacement scheduling, readiness and actual client requests. Finish by restoring the node’s intended scheduling state.

Repeat the exercise with one unhealthy replica and, separately, insufficient spare capacity. Record which step stops and whether that stop protects the service requirement. Do not use force or deletion options to bypass the budget merely to make the test complete.

A useful acceptance statement is: “During one planned worker outage, the agreed request workload remains within its error and latency targets, replacement capacity returns, and the maintenance workflow completes without overriding protection.” This is measurable. “We have a PDB” is only a description of an object.