Every cluster needs node maintenance sooner or later. Kernel patches, node pool upgrades and hardware swaps all start the same way, with kubectl drain. A drain asks the API server to evict each pod on the node, and a PodDisruptionBudget (PDB) is what lets an application say "not yet" when losing one more pod would hurt.
Most teams meet PDBs the hard way. Either they have none and a drain takes out two replicas of a three-replica database at once, or they have a strict one and the drain hangs forever. This guide walks through draining a node with budgets in place, reading the output when a drain blocks, and getting unstuck without turning the protection off.
What a drain does
kubectl drain first cordons the node. Cordoning marks it unschedulable, so the scheduler stops placing new pods there. Then it calls the Eviction API for every pod on the node instead of deleting them directly.
That difference matters. A plain delete always succeeds. An eviction goes through admission checks, and the API server refuses it when it would break a PodDisruptionBudget. The upstream docs list the possible answers. 200 OK means the eviction went through and the pod is being deleted. 429 Too Many Requests means a budget blocks it for now, and kubectl retries. 500 Internal Server Error means a misconfiguration, such as two budgets selecting the same pod.
A PDB only covers voluntary disruptions, which are the ones that go through the Eviction API. If a node loses power, no budget can keep its pods alive. The budget only stops you from making things worse on purpose.
Step 1: see what runs on the node
Before you drain anything, look at what will move. Start with the node list, then the pods on the node you plan to drain.
kubectl get nodes
kubectl get pods --all-namespaces --field-selector spec.nodeName=worker-2 -o wide
Look for three kinds of pods. DaemonSet pods will stay, since the DaemonSet controller would put them straight back. Pods with no controller, such as a pod someone created by hand, won't be recreated anywhere once evicted. Pods that use emptyDir volumes will lose that data. kubectl refuses to evict the last two kinds unless you pass a flag, which is covered in step 4.
Step 2: give each replicated workload a budget
A budget selects pods with a label selector and says how many may be unavailable at once. Here is a small Deployment and a budget that allows one of its pods to be down at a time.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
namespace: shop
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: nginx:1.27
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /
port: 80
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web
namespace: shop
spec:
maxUnavailable: 1
selector:
matchLabels:
app: web
You can set minAvailable or maxUnavailable, never both in one budget. The Kubernetes docs recommend maxUnavailable because it follows the replica count of the controller. If you later scale web to six replicas, maxUnavailable: 1 still means one at a time, while minAvailable: 2 would quietly allow four evictions.
Both fields take a number or a percentage. Percentages round up. With seven pods and minAvailable: "50%", Kubernetes requires four pods to stay available, not three.
There is one catch with percentages. They only work when the PDB's pods belong to a Deployment, ReplicaSet, StatefulSet, ReplicationController or a custom resource with a scale subresource, because Kubernetes needs a replica count to work from. For bare pods or pods from an operator without a scale subresource, you can only use an integer minAvailable.
Keep the selector tight. If two budgets select the same pod, the Eviction API refuses to evict it and returns a 500 error, and your drain stops on that pod.
Step 3: check the budgets before you start
Apply the manifest and look at the budget's status.
kubectl apply -f web.yaml
kubectl get pdb --all-namespaces
NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
shop web N/A 1 1 2m
The column to watch is ALLOWED DISRUPTIONS. It is the number of pods the disruption controller will let you evict right now. Any budget that shows 0 will block a drain on a node that hosts one of its pods.
A zero has a few common causes. The budget may be too strict on purpose, such as maxUnavailable: 0 or a minAvailable equal to the replica count. A single-replica Deployment with minAvailable: 1 is the classic case. Or some pods aren't Ready, so the application already counts as disrupted. The docs are blunt about the strict case. A budget that requires zero voluntary evictions means a drain on that node never completes.
For more detail, read the budget's status block.
kubectl get pdb web -n shop -o yaml
The status shows currentHealthy, desiredHealthy, disruptionsAllowed and expectedPods. A pod counts as healthy when its Ready condition is True. If currentHealthy is below desiredHealthy, find out why pods aren't Ready before you touch the node.
Step 4: drain with a timeout
Now drain the node. These are the flags most clusters need.
kubectl drain worker-2 \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout=10m
--ignore-daemonsets lets the drain continue past DaemonSet pods, which keep running on the node until it is removed. --delete-emptydir-data accepts that pods using emptyDir lose that local data. Only pass it once you have checked in step 1 that nothing on the node keeps data you care about in emptyDir.
The --timeout flag deserves more attention than it gets. Its default is zero, which kubectl treats as no limit. In an automated upgrade, a blocked budget then turns into a job that hangs until someone notices. With a timeout, the drain fails with a clear error and your pipeline can alert.
Two other flags are worth knowing. --force lets kubectl delete pods that have no controller. Those pods are gone for good afterwards, so recreate them elsewhere first. --grace-period overrides each pod's termination grace period. Its default of -1 keeps the value from the pod spec, which is almost always what you want.
Step 5: read a blocked drain
When a budget blocks an eviction, kubectl doesn't give up. It prints a line and tries again five seconds later.
evicting pod shop/web-7c9d8f6b5-x2k4p
error when evicting pods/"web-7c9d8f6b5-x2k4p" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
The message names the pod but not the budget. Find the budget by matching labels.
kubectl get pod web-7c9d8f6b5-x2k4p -n shop --show-labels
kubectl get pdb -n shop
kubectl describe pdb web -n shop
Then check whether the application is healthy. In a three-replica Deployment with maxUnavailable: 1, a drain blocks when one replica is already down somewhere else. That is often a pod stuck in Pending because the cordoned node took away the room it needed, or a pod that never became Ready.
kubectl get pods -n shop -l app=web -o wide
kubectl describe pod <pending-or-unready-pod> -n shop
Step 6: unblock it the safe way
The fix depends on why the budget says no. If the workload is healthy but the budget is too strict, change the budget or add capacity. Scaling a single-replica Deployment to two before maintenance turns a permanent block into a normal rolling move.
kubectl scale deployment web -n shop --replicas=4
If the block comes from pods that are broken anyway, look at unhealthyPodEvictionPolicy. It became stable in Kubernetes 1.31. The default, IfHealthyBudget, only lets a drain evict a running but unready pod when the application still meets its budget. That protects an app during a rollout, but it also means one pod in CrashLoopBackOff can block a node drain indefinitely. With AlwaysAllow, running pods that aren't Ready can always be evicted, and the budget keeps guarding the healthy ones.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web
namespace: shop
spec:
maxUnavailable: 1
unhealthyPodEvictionPolicy: AlwaysAllow
selector:
matchLabels:
app: web
The upstream guide to draining nodes recommends AlwaysAllow for exactly this reason, so a drain can get past misbehaving pods. Pods in Pending, Succeeded or Failed phase can be evicted under either policy.
When none of that applies and you have to move on, delete the single blocking pod yourself. A direct delete skips the budget for that pod only, and you made the call knowingly.
kubectl delete pod web-7c9d8f6b5-x2k4p -n shop
Avoid --disable-eviction as a habit. It makes the whole drain use plain deletes, and the kubectl help warns that it bypasses every PodDisruptionBudget on the node. That turns a small, contained decision about one pod into an unchecked one about all of them.
Step 7: put the node back
When kubectl drain returns without an error, every pod it was responsible for has been evicted with its grace period respected. You can reboot the machine, replace it or delete the VM.
If the node stays in the cluster, it is still cordoned. Mark it schedulable again when the maintenance is done.
kubectl uncordon worker-2
kubectl get nodes worker-2
Uncordoning doesn't move pods back. The scheduler only uses the node for new pods, so the cluster rebalances over time as workloads roll. If you drain a whole node pool in sequence, check kubectl get pdb -A between nodes. A budget that dropped to zero allowed disruptions on the last node will block the next one too, and it is easier to fix while only one node is out of service.
Some pods can still land on a drained node. A pod that tolerates the node.kubernetes.io/unschedulable taint can be scheduled there, and a pod with nodeName set directly skips the scheduler entirely. Keep that toleration for DaemonSets only.