# How to drain Kubernetes nodes safely with PodDisruptionBudgets > Drain a node for maintenance without taking an app down: write PodDisruptionBudgets, read a blocked drain and unblock it without bypassing eviction. Published: 2026-11-05 Author: k8wiz Team URL: https://k8wiz.com/articles/drain-kubernetes-nodes-safely-with-poddisruptionbudgets Every cluster needs node maintenance sooner or later. Kernel patches, node pool upgrades and hardware swaps all start the same way, with `kubectl drain`. A drain asks the API server to evict each pod on the node, and a PodDisruptionBudget (PDB) is what lets an application say "not yet" when losing one more pod would hurt. Most teams meet PDBs the hard way. Either they have none and a drain takes out two replicas of a three-replica database at once, or they have a strict one and the drain hangs forever. This guide walks through draining a node with budgets in place, reading the output when a drain blocks, and getting unstuck without turning the protection off. ## What a drain does `kubectl drain` first cordons the node. Cordoning marks it unschedulable, so the scheduler stops placing new pods there. Then it calls the Eviction API for every pod on the node instead of deleting them directly. That difference matters. A plain delete always succeeds. An eviction goes through admission checks, and the API server refuses it when it would break a PodDisruptionBudget. The upstream docs list the possible answers. `200 OK` means the eviction went through and the pod is being deleted. `429 Too Many Requests` means a budget blocks it for now, and kubectl retries. `500 Internal Server Error` means a misconfiguration, such as two budgets selecting the same pod. A PDB only covers voluntary disruptions, which are the ones that go through the Eviction API. If a node loses power, no budget can keep its pods alive. The budget only stops you from making things worse on purpose. ## Step 1: see what runs on the node Before you drain anything, look at what will move. Start with the node list, then the pods on the node you plan to drain. ```bash kubectl get nodes kubectl get pods --all-namespaces --field-selector spec.nodeName=worker-2 -o wide ``` Look for three kinds of pods. DaemonSet pods will stay, since the DaemonSet controller would put them straight back. Pods with no controller, such as a pod someone created by hand, won't be recreated anywhere once evicted. Pods that use `emptyDir` volumes will lose that data. kubectl refuses to evict the last two kinds unless you pass a flag, which is covered in step 4. ## Step 2: give each replicated workload a budget A budget selects pods with a label selector and says how many may be unavailable at once. Here is a small Deployment and a budget that allows one of its pods to be down at a time. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web namespace: shop spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: containers: - name: web image: nginx:1.27 ports: - containerPort: 80 readinessProbe: httpGet: path: / port: 80 --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web namespace: shop spec: maxUnavailable: 1 selector: matchLabels: app: web ``` You can set `minAvailable` or `maxUnavailable`, never both in one budget. The Kubernetes docs recommend `maxUnavailable` because it follows the replica count of the controller. If you later scale `web` to six replicas, `maxUnavailable: 1` still means one at a time, while `minAvailable: 2` would quietly allow four evictions. Both fields take a number or a percentage. Percentages round up. With seven pods and `minAvailable: "50%"`, Kubernetes requires four pods to stay available, not three. There is one catch with percentages. They only work when the PDB's pods belong to a Deployment, ReplicaSet, StatefulSet, ReplicationController or a custom resource with a `scale` subresource, because Kubernetes needs a replica count to work from. For bare pods or pods from an operator without a scale subresource, you can only use an integer `minAvailable`. Keep the selector tight. If two budgets select the same pod, the Eviction API refuses to evict it and returns a 500 error, and your drain stops on that pod. ## Step 3: check the budgets before you start Apply the manifest and look at the budget's status. ```bash kubectl apply -f web.yaml kubectl get pdb --all-namespaces ``` ```text NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE shop web N/A 1 1 2m ``` The column to watch is `ALLOWED DISRUPTIONS`. It is the number of pods the disruption controller will let you evict right now. Any budget that shows `0` will block a drain on a node that hosts one of its pods. A zero has a few common causes. The budget may be too strict on purpose, such as `maxUnavailable: 0` or a `minAvailable` equal to the replica count. A single-replica Deployment with `minAvailable: 1` is the classic case. Or some pods aren't Ready, so the application already counts as disrupted. The docs are blunt about the strict case. A budget that requires zero voluntary evictions means a drain on that node never completes. For more detail, read the budget's status block. ```bash kubectl get pdb web -n shop -o yaml ``` The status shows `currentHealthy`, `desiredHealthy`, `disruptionsAllowed` and `expectedPods`. A pod counts as healthy when its `Ready` condition is `True`. If `currentHealthy` is below `desiredHealthy`, find out why pods aren't Ready before you touch the node. ## Step 4: drain with a timeout Now drain the node. These are the flags most clusters need. ```bash kubectl drain worker-2 \ --ignore-daemonsets \ --delete-emptydir-data \ --timeout=10m ``` `--ignore-daemonsets` lets the drain continue past DaemonSet pods, which keep running on the node until it is removed. `--delete-emptydir-data` accepts that pods using `emptyDir` lose that local data. Only pass it once you have checked in step 1 that nothing on the node keeps data you care about in `emptyDir`. The `--timeout` flag deserves more attention than it gets. Its default is zero, which kubectl treats as no limit. In an automated upgrade, a blocked budget then turns into a job that hangs until someone notices. With a timeout, the drain fails with a clear error and your pipeline can alert. Two other flags are worth knowing. `--force` lets kubectl delete pods that have no controller. Those pods are gone for good afterwards, so recreate them elsewhere first. `--grace-period` overrides each pod's termination grace period. Its default of `-1` keeps the value from the pod spec, which is almost always what you want. ## Step 5: read a blocked drain When a budget blocks an eviction, kubectl doesn't give up. It prints a line and tries again five seconds later. ```text evicting pod shop/web-7c9d8f6b5-x2k4p error when evicting pods/"web-7c9d8f6b5-x2k4p" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. ``` The message names the pod but not the budget. Find the budget by matching labels. ```bash kubectl get pod web-7c9d8f6b5-x2k4p -n shop --show-labels kubectl get pdb -n shop kubectl describe pdb web -n shop ``` Then check whether the application is healthy. In a three-replica Deployment with `maxUnavailable: 1`, a drain blocks when one replica is already down somewhere else. That is often a pod stuck in `Pending` because the cordoned node took away the room it needed, or a pod that never became Ready. ```bash kubectl get pods -n shop -l app=web -o wide kubectl describe pod -n shop ``` ## Step 6: unblock it the safe way The fix depends on why the budget says no. If the workload is healthy but the budget is too strict, change the budget or add capacity. Scaling a single-replica Deployment to two before maintenance turns a permanent block into a normal rolling move. ```bash kubectl scale deployment web -n shop --replicas=4 ``` If the block comes from pods that are broken anyway, look at `unhealthyPodEvictionPolicy`. It became stable in Kubernetes 1.31. The default, `IfHealthyBudget`, only lets a drain evict a running but unready pod when the application still meets its budget. That protects an app during a rollout, but it also means one pod in `CrashLoopBackOff` can block a node drain indefinitely. With `AlwaysAllow`, running pods that aren't Ready can always be evicted, and the budget keeps guarding the healthy ones. ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web namespace: shop spec: maxUnavailable: 1 unhealthyPodEvictionPolicy: AlwaysAllow selector: matchLabels: app: web ``` The upstream guide to draining nodes recommends `AlwaysAllow` for exactly this reason, so a drain can get past misbehaving pods. Pods in `Pending`, `Succeeded` or `Failed` phase can be evicted under either policy. When none of that applies and you have to move on, delete the single blocking pod yourself. A direct delete skips the budget for that pod only, and you made the call knowingly. ```bash kubectl delete pod web-7c9d8f6b5-x2k4p -n shop ``` Avoid `--disable-eviction` as a habit. It makes the whole drain use plain deletes, and the kubectl help warns that it bypasses every PodDisruptionBudget on the node. That turns a small, contained decision about one pod into an unchecked one about all of them. ## Step 7: put the node back When `kubectl drain` returns without an error, every pod it was responsible for has been evicted with its grace period respected. You can reboot the machine, replace it or delete the VM. If the node stays in the cluster, it is still cordoned. Mark it schedulable again when the maintenance is done. ```bash kubectl uncordon worker-2 kubectl get nodes worker-2 ``` Uncordoning doesn't move pods back. The scheduler only uses the node for new pods, so the cluster rebalances over time as workloads roll. If you drain a whole node pool in sequence, check `kubectl get pdb -A` between nodes. A budget that dropped to zero allowed disruptions on the last node will block the next one too, and it is easier to fix while only one node is out of service. Some pods can still land on a drained node. A pod that tolerates the `node.kubernetes.io/unschedulable` taint can be scheduled there, and a pod with `nodeName` set directly skips the scheduler entirely. Keep that toleration for DaemonSets only.