Make room for a critical Deployment with a PriorityClass
Intermediate · Kubernetes · about 10 minThe batch-worker Deployment requests 800m CPU per Pod, and its six replicas fill the three workers. The payments Deployment was rolled out after them, and both of its Pods are stuck in Pending.
About this scenario
The batch-worker Deployment requests 800m CPU per Pod, and its six replicas fill the three workers. The payments Deployment was rolled out after them, and both of its Pods are stuck in Pending.
The scheduler only counts what Pods request, so nothing is free. Give payments a higher priority than the batch work, and let the scheduler preempt what it has to. Don't scale the batch work down by hand.
Objectives
- Find out why payments is Pending
- Create a PriorityClass that can preempt
- payments runs 2 ready replicas with that priority
- batch-worker still asks for 6 replicas
What you'll practice
- The scheduler places Pods by their requests, not by what they use. A Pod that requests nothing fits anywhere; one that requests more than any node has left stays Pending.
- A FailedScheduling event lists every node and why it was rejected, and whether preemption could help.
- A Pod's priority comes from its PriorityClass when the Pod is created. A higher-priority Pod that can't fit evicts lower-priority Pods from one node, and waits for them to stop.
- The evicted batch Pods go back to Pending until room frees up. A PodDisruptionBudget makes the scheduler prefer other victims;
preemptionPolicy: Nevergives priority in the queue without evicting anyone.
How the playground works
The cluster runs entirely in your browser: a simulated API server, scheduler, kubelets and controllers, with virtual nodes in three zones. kubectl output follows the real formats, errors use the real messages, and the add-ons (CSI storage, Traefik, Envoy Gateway, cert-manager, Flux, Argo CD) behave like their real counterparts for the common paths. It's a simulation, so some behavior is simplified. Your progress is saved in this browser.