About this scenario
One worker node lost power and won't come back. The cloud provider confirms the instance is gone.
Deployments moved on long ago, but the db StatefulSet's db-1 is stuck in Terminating and no new db-1 appears. Its data is on an EBS volume, which only attaches to nodes in its own zone.
Bring db-1 back with its data. sim node add NAME --zone ZONE adds a node, as an autoscaler would.
Objectives
- See why db-1 isn't replaced
- Release db-1 from the dead node
- db-1 is Running again
- db-1 kept its data
What you'll practice
- A Pod on an unreachable node stays
Terminating: nothing can confirm it stopped. StatefulSets wait for that confirmation so two copies never share one identity and one volume. - The
node.kubernetes.io/out-of-service taint tells Kubernetes the node is really off. Its Pods are deleted at once and its volumes detach without the usual 6 minute wait. - Force-deleting the Pod (
--grace-period=0 --force) also frees the name, but the volume stays attached to the dead node until the attach/detach controller gives up. - Zonal volumes like EBS pin a Pod to one zone. Keep spare capacity in every zone a StatefulSet uses.
How the playground works
The cluster runs entirely in your browser: a simulated API server, scheduler, kubelets and controllers, with virtual nodes in three zones. kubectl output follows the real formats, errors use the real messages, and the add-ons (CSI storage, Traefik, Envoy Gateway, cert-manager, Flux, Argo CD) behave like their real counterparts for the common paths. It's a simulation, so some behavior is simplified. Your progress is saved in this browser.