# Recover a StatefulSet Pod from a dead node

> A node died and db-1 stays Terminating, never replaced. Tell Kubernetes the node is gone and bring db-1 back with its own volume.

Level: Advanced · Topic: Storage · About 12 minutes
Try it in the browser: https://k8wiz.com/playground/recover-a-statefulset-pod-from-a-dead-node

The playground is a simulated Kubernetes cluster that runs in the browser. kubectl output and errors follow the real ones.

## The situation

One worker node lost power and won't come back. The cloud provider confirms the instance is gone.

Deployments moved on long ago, but the `db` StatefulSet's `db-1` is stuck in `Terminating` and no new `db-1` appears. Its data is on an EBS volume, which only attaches to nodes in its own zone.

Bring `db-1` back with its data. `sim node add NAME --zone ZONE` adds a node, as an autoscaler would.

## Objectives

1. See why db-1 isn't replaced (`kubectl get pods -o wide` and `kubectl get nodes`)
2. Release db-1 from the dead node (Mark the node out of service, or delete the Pod for good)
3. db-1 is Running again
4. db-1 kept its data (The same claim and volume as before the failure)

## Hints

1. The kubelet that would confirm `db-1` stopped is gone, so the Pod stays `Terminating`. A StatefulSet never starts a second `db-1` while the first might still run: two databases on one volume would corrupt it.
2. Once you know the node is gone, say so: `kubectl taint node NODE node.kubernetes.io/out-of-service=nodeshutdown:NoExecute`. Kubernetes then deletes its Pods and detaches its volumes right away.
3. The new `db-1` stays Pending with `volume node affinity conflict`: its volume is in the dead node's zone. `kubectl get node NODE -L topology.kubernetes.io/zone` shows it.
4. `sim node add worker-4 --zone eu-west-1b` (use the dead node's zone).

## Solution

```bash
kubectl get pods -o wide
```
```bash
kubectl get nodes -L topology.kubernetes.io/zone
```
```bash
kubectl taint node worker-2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
```
```bash
sim node add worker-4 --zone eu-west-1b
```
```bash
sleep 60
```

## What it teaches

- A Pod on an unreachable node stays `Terminating`: nothing can confirm it stopped. StatefulSets wait for that confirmation so two copies never share one identity and one volume.
- The `node.kubernetes.io/out-of-service` taint tells Kubernetes the node is really off. Its Pods are deleted at once and its volumes detach without the usual 6 minute wait.
- Force-deleting the Pod (`--grace-period=0 --force`) also frees the name, but the volume stays attached to the dead node until the attach/detach controller gives up.
- Zonal volumes like EBS pin a Pod to one zone. Keep spare capacity in every zone a StatefulSet uses.