# Recover data from a Released PersistentVolume

> A claim was deleted and the database came back empty on a new disk. The old volume is Released, not gone. Rebind it to the StatefulSet.

Level: Advanced · Topic: Storage · About 15 minutes
Try it in the browser: https://k8wiz.com/playground/recover-data-from-a-released-persistentvolume

The playground is a simulated Kubernetes cluster that runs in the browser. kubectl output and errors follow the real ones.

## The situation

During a cleanup someone deleted the `db` StatefulSet and its claim, `data-db-0`. The StatefulSet was applied again an hour later and Postgres started, with an empty database: `kubectl logs db-0` shows it creating a new cluster.

The StorageClass, `gp3-retain`, has `reclaimPolicy: Retain`. The old volume, labelled `team=orders`, still exists with every order on it. Kubernetes won't hand it to a new claim by itself.

Put `db-0` back on the original volume. The empty replacement volume can go.

## Objectives

1. Find the volume that was left behind (`kubectl get pv --show-labels`)
2. Make the original volume claimable again (A Released volume still points at the deleted claim's UID in `spec.claimRef`)
3. data-db-0 is bound to the original volume
4. db-0 runs on it and Postgres finds its database (`kubectl logs db-0` says it skips initialization)

## Hints

1. `kubectl get pv --show-labels` shows two volumes. The one labelled `team=orders` is `Released`: its claim is gone, and `Retain` kept the volume and its data. The other is the new, empty one that `data-db-0` is bound to.
2. `kubectl get pv NAME -o yaml` shows `spec.claimRef` with the UID of the deleted claim. No claim can ever match that UID again. Remove the UID and the volume becomes `Available`, still reserved for a claim named `data-db-0` in `default`: `kubectl patch pv NAME --type=json -p '[{"op":"remove","path":"/spec/claimRef/uid"}]'`.
3. The current claim is bound to the empty volume, and a bound claim can't be pointed elsewhere. Stop the database with `kubectl scale statefulset db --replicas=0`, then `kubectl delete pvc data-db-0`.
4. `kubectl scale statefulset db --replicas=1` recreates the claim from the template. It binds to the reserved volume, since the name, class and size match, and the scheduler puts `db-0` in that volume's zone.

## Solution

```bash
kubectl get pv --show-labels
```
```bash
OLD=$(kubectl get pv -l team=orders -o jsonpath='{.items[0].metadata.name}')
```
```bash
kubectl patch pv $OLD --type=json -p '[{"op":"remove","path":"/spec/claimRef/uid"}]'
```
```bash
kubectl scale statefulset db --replicas=0
```
```bash
kubectl wait --for=delete pod/db-0 --timeout=60s
```
```bash
kubectl delete pvc data-db-0
```
```bash
kubectl scale statefulset db --replicas=1
```
```bash
kubectl rollout status statefulset/db
```
```bash
kubectl logs db-0
```

## What it teaches

- With `reclaimPolicy: Retain`, deleting a claim leaves its PersistentVolume as `Released` with the data intact. With `Delete`, the default for most dynamic classes, the disk is gone.
- A Released volume is not reused, because `spec.claimRef` still names the old claim by UID. Removing the UID makes it `Available` and reserved for a claim of that name; removing the whole `claimRef` makes it free for any claim that fits.
- A claim binds for life. To move a workload to another volume you delete the claim and create one that binds to the volume you want.
- A StatefulSet recreates missing claims from its template, and deleting the StatefulSet never deletes them. The claim is what you must protect.
- Postgres initializes an empty data directory without complaint. A database that starts cleanly on the wrong volume looks healthy.