# A StatefulSet database: headless Service, per-Pod volumes, snapshots and restore

> The objects around a replicated database on Kubernetes: a StatefulSet with a headless Service and one PersistentVolumeClaim per Pod, a PodDisruptionBudget, and a VolumeSnapshot restored into a new claim.

Interactive graph: https://k8wiz.com/graphs/statefulset-headless-service-volumesnapshot-restore

A StatefulSet gives each replica a fixed name and its own volume. `db-0` is always `db-0`, it always gets the claim `data-db-0`, and it has a DNS name of its own. That is what a database needs and what a Deployment can't give.

The second half of the graph is the backup path: a CSI snapshot of one replica's volume, and a new claim created from it.

## Resources

### Service db

you write it, v1, namespace data

A headless Service (`clusterIP: None`). It has no virtual IP: DNS returns the Pod addresses, and each Pod gets a name such as `db-0.db.data.svc.cluster.local`. Replicas use those names to find each other.

### StatefulSet db

you write it, apps/v1, namespace data

Creates `db-0`, `db-1` and `db-2` in order, each from the same template, and one claim per Pod from `volumeClaimTemplates`. `serviceName` ties the Pods to the headless Service for their DNS names.

### Pod ×3 db-0, db-1, db-2

created by a controller, v1, namespace data

Stable names with an ordinal. `db-1` starts only when `db-0` is Ready, and a rolling update goes from the highest ordinal down. A Pod that is replaced comes back with the same name and the same claim.

### PersistentVolumeClaim ×3 data-db-0, data-db-1, data-db-2

created by a controller, v1, namespace data

One per Pod, named `<template>-<pod>`. The StatefulSet creates them and, with the retention policy on `Retain`, never deletes them, not on scale-down and not when the StatefulSet is deleted.

### StorageClass gp3

comes with an add-on, storage.k8s.io/v1

Provisions each volume when its Pod is scheduled (`WaitForFirstConsumer`), in that Pod's zone. A volume lives in one zone, so its Pod is tied to that zone for good.

### PodDisruptionBudget db

you write it, policy/v1, namespace data

Lets a node drain or an autoscaler evict one database Pod at a time. A second eviction is refused until the first Pod is Ready again, which protects the quorum.

### VolumeSnapshotClass ebs-snapshots

you write it, snapshot.storage.k8s.io/v1

Which CSI driver takes snapshots and what happens to the cloud snapshot when the object is deleted. `Retain` keeps it.

### VolumeSnapshot db-0-2026-11-13

you write it, snapshot.storage.k8s.io/v1, namespace data

A request for a point-in-time copy of one claim. `status.readyToUse` turns true when the driver has finished. It is crash-consistent: what Postgres would find after a power cut.

### VolumeSnapshotContent snapcontent-8c1f…

created by a controller, snapshot.storage.k8s.io/v1

The cluster-scoped record of the real snapshot, with its ID at the cloud provider. It is to a VolumeSnapshot what a PersistentVolume is to a claim.

### PersistentVolumeClaim data-db-restore-0

you write it, v1, namespace data

A new claim whose `dataSource` is the snapshot. The driver creates a volume from it, at least as large as the original. Named `data-<statefulset>-0`, it is adopted by a StatefulSet called `db-restore`.

## How the resources connect

- StatefulSet db → Service db: serviceName (references)
- Service db → Pod ×3 db-0, db-1, db-2: selector, per-Pod DNS (selects)
- StatefulSet db → Pod ×3 db-0, db-1, db-2: creates in order (creates)
- StatefulSet db → PersistentVolumeClaim ×3 data-db-0, data-db-1, data-db-2: volumeClaimTemplates (creates)
- Pod ×3 db-0, db-1, db-2 → PersistentVolumeClaim ×3 data-db-0, data-db-1, data-db-2: mounts its own (references)
- PersistentVolumeClaim ×3 data-db-0, data-db-1, data-db-2 → StorageClass gp3: storageClassName (references)
- PodDisruptionBudget db → Pod ×3 db-0, db-1, db-2: selector (selects)
- VolumeSnapshot db-0-2026-11-13 → PersistentVolumeClaim ×3 data-db-0, data-db-1, data-db-2: source (references)
- VolumeSnapshot db-0-2026-11-13 → VolumeSnapshotClass ebs-snapshots: volumeSnapshotClassName (references)
- VolumeSnapshot db-0-2026-11-13 → VolumeSnapshotContent snapcontent-8c1f…: driver creates (creates)
- PersistentVolumeClaim data-db-restore-0 → VolumeSnapshot db-0-2026-11-13: dataSource (references)

## How it works

1. **Pods and claims come up in order.** The StatefulSet creates `data-db-0` and `db-0`, waits for the Pod to be Ready, then moves to `db-1`. Each claim is provisioned in the zone its Pod landed in.
2. **Each Pod has a DNS name.** Through the headless Service, `db-0.db.data.svc.cluster.local` resolves to that one Pod. The primary and its replicas address each other by these names, which survive restarts.
3. **A replaced Pod finds its data.** When `db-1` is deleted or its node drained, the StatefulSet creates a new `db-1` that mounts `data-db-1` again. The budget lets only one Pod be evicted at a time.
4. **A snapshot captures one volume.** Applying the VolumeSnapshot makes the snapshot controller ask the CSI driver for a copy of `data-db-0`. The driver records it in a VolumeSnapshotContent and sets `readyToUse`.
5. **Restore into a new claim.** A claim with the snapshot as `dataSource` gets a volume holding the snapshot's data. A second StatefulSet named `db-restore` picks it up by name, and you check the data there before you point anything at it.

## Common mistakes

- `kubectl delete statefulset db` leaves the claims and the data. Deleting a claim with a `Delete` reclaim policy destroys the disk. Know which one you are typing.
- `volumeClaimTemplates` can't be edited. To grow the volumes, expand each claim (the class needs `allowVolumeExpansion`), then recreate the StatefulSet with `--cascade=orphan` and the new size.
- A volume is zonal. If its zone has no node with room, the Pod stays Pending with `volume node affinity conflict`, however empty the other zones are.
- A snapshot of a running database is crash-consistent only. For a clean one, use the database's own backup tool or take the snapshot from a replica you paused.
- VolumeSnapshot needs the snapshot CRDs and the snapshot controller, which many clusters don't install with the CSI driver. Without them the object is accepted by nothing and `kubectl get volumesnapshot` fails.

## The manifests you write

```yaml
apiVersion: v1
kind: Service
metadata:
  name: db
  namespace: data
spec:
  clusterIP: None
  selector:
    app: db
  ports:
    - name: postgres
      port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
  namespace: data
spec:
  serviceName: db
  replicas: 3
  selector:
    matchLabels:
      app: db
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  template:
    metadata:
      labels:
        app: db
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: postgres
          image: postgres:18
          ports:
            - containerPort: 5432
          envFrom:
            - secretRef:
                name: db-credentials
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: db
  namespace: data
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: db
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshots
driver: ebs.csi.aws.com
deletionPolicy: Retain
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-0-2026-11-13
  namespace: data
spec:
  volumeSnapshotClassName: ebs-snapshots
  source:
    persistentVolumeClaimName: data-db-0
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-db-restore-0
  namespace: data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp3
  resources:
    requests:
      storage: 100Gi
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: db-0-2026-11-13
```