# PVC, StorageClass and CSI: how a cloud disk ends up in your Pod

> Dynamic provisioning with the AWS EBS CSI driver, object by object: StatefulSet, PVC, StorageClass, PV, CSIDriver, VolumeAttachment and the controller and node plugins that create, attach and mount the disk.

Interactive graph: https://k8wiz.com/graphs/pvc-storageclass-csi-dynamic-provisioning

A PersistentVolumeClaim is a request for storage. Nothing in it says where the disk comes from. The StorageClass names a CSI driver, and that driver's controller creates a real disk in the cloud and a PersistentVolume that points at it. The kubelet and the driver's node plugin then attach and mount the disk on the node that runs the Pod.

The example uses the AWS EBS CSI driver. Swap `ebs.csi.aws.com` for `pd.csi.storage.gke.io` or `disk.csi.azure.com` and the objects stay the same. The volume binding mode matters more than anything else here, because a disk lives in one availability zone and the Pod has to run in that zone too.

## Resources

### StatefulSet postgres

you write it, apps/v1, namespace db

Runs one Pod per replica with a stable name, `postgres-0`, and creates one PVC per Pod from `volumeClaimTemplates`. If the Pod is recreated, it gets the same PVC back.

### Pod postgres-0

created by a controller, v1, namespace db

Created by the StatefulSet. Its `volumes` section points at the claim `data-postgres-0`. It stays in `ContainerCreating` until the disk is attached and mounted.

### PersistentVolumeClaim data-postgres-0

created by a controller, v1, namespace db

Created by the StatefulSet from its template, named `<template>-<pod>`. It asks for 50Gi of `ReadWriteOnce` storage of class `gp3`. It shows `Pending` until a PV is bound to it.

### StorageClass gp3

you write it, storage.k8s.io/v1

Cluster-scoped. Names the CSI driver in `provisioner` and passes it `parameters`. `WaitForFirstConsumer` delays the disk until the Pod is scheduled, so the disk is created in the Pod's zone.

### CSIDriver ebs.csi.aws.com

comes with an add-on, storage.k8s.io/v1

Registers the driver with Kubernetes and says how to treat it, for example `attachRequired: true`, which makes Kubernetes create a VolumeAttachment before mounting.

### Deployment ebs-csi-controller

comes with an add-on, apps/v1, namespace kube-system

The driver's controller plugin, with sidecars from the Kubernetes CSI project. `external-provisioner` watches PVCs and calls `CreateVolume`. `external-attacher` watches VolumeAttachments and calls `ControllerPublishVolume`. It needs AWS permissions to create and attach EBS volumes.

### PersistentVolume pvc-3f1c9a7e-…

created by a controller, v1

Created by `external-provisioner` once the disk exists. `csi.volumeHandle` is the EBS volume ID, and `nodeAffinity` pins it to the disk's zone, so the scheduler never puts a Pod using it elsewhere.

### EBS volume vol-0a1b2c3d4e5f67890

outside the cluster

The actual disk, a 50 GiB encrypted gp3 volume in `eu-west-1a`. It is billed whether a Pod uses it or not.

### Node ip-10-0-1-23

comes with an add-on, v1

The EC2 instance the scheduler picked for `postgres-0`, in zone `eu-west-1a`. The EBS volume gets attached to this instance as a block device.

### VolumeAttachment csi-8e2f…

created by a controller, storage.k8s.io/v1

Created by the attach/detach controller in kube-controller-manager. It asks for this PV to be attached to this node. `external-attacher` does the attach and sets `status.attached: true`.

### DaemonSet ebs-csi-node

comes with an add-on, apps/v1, namespace kube-system

The driver's node plugin, one Pod on every node. The kubelet calls it to format the device the first time (`NodeStageVolume`) and to mount it into the Pod's directory (`NodePublishVolume`).

## How the resources connect

- StatefulSet postgres → Pod postgres-0: creates (creates)
- StatefulSet postgres → PersistentVolumeClaim data-postgres-0: volumeClaimTemplates (creates)
- Pod postgres-0 → PersistentVolumeClaim data-postgres-0: persistentVolumeClaim (references)
- Pod postgres-0 → Node ip-10-0-1-23: scheduled on (references)
- PersistentVolumeClaim data-postgres-0 → StorageClass gp3: storageClassName (references)
- StorageClass gp3 → CSIDriver ebs.csi.aws.com: provisioner (references)
- Deployment ebs-csi-controller → PersistentVolumeClaim data-postgres-0: watches (references)
- Deployment ebs-csi-controller → EBS volume vol-0a1b2c3d4e5f67890: CreateVolume (creates)
- Deployment ebs-csi-controller → PersistentVolume pvc-3f1c9a7e-…: creates (creates)
- PersistentVolumeClaim data-postgres-0 → PersistentVolume pvc-3f1c9a7e-…: bound via volumeName (references)
- PersistentVolume pvc-3f1c9a7e-… → EBS volume vol-0a1b2c3d4e5f67890: csi.volumeHandle (references)
- VolumeAttachment csi-8e2f… → PersistentVolume pvc-3f1c9a7e-…: source (references)
- VolumeAttachment csi-8e2f… → Node ip-10-0-1-23: nodeName (references)
- Deployment ebs-csi-controller → VolumeAttachment csi-8e2f…: attaches disk (references)
- EBS volume vol-0a1b2c3d4e5f67890 → Node ip-10-0-1-23: attached as /dev/nvme1n1 (traffic)
- DaemonSet ebs-csi-node → Pod postgres-0: mounts volume (traffic)

## How it works

1. **The StatefulSet creates a Pod and a claim.** From `volumeClaimTemplates`, the StatefulSet creates the PVC `data-postgres-0` before the Pod `postgres-0`. The claim asks for 50Gi of class `gp3` and starts out `Pending`.
2. **The scheduler picks a node first.** Because the class uses `WaitForFirstConsumer`, nothing is provisioned yet. The scheduler places the Pod on `ip-10-0-1-23` in `eu-west-1a` and writes that node into the claim's `volume.kubernetes.io/selected-node` annotation.
3. **The provisioner creates the disk and the PV.** `external-provisioner` sees a claim for a class whose `provisioner` is `ebs.csi.aws.com` and calls `CreateVolume` with the node's zone. AWS creates the volume. The provisioner then creates a PV pointing at it, and the PV controller binds the claim to it.
4. **The disk gets attached to the node.** The attach/detach controller creates a VolumeAttachment for this PV and node. `external-attacher` calls `ControllerPublishVolume`, AWS attaches the EBS volume to the EC2 instance, and the VolumeAttachment reports `attached: true`.
5. **The kubelet mounts it into the Pod.** The kubelet asks the node plugin to stage the volume, which formats the device as ext4 the first time and mounts it in a staging directory. It then asks it to publish the volume, which mounts it into the Pod's directory. The container starts with its data at `/var/lib/postgresql/data`.
6. **What happens on delete.** Deleting the StatefulSet or the Pod leaves the PVC in place, so the data survives. Deleting the PVC deletes the PV, and with `reclaimPolicy: Delete` the driver deletes the EBS volume too. Use `Retain` for data you can't recreate.

## Common mistakes

- `volumeBindingMode: Immediate` creates the disk as soon as the claim exists, in whatever zone the driver picks. If the Pod then can't be scheduled in that zone, it stays `Pending` with a `volume node affinity conflict` event.
- `ReadWriteOnce` means one node at a time. When a Pod moves to another node, the old attachment must be removed first, and if the old node is unreachable you'll see `Multi-Attach error` until the attach/detach controller force-detaches it, by default after about six minutes.
- Resizing works only with `allowVolumeExpansion: true`. Then you raise `spec.resources.requests.storage` on the PVC. Editing `volumeClaimTemplates` in a StatefulSet is not allowed, so resize each PVC directly.
- A claim stuck in `Pending` says why in its events. `kubectl describe pvc data-postgres-0 -n db` shows messages from `external-provisioner`, such as missing IAM permissions, and the controller logs have the details.
- Since Kubernetes 1.32 a StatefulSet can delete its claims for you with `persistentVolumeClaimRetentionPolicy`. Leave it at the default `Retain` for databases unless you want data gone when you scale down.

## The manifests you write

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: db
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:17
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests:
            storage: 50Gi
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
```