# External Secrets Operator: from a cloud secret manager to a Pod, with rotation

> How a secret travels from AWS Secrets Manager to a container: a ClusterSecretStore that authenticates with a ServiceAccount, an ExternalSecret that writes a Kubernetes Secret, the Deployment that reads it and Reloader restarting Pods on rotation.

Interactive graph: https://k8wiz.com/graphs/external-secrets-clustersecretstore-externalsecret

Keeping secrets out of Git usually means keeping them in a secret manager and copying them into the cluster. The External Secrets Operator does the copying. An `ExternalSecret` says which keys to fetch and what Secret to write; a store object says where to fetch from and how to authenticate.

The copy is one-way and repeated. When the value changes upstream, the Secret follows at the next refresh. Whether the application notices is a separate question, and the last part of this graph.

## Resources

### AWS Secrets Manager prod/checkout/database

outside the cluster

The source of truth: a JSON secret with `username` and `password`, rotated every 30 days by a Lambda function.

### ServiceAccount external-secrets

comes with an add-on, v1, namespace external-secrets

The operator's identity. Its annotation maps it to an IAM role that may read secrets under `prod/`, so no long-lived AWS key sits in the cluster.

### ClusterSecretStore aws-prod

you write it, external-secrets.io/v1

One connection to one backend, usable from any namespace the `conditions` allow. A namespaced `SecretStore` does the same for a single namespace.

### ExternalSecret checkout-db

you write it, external-secrets.io/v1, namespace shop

Which remote keys to fetch, how often, and the Secret to write. The template builds a connection string from the two fetched values. This object holds no secret and is safe in Git.

### Secret checkout-db

created by a controller, v1, namespace shop

Written and owned by the operator. With `creationPolicy: Owner`, deleting the ExternalSecret deletes it. Edits made by hand are overwritten at the next refresh.

### Deployment checkout

you write it, apps/v1, namespace shop

Reads `DATABASE_URL` from the Secret as an environment variable. The annotation asks Reloader to restart it when a Secret it uses changes.

### Pod ×3 checkout-…

created by a controller, v1, namespace shop

Environment variables are read once, when the container starts. A running Pod keeps the old password until it is replaced.

### Deployment reloader

comes with an add-on, apps/v1, namespace reloader

Stakater Reloader. It watches Secrets and ConfigMaps, and when one changes it patches the Pod template of the workloads that use it, which starts an ordinary rolling update.

## How the resources connect

- ClusterSecretStore aws-prod → ServiceAccount external-secrets: serviceAccountRef (references)
- ClusterSecretStore aws-prod → AWS Secrets Manager prod/checkout/database: GetSecretValue (traffic)
- ExternalSecret checkout-db → ClusterSecretStore aws-prod: secretStoreRef (references)
- ExternalSecret checkout-db → Secret checkout-db: creates and refreshes (creates)
- Deployment checkout → Secret checkout-db: envFrom (references)
- Deployment checkout → Pod ×3 checkout-…: ReplicaSet creates (creates)
- Deployment reloader → Secret checkout-db: watches (references)
- Deployment reloader → Deployment checkout: patches template (references)

## How it works

1. **The store authenticates.** The operator asks the API server for a token for its ServiceAccount and exchanges it for temporary AWS credentials for the role in the annotation. The store reports `Valid` once that works.
2. **The ExternalSecret fetches.** For each entry in `data`, the operator reads the remote secret and takes one property of its JSON. The `shop` namespace is labelled `env: prod`, so the store's conditions let it in.
3. **The Secret is written.** The template is rendered with the fetched values and the result becomes the Secret's data. The ExternalSecret turns `SecretSynced`, and `kubectl get externalsecret` shows the last refresh time.
4. **The Pods read it at start.** The Deployment's Pods start with `DATABASE_URL` in their environment. If the Secret doesn't exist yet, they wait in `CreateContainerConfigError` until it does.
5. **Rotation reaches the Secret.** The password is rotated upstream. Within the hour the next refresh sees a new value and updates the Secret. Nothing has restarted yet, and the running Pods still hold the old password.
6. **Reloader rolls the Pods.** Reloader sees the Secret change, finds the Deployment that uses it and adds a hash to the Pod template. The Deployment rolls, and the new Pods start with the new value.

## Common mistakes

- `SecretSyncedError` with `could not get secret data from provider` usually means the remote key or `property` is wrong, or the IAM role can't read that path. `kubectl describe externalsecret` shows the provider's message.
- A refresh updates the Secret, not the process. Environment variables need a restart; files mounted from the Secret update by themselves after about a minute, and the application still has to reread them.
- With `refreshInterval: 1h`, a rotated password can be stale for up to an hour. Keep the old credential valid for longer than the interval, or annotate the ExternalSecret to force a refresh.
- A ClusterSecretStore without `conditions` lets every namespace read everything its role can. Scope it by namespace label, or give teams their own SecretStore and role.
- `creationPolicy: Owner` deletes the Secret with the ExternalSecret. Removing the ExternalSecret from Git with pruning on takes the application's credentials with it.

## The manifests you write

```yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-prod
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
  conditions:
    - namespaceSelector:
        matchLabels:
          env: prod
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-db
  namespace: shop
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-prod
  target:
    name: checkout-db
    creationPolicy: Owner
    template:
      data:
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@db.shop.svc:5432/checkout"
  data:
    - secretKey: username
      remoteRef:
        key: prod/checkout/database
        property: username
    - secretKey: password
      remoteRef:
        key: prod/checkout/database
        property: password
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/acme/checkout:2.4.1
          envFrom:
            - secretRef:
                name: checkout-db
```