# Fix ImagePullBackOff from a private registry with imagePullSecrets

> Two Deployments can't pull from a private registry: 401 Unauthorized. Fix a pull secret that names the wrong server and attach it to the ServiceAccount.

Level: Intermediate · Topic: Security · About 12 minutes
Try it in the browser: https://k8wiz.com/playground/pull-images-from-a-private-registry

The playground is a simulated Kubernetes cluster that runs in the browser. kubectl output and errors follow the real ones.

## The situation

The `shop` namespace runs two images from the company registry, `registry.acme.io`, which refuses anonymous pulls. Both Deployments sit in `ImagePullBackOff`.

`checkout` names a pull secret, `regcred`, and still gets `401 Unauthorized`. `worker` names no secret at all, and the team doesn't want `imagePullSecrets` copied into every manifest.

The registry's robot account is in `registry-credentials.txt` in the editor. Get both Deployments running, with `worker`'s manifest left as it is.

## Objectives

1. Read why the pulls fail (`kubectl describe pod` or `kubectl get events`)
2. Find what's wrong inside the regcred secret (Decode `.dockerconfigjson` and compare the server with the image's registry)
3. checkout runs 2 ready replicas
4. worker runs 2 replicas, pulling through the ServiceAccount's secret
5. worker's Deployment still names no imagePullSecrets

## Hints

1. `kubectl describe pod -l app=checkout` ends with the registry's answer: `401 Unauthorized` while fetching an anonymous token. The kubelet sent no credentials, even though the Pod names `regcred`.
2. `kubectl get secret regcred -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d` prints the secret's contents. Credentials are stored per server, and the kubelet only uses the entry whose server matches the image's registry.
3. The secret says `registry.acme.com`. Recreate it: `kubectl delete secret regcred`, then `kubectl create secret docker-registry regcred --docker-server=registry.acme.io --docker-username=robot-shop-pull --docker-password=Xk29-pull-7dQ`. The kubelet retries the pull by itself.
4. For `worker`, attach the secret to the namespace's ServiceAccount: `kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"regcred"}]}'`. Pods get it when they are created, so `kubectl rollout restart deployment/worker` afterwards.

## Solution

```bash
kubectl describe pod -l app=checkout
```
```bash
kubectl get secret regcred -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
```
```bash
kubectl delete secret regcred
```
```bash
kubectl create secret docker-registry regcred --docker-server=registry.acme.io --docker-username=robot-shop-pull --docker-password=Xk29-pull-7dQ
```
```bash
kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"regcred"}]}'
```
```bash
kubectl rollout restart deployment/worker
```
```bash
kubectl rollout status deployment/worker
```
```bash
kubectl rollout status deployment/checkout
```

## What it teaches

- A `kubernetes.io/dockerconfigjson` secret holds credentials per registry server. The kubelet picks the entry that matches the image's registry, and one typo in the server name means it sends nothing.
- `401 Unauthorized` on an anonymous token means no credentials were sent. With credentials that are wrong, the registry rejects the token request for that user.
- Secrets are namespaced: a pull secret has to live in the Pod's namespace.
- `imagePullSecrets` on a ServiceAccount are added to every Pod that uses the account and names none of its own. That happens at admission, so existing Pods need to be recreated.