# Fix a Helm release stuck in pending-upgrade

> Every helm upgrade fails with "another operation is in progress". Read the history, unlock the release with a rollback, fix the values and ship 1.5.0.

Level: Intermediate · Topic: Operations · About 12 minutes
Try it in the browser: https://k8wiz.com/playground/fix-a-helm-release-stuck-in-pending-upgrade

The playground is a simulated Kubernetes cluster that runs in the browser. kubectl output and errors follow the real ones.

## The situation

The pipeline that deploys `web` to the `shop` namespace ran `helm upgrade web k8wiz/web --version 1.5.0 -f values-prod.yaml --wait`. The new Pods never became ready, the CI runner hit its job timeout and killed Helm with `SIGKILL`.

Since then every `helm upgrade` fails with `another operation (install/upgrade/rollback) is in progress`. Nothing is in progress. Unlock the release without losing track of what is deployed, find why the upgrade never finished, and ship chart 1.5.0.

`values-prod.yaml` is in the editor. The chart repository is already added as `k8wiz`.

## Objectives

1. Find the revision that locks the release (`helm history web` or `helm list --pending`)
2. Unlock the release by rolling back to the last good revision (A rollback runs even when the latest revision is pending)
3. Ship chart 1.5.0 with the production values (The release is `deployed` on web-1.5.0, with 3 ready replicas)
4. All three Pods run hello 2.1 and none is stuck pulling an image

## Hints

1. `helm history web` shows revision 2 still `deployed` and revision 3 `pending-upgrade` with the description `Preparing upgrade`. Helm writes a revision as pending before it applies anything, and reads that status as a lock.
2. `helm rollback web 2 --wait` creates revision 4 from revision 2's manifest. The pending revision stays in the history, which is harmless: Helm only looks at the latest one.
3. `kubectl get pods` during the stuck upgrade shows one Pod in `ImagePullBackOff`. `kubectl describe pod` names the image: `ghcr.io/k8wiz/hello:2.4`. Run `images` to see which tags exist.
4. Chart 1.5.0 ships app version 2.1. Set `tag: "2.1"` in values-prod.yaml, or remove the `image` block so the chart's own default applies, then `helm upgrade web k8wiz/web --version 1.5.0 -f values-prod.yaml --wait`.

## Solution

```bash
helm history web
```
```bash
helm rollback web 2 --wait
```
```bash
sed -i 's/tag: "2.4"/tag: "2.1"/' values-prod.yaml
```
```bash
helm upgrade web k8wiz/web --version 1.5.0 -f values-prod.yaml --wait --timeout 3m
```

## What it teaches

- Each revision of a release is a Secret named `sh.helm.release.v1.NAME.vN`. Helm writes it as `pending-upgrade` first, applies the manifest, then updates the status. A Helm process that dies in between leaves the pending status, and that status is the lock.
- `helm rollback` is the clean way out: it doesn't check the lock, and it puts back whatever the half-finished upgrade changed. Deleting the pending Secret also unlocks the release but leaves the cluster as the failed attempt left it.
- Set the CI job timeout above Helm's `--timeout`, so Helm reaches its own limit first, records `failed` and exits. A failed revision doesn't block the next upgrade.
- `--rollback-on-failure`, the Helm 4 name for `--atomic`, rolls a failed upgrade back by itself and turns on waiting.