About this scenario
The pipeline that deploys web to the shop namespace ran helm upgrade web k8wiz/web --version 1.5.0 -f values-prod.yaml --wait. The new Pods never became ready, the CI runner hit its job timeout and killed Helm with SIGKILL.
Since then every helm upgrade fails with another operation (install/upgrade/rollback) is in progress. Nothing is in progress. Unlock the release without losing track of what is deployed, find why the upgrade never finished, and ship chart 1.5.0.
values-prod.yaml is in the editor. The chart repository is already added as k8wiz.
Objectives
- Find the revision that locks the release
- Unlock the release by rolling back to the last good revision
- Ship chart 1.5.0 with the production values
- All three Pods run hello 2.1 and none is stuck pulling an image
What you'll practice
- Each revision of a release is a Secret named
sh.helm.release.v1.NAME.vN. Helm writes it as pending-upgrade first, applies the manifest, then updates the status. A Helm process that dies in between leaves the pending status, and that status is the lock. helm rollback is the clean way out: it doesn't check the lock, and it puts back whatever the half-finished upgrade changed. Deleting the pending Secret also unlocks the release but leaves the cluster as the failed attempt left it.- Set the CI job timeout above Helm's
--timeout, so Helm reaches its own limit first, records failed and exits. A failed revision doesn't block the next upgrade. --rollback-on-failure, the Helm 4 name for --atomic, rolls a failed upgrade back by itself and turns on waiting.
How the playground works
The cluster runs entirely in your browser: a simulated API server, scheduler, kubelets and controllers, with virtual nodes in three zones. kubectl output follows the real formats, errors use the real messages, and the add-ons (CSI storage, Traefik, Envoy Gateway, cert-manager, Helm, Flux, Argo CD, Kyverno, Cilium, Istio) behave like their real counterparts for the common paths. It's a simulation, so some behavior is simplified. Your progress is saved in this browser.