About this scenario
orders calls inventory at http://inventory:8080/stock. Since a disk failed on worker-3 last night, about one call in three returns HTTP 500. The inventory Pod on that node can't read its local cache, but /healthz still answers, so its readiness probe passes and the Service keeps sending it traffic.
The node team replaces the disk tomorrow, and the inventory team asked you to leave that Pod running so they can debug it. Don't delete it, cordon the node or change the Deployment.
Make the mesh cope. orders should get no errors, and its sidecar should stop sending traffic to the broken Pod. Test with kubectl exec deploy/orders -- curl -s http://inventory:8080/stock.
Objectives
- Find which inventory Pod answers 500
- orders' sidecar ejects the failing Pod
- orders gets no errors, even from a sidecar that hasn't ejected the Pod yet
- The failing Pod keeps running on worker-3, and the Deployment is unchanged
What you'll practice
- A readiness probe only tests what it calls. A Pod whose
/healthz works and whose real endpoint fails stays in the Service, and Kubernetes keeps sending it a share of the traffic. - Istio's default retry policy is 2 attempts on connection failures and 503. A 500 isn't retried unless the route's
retryOn covers it: 5xx does, gateway-error doesn't. - Outlier detection runs in each client's sidecar. After
consecutive5xxErrors errors in a row from one endpoint, that sidecar ejects it for baseEjectionTime, longer each time it's ejected again, then tries it again. - Retries hide errors from users, and ejection stops the traffic to the bad Pod. Retries alone keep sending a share of the attempts to a broken Pod, and ejection alone lets a few requests fail before it starts. Use both.
istioctl proxy-config endpoint shows what one sidecar thinks of each endpoint. OUTLIER CHECK: FAILED means that sidecar has ejected it.
How the playground works
The cluster runs entirely in your browser: a simulated API server, scheduler, kubelets and controllers, with virtual nodes in three zones. kubectl output follows the real formats, errors use the real messages, and the add-ons (CSI storage, Traefik, Envoy Gateway, cert-manager, Helm, Flux, Argo CD, Kyverno, Cilium, Istio) behave like their real counterparts for the common paths. It's a simulation, so some behavior is simplified. Your progress is saved in this browser.