# Route around a failing Pod with Istio retries and outlier detection

> One inventory Pod returns 500 but passes its readiness probe. Keep it running, and make the mesh stop sending it traffic with retries and outlier detection.

Level: Advanced · Topic: Networking · About 20 minutes
Try it in the browser: https://k8wiz.com/playground/route-around-a-failing-pod-with-istio-retries-and-outlier-detection

The playground is a simulated Kubernetes cluster that runs in the browser. kubectl output and errors follow the real ones.

## The situation

`orders` calls `inventory` at `http://inventory:8080/stock`. Since a disk failed on `worker-3` last night, about one call in three returns HTTP 500. The inventory Pod on that node can't read its local cache, but `/healthz` still answers, so its readiness probe passes and the Service keeps sending it traffic.

The node team replaces the disk tomorrow, and the inventory team asked you to leave that Pod running so they can debug it. Don't delete it, cordon the node or change the Deployment.

Make the mesh cope. `orders` should get no errors, and its sidecar should stop sending traffic to the broken Pod. Test with `kubectl exec deploy/orders -- curl -s http://inventory:8080/stock`.

## Objectives

1. Find which inventory Pod answers 500 (Its log, or the upstream address in orders' sidecar log, then `kubectl get pods -o wide`)
2. orders' sidecar ejects the failing Pod (`istioctl proxy-config endpoint deploy/orders --cluster "outbound|8080||inventory.shop.svc.cluster.local"`)
3. orders gets no errors, even from a sidecar that hasn't ejected the Pod yet (Retries that cover a 500)
4. The failing Pod keeps running on worker-3, and the Deployment is unchanged

## Hints

1. Send a few requests, then `kubectl logs deploy/orders -c istio-proxy --tail=6`. The 500s carry `via_upstream`, so the app sent them, and the field after the authority is the Pod that answered. `kubectl get pods -o wide` maps that IP to the Pod on worker-3, and `kubectl logs` on that Pod shows the cache error.
2. Istio retries by default, but only connection failures and 503. A 500 goes straight back to the caller. In a VirtualService for `inventory`, set `retries: {attempts: 2, retryOn: 5xx}`. `gateway-error` wouldn't help: it covers 502, 503 and 504 only. Istio sends each retry to a different Pod.
3. Retries still send a third of the attempts to the broken Pod. Outlier detection makes the sidecar eject it: a DestinationRule for `inventory` with `trafficPolicy.outlierDetection` set to `consecutive5xxErrors: 3`, `interval: 10s` and `baseEjectionTime: 60s`.
4. Ejection happens in each client's sidecar after it sees the errors, so send some traffic: `for i in $(seq 1 12); do kubectl exec deploy/orders -- curl -s -o /dev/null -w '%{http_code}' http://inventory:8080/stock; done`. Then `istioctl proxy-config endpoint deploy/orders --cluster "outbound|8080||inventory.shop.svc.cluster.local"` shows `FAILED` in the OUTLIER CHECK column for one endpoint.

## Solution

```bash
for i in $(seq 1 6); do kubectl exec deploy/orders -- curl -s -o /dev/null -w '%{http_code}' http://inventory:8080/stock; done
```
```bash
kubectl logs deploy/orders -c istio-proxy --tail=6
```
```bash
kubectl get pods -o wide
```
```bash
kubectl logs -l app=inventory --tail=2
```
```bash
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: inventory
  namespace: shop
spec:
  hosts:
  - inventory
  http:
  - route:
    - destination:
        host: inventory
    retries:
      attempts: 2
      retryOn: 5xx
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: inventory
  namespace: shop
spec:
  host: inventory
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 60s
EOF
```
```bash
for i in $(seq 1 12); do kubectl exec deploy/orders -- curl -s -o /dev/null -w '%{http_code}' http://inventory:8080/stock; done
```
```bash
istioctl proxy-config endpoint deploy/orders --cluster "outbound|8080||inventory.shop.svc.cluster.local"
```

## What it teaches

- A readiness probe only tests what it calls. A Pod whose `/healthz` works and whose real endpoint fails stays in the Service, and Kubernetes keeps sending it a share of the traffic.
- Istio's default retry policy is 2 attempts on connection failures and 503. A 500 isn't retried unless the route's `retryOn` covers it: `5xx` does, `gateway-error` doesn't.
- Outlier detection runs in each client's sidecar. After `consecutive5xxErrors` errors in a row from one endpoint, that sidecar ejects it for `baseEjectionTime`, longer each time it's ejected again, then tries it again.
- Retries hide errors from users, and ejection stops the traffic to the bad Pod. Retries alone keep sending a share of the attempts to a broken Pod, and ejection alone lets a few requests fail before it starts. Use both.
- `istioctl proxy-config endpoint` shows what one sidecar thinks of each endpoint. `OUTLIER CHECK: FAILED` means that sidecar has ejected it.
