# Fix an Istio canary that fails one request in ten

> A VirtualService sends 10% of traffic to reviews v2, and those requests get 503. Read Envoy's response flag, fix the subset, and get beta testers onto v2.

Level: Advanced · Topic: Networking · About 20 minutes
Try it in the browser: https://k8wiz.com/playground/fix-an-istio-canary-with-virtualservice-and-destinationrule-subsets

The playground is a simulated Kubernetes cluster that runs in the browser. kubectl output and errors follow the real ones.

## The situation

`reviews` in the `shop` namespace runs two Deployments behind one Service: `reviews-v1` with 3 Pods and `reviews-v2` with 1. The team wants a canary: 10% of requests on v2, and everyone who sends the header `x-user-group: beta` on v2 every time.

A teammate applied a DestinationRule with a `v1` and a `v2` subset and a VirtualService with the weights. Since then about one request in ten from `productpage` fails with a 503, and beta testers say they never see v2.

Fix both problems with Istio's own tools. Test with `kubectl exec deploy/productpage -- curl -s http://reviews:8080/`: the JSON says which version answered. `istioctl analyze` and the sidecar's access log (`kubectl logs deploy/productpage -c istio-proxy`) are good places to start.

## Objectives

1. Find what answers the 503s (The response flag in `kubectl logs deploy/productpage -c istio-proxy`, or `istioctl analyze`)
2. No request from productpage to reviews fails
3. About 10% of requests reach v2 (Sampled from productpage: between 5% and 20%)
4. Requests with `x-user-group: beta` always reach v2
5. Both Deployments keep their replicas, and the VirtualService still does the split

## Hints

1. Send a few requests, then read the client's sidecar: `kubectl logs deploy/productpage -c istio-proxy --tail=5`. The failing lines read `503 UH no_healthy_upstream` with the cluster `outbound|8080|v2|reviews.shop.svc.cluster.local`. UH means the cluster exists but has no endpoint to send to.
2. `istioctl analyze -n shop` says the same thing as IST0173: the `v2` subset selects no Pods. Compare its labels with the Pods' labels: `kubectl get pods -L version`. The v2 Pods carry `version: "2.0"`, not `version: v2`.
3. The Deployment's selector is immutable, so fix the DestinationRule: set the `v2` subset's labels to `version: "2.0"` in `reviews-destinationrule.yaml` and apply it. `istioctl proxy-config endpoint deploy/productpage --cluster "outbound|8080|v2|reviews.shop.svc.cluster.local"` should then list one endpoint.
4. Envoy tries HTTP routes in order and takes the first that matches. The `canary` route has no `match`, so it catches everything and `beta-testers` below it never runs. `istioctl analyze` reports that as IST0130. Move `beta-testers` above `canary`.
5. Check the split with a loop: `for i in $(seq 1 20); do kubectl exec deploy/productpage -- curl -s http://reviews:8080/ | grep version; done`.

## Solution

```bash
for i in $(seq 1 10); do kubectl exec deploy/productpage -- curl -s -o /dev/null -w '%{http_code}' http://reviews:8080/; done
```
```bash
kubectl logs deploy/productpage -c istio-proxy --tail=10
```
```bash
istioctl analyze -n shop
```
```bash
kubectl get pods -L version
```
```bash
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
  namespace: shop
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: "2.0"
EOF
```
```bash
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
  namespace: shop
spec:
  hosts:
  - reviews
  http:
  - name: beta-testers
    match:
    - headers:
        x-user-group:
          exact: beta
    route:
    - destination:
        host: reviews
        subset: v2
  - name: canary
    route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10
EOF
```
```bash
istioctl analyze -n shop
```
```bash
kubectl exec deploy/productpage -- curl -s -H 'x-user-group: beta' http://reviews:8080/
```

## What it teaches

- A subset is a label selector inside a DestinationRule. A VirtualService routes to it by name, and Envoy builds one cluster per subset, named `outbound|PORT|SUBSET|HOST`.
- Envoy's response flag says which part failed. `UH` means the cluster has no healthy endpoint, usually a subset whose labels match no Pod. `NC` means the cluster doesn't exist at all, usually a subset no DestinationRule defines.
- HTTP routes are tried in order and the first match wins. A route without `match` catches everything, so it belongs last. `istioctl analyze` flags the rules below it with IST0130.
- Weights apply to requests, not Pods. With 3 v1 Pods and 1 v2 Pod, plain Kubernetes load balancing gives v2 25% of the traffic. The VirtualService gives it the 10% you asked for, whatever the replica counts.
- When you add a subset, apply the DestinationRule first and the VirtualService a few seconds later. In the other order, proxies briefly route to a cluster they don't have yet.
