# How to troubleshoot 503 errors in Istio with Envoy response flags

> Read the Envoy response flag in Istio access logs to find why a request got a 503, then fix the route, endpoint, TLS or circuit breaker behind it.

Published: 2026-11-19
Author: k8wiz Team
URL: https://k8wiz.com/articles/troubleshoot-istio-503-errors-with-envoy-response-flags

A 503 inside an Istio mesh rarely comes from your application. Most of the time an Envoy proxy generated it, either the sidecar next to the caller or the one next to the server, because it could not find a route, a healthy endpoint or a working connection. The status code alone tells you almost nothing. The response flag that Envoy writes next to it tells you which of those things went wrong.

This guide shows how to get that flag, how to tell which proxy wrote it, and what to check for each of the flags you will see most often. The commands work with a sidecar mesh installed with `istioctl` or Helm.

## Step 1: turn on access logs

Envoy writes one access log line per request, and that line carries the response flag. Istio does not always have access logs enabled, so start there. The Telemetry API is the recommended way, and a resource in the root namespace applies to the whole mesh.

```yaml
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy
```

On a busy mesh, logging every request is expensive. You can keep only the failures with a CEL filter. Requests that fail before any response exists have no `response.code` attribute, so the expression has to handle that case as well, or you will lose the connection failures you are hunting for.

```yaml
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy
      filter:
        expression: "!has(response.code) || response.code >= 500"
```

If you installed Istio with an `IstioOperator` file and prefer mesh config, setting `meshConfig.accessLogFile: /dev/stdout` has the same basic effect. Pick one method and stick to it, so nobody has to guess where logging is configured.

## Step 2: find the flag and the proxy that set it

The logs go to the standard output of the `istio-proxy` container. In Istio's default text format, the response code comes right after the quoted request line, and the response flags come right after the code. A `-` means no flag was set.

Since URL paths cannot contain raw spaces, the code is the fifth field and the flag is the sixth. This one-liner counts the flags on recent 503s from a deployment's sidecar:

```bash
kubectl logs deploy/productpage -c istio-proxy -n default --since=15m \
  | awk '$5 == 503 {print $6}' | sort | uniq -c | sort -rn
```

Run it on both sides of the call. The client sidecar logs the request on its way out, and its upstream cluster looks like `outbound|9080||reviews.default.svc.cluster.local`. The server sidecar logs it on the way in, with an upstream cluster like `inbound|9080||`. If the 503 appears only in the client log, the request never reached the server pod, and you should focus on routing and endpoints. If the server sidecar logged it too, look at the server side: its listener, its protocol handling or the application itself.

The default format also includes `%RESPONSE_CODE_DETAILS%` and `%UPSTREAM_TRANSPORT_FAILURE_REASON%`. When the flag points at a connection or TLS problem, those two fields often name the exact error, so read the whole line before you move on.

A response code of 503 with flag `-` and `via_upstream` in the details means Envoy passed the 503 through from the application. In that case, the mesh is fine and the bug is in the service.

## Step 3: NR or NC, no route or no cluster

`NR` means Envoy found no route for the request. `NC` means the route existed but pointed at a cluster the proxy does not have. Both are configuration problems, and nothing on the network will fix them.

Start with `istioctl analyze`. It catches the common mistakes, such as a VirtualService that references a host, gateway or subset that does not exist. Those show up as `IST0101` (ReferencedResourceNotFound).

```bash
istioctl analyze --all-namespaces
```

Then look at what the client proxy actually has. Route tables in a sidecar are named after the port, so for a service on port 9080 you filter on that name.

```bash
istioctl proxy-config routes deployment/productpage.default --name 9080 -o json
istioctl proxy-config cluster deployment/productpage.default --fqdn reviews.default.svc.cluster.local
```

A missing host in the route output usually means the VirtualService `hosts` field does not match what the caller puts in the Host header, or the VirtualService is not exported to the caller's namespace through `exportTo`. A route present but a cluster missing usually means the DestinationRule that defines the subset is missing or is not visible to the caller.

## Step 4: UH, no healthy upstream

`UH` means the cluster exists but has no healthy endpoint to send to. Either nothing matches it, or everything that matches has been marked unhealthy. Ask the proxy what it sees for that exact cluster. Cluster names follow the pattern `direction|port|subset|host`, and the subset slot is empty when no subset applies.

```bash
istioctl proxy-config endpoints deployment/productpage.default \
  --cluster "outbound|9080|v2|reviews.default.svc.cluster.local"
```

An empty list for a subset cluster is the classic cause. The DestinationRule defines `v2` with a label like `version: v2`, but no running pod carries that label, perhaps because a rollout renamed the label or the v2 deployment was scaled to zero. `istioctl analyze` reports this as `IST0173`, and the message says outright that it may lead to 503 UH.

If the endpoints are listed but marked unhealthy, check `outlierDetection` in the DestinationRule. With aggressive settings such as `consecutive5xxErrors: 1` and `maxEjectionPercent: 100`, a short burst of errors can eject every pod at once. The service then returns 503 UH for the whole `baseEjectionTime`, even after the pods recover.

## Step 5: UF and URX, the connection failed

`UF` means the proxy could not open a connection to the upstream. `URX` means Envoy gave up because it hit the retry limit. They often travel together, because by default Istio retries a failed request twice, and `connect-failure` is one of the default retry conditions. A pod that refuses connections therefore produces a burst of failed attempts before the caller sees a 503.

The first thing to rule out is a TLS mismatch. A DestinationRule that sets `trafficPolicy.tls.mode: DISABLE` makes the client send plain text. If the server's PeerAuthentication is `STRICT`, the server sidecar rejects that connection. The same thing happens when automatic mTLS is turned off and a DestinationRule has no `tls` block at all, since the mode then defaults to `DISABLE`. The fix is to let auto mTLS do its job or to set the mode explicitly.

```yaml
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
  namespace: default
spec:
  host: reviews.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
```

If TLS is not the problem, check the application. Make sure the container listens on the `targetPort` the Service declares. A container that listens on a different port, or that has not finished starting, refuses the connection, and Envoy reports that as `UF`. List the EndpointSlices of the Service to see which pod IPs and ports it points at, then compare them with what the container really opens.

```bash
kubectl get endpointslices -n default -l kubernetes.io/service-name=reviews -o wide
```

## Step 6: UO, the circuit breaker tripped

`UO` stands for upstream overflow. The request was rejected before it left the client proxy because a limit from `connectionPool` in the DestinationRule was reached. This is the circuit breaker doing what you told it to do, so the question is whether the limits fit the real load.

The proxy stats show how often it happened. A rising `upstream_rq_pending_overflow` counter on the cluster confirms that requests are being rejected for lack of room in the pending queue.

```bash
kubectl exec deploy/productpage -c istio-proxy -n default -- \
  pilot-agent request GET stats | grep "reviews" | grep pending
```

Values like `maxConnections: 1` and `http1MaxPendingRequests: 1` make good demos but break real services. Raise the limits to fit measured peak concurrency plus some headroom.

```yaml
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
  namespace: default
spec:
  host: reviews.default.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 200
      http:
        http1MaxPendingRequests: 100
```

## Step 7: UC and DPE, protocol trouble

`UC` means the upstream connection was terminated while the request was in flight. `DPE` means the downstream request had an HTTP protocol error. Both often trace back to Istio treating a port as the wrong protocol.

Istio picks the protocol from the Service port. You set it with `appProtocol`, or with a port name like `http-web` or `tcp-db` (the `<protocol>[-<suffix>]` pattern). When both are set, `appProtocol` wins. Without either, the sidecar sniffs the traffic, and server-first protocols such as MySQL do not work with sniffing. `istioctl analyze` flags unnamed ports as `IST0118`. Declaring the protocol explicitly removes a whole class of surprises.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: default
spec:
  selector:
    app: payments
  ports:
  - name: http-api
    port: 8080
    appProtocol: http
  - name: tcp-db
    port: 5432
```

The Istio docs describe a specific `UC` case with headless services. A client calls a StatefulSet pod by its IP on an `http` port, so the Host header is that IP. The server sidecar finds no route for that host and fails with `503 UC`. Calling the pod by its DNS name (`web-0.nginx.default`), sending a proper Host header, or declaring the port as `tcp` all fix it.

`DPE` also shows up, with a 400 rather than a 503, when an application sends HTTPS to a port declared as `http`, for example a ServiceEntry for an external API on port 443 with `protocol: HTTP`. Envoy tries to parse encrypted bytes as HTTP and fails. Declare the port as `HTTPS` (or `TLS`) instead.

## Step 8: make sure the config reached the proxies

Everything above assumes the proxy runs the configuration you think it does. Istio pushes config in an eventually consistent way, so check that before you blame a resource.

```bash
istioctl proxy-status
```

`SYNCED` means the proxy acknowledged the last config istiod sent. `STALE` means istiod sent an update and got no acknowledgement, which usually points at a networking problem between the proxy and istiod, or at istiod itself being overloaded.

Propagation delay also explains 503s that appear for a few seconds during a change and then vanish. If a VirtualService that routes to a new subset reaches the proxies before the DestinationRule that defines it, Envoy refers to a cluster that does not exist yet. Applying both in one `kubectl apply` does not prevent this. Use make-before-break: add the subset to the DestinationRule first, wait a few seconds, then update the VirtualService. When removing a subset, reverse the order. That habit removes most of the transient 503s people see during canary rollouts.
