A 503 inside an Istio mesh rarely comes from your application. Most of the time an Envoy proxy generated it, either the sidecar next to the caller or the one next to the server, because it could not find a route, a healthy endpoint or a working connection. The status code alone tells you almost nothing. The response flag that Envoy writes next to it tells you which of those things went wrong.
This guide shows how to get that flag, how to tell which proxy wrote it, and what to check for each of the flags you will see most often. The commands work with a sidecar mesh installed with istioctl or Helm.
Step 1: turn on access logs
Envoy writes one access log line per request, and that line carries the response flag. Istio does not always have access logs enabled, so start there. The Telemetry API is the recommended way, and a resource in the root namespace applies to the whole mesh.
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
accessLogging:
- providers:
- name: envoy
On a busy mesh, logging every request is expensive. You can keep only the failures with a CEL filter. Requests that fail before any response exists have no response.code attribute, so the expression has to handle that case as well, or you will lose the connection failures you are hunting for.
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
accessLogging:
- providers:
- name: envoy
filter:
expression: "!has(response.code) || response.code >= 500"
If you installed Istio with an IstioOperator file and prefer mesh config, setting meshConfig.accessLogFile: /dev/stdout has the same basic effect. Pick one method and stick to it, so nobody has to guess where logging is configured.
Step 2: find the flag and the proxy that set it
The logs go to the standard output of the istio-proxy container. In Istio's default text format, the response code comes right after the quoted request line, and the response flags come right after the code. A - means no flag was set.
Since URL paths cannot contain raw spaces, the code is the fifth field and the flag is the sixth. This one-liner counts the flags on recent 503s from a deployment's sidecar:
kubectl logs deploy/productpage -c istio-proxy -n default --since=15m \
| awk '$5 == 503 {print $6}' | sort | uniq -c | sort -rn
Run it on both sides of the call. The client sidecar logs the request on its way out, and its upstream cluster looks like outbound|9080||reviews.default.svc.cluster.local. The server sidecar logs it on the way in, with an upstream cluster like inbound|9080||. If the 503 appears only in the client log, the request never reached the server pod, and you should focus on routing and endpoints. If the server sidecar logged it too, look at the server side: its listener, its protocol handling or the application itself.
The default format also includes %RESPONSE_CODE_DETAILS% and %UPSTREAM_TRANSPORT_FAILURE_REASON%. When the flag points at a connection or TLS problem, those two fields often name the exact error, so read the whole line before you move on.
A response code of 503 with flag - and via_upstream in the details means Envoy passed the 503 through from the application. In that case, the mesh is fine and the bug is in the service.
Step 3: NR or NC, no route or no cluster
NR means Envoy found no route for the request. NC means the route existed but pointed at a cluster the proxy does not have. Both are configuration problems, and nothing on the network will fix them.
Start with istioctl analyze. It catches the common mistakes, such as a VirtualService that references a host, gateway or subset that does not exist. Those show up as IST0101 (ReferencedResourceNotFound).
istioctl analyze --all-namespaces
Then look at what the client proxy actually has. Route tables in a sidecar are named after the port, so for a service on port 9080 you filter on that name.
istioctl proxy-config routes deployment/productpage.default --name 9080 -o json
istioctl proxy-config cluster deployment/productpage.default --fqdn reviews.default.svc.cluster.local
A missing host in the route output usually means the VirtualService hosts field does not match what the caller puts in the Host header, or the VirtualService is not exported to the caller's namespace through exportTo. A route present but a cluster missing usually means the DestinationRule that defines the subset is missing or is not visible to the caller.
Step 4: UH, no healthy upstream
UH means the cluster exists but has no healthy endpoint to send to. Either nothing matches it, or everything that matches has been marked unhealthy. Ask the proxy what it sees for that exact cluster. Cluster names follow the pattern direction|port|subset|host, and the subset slot is empty when no subset applies.
istioctl proxy-config endpoints deployment/productpage.default \
--cluster "outbound|9080|v2|reviews.default.svc.cluster.local"
An empty list for a subset cluster is the classic cause. The DestinationRule defines v2 with a label like version: v2, but no running pod carries that label, perhaps because a rollout renamed the label or the v2 deployment was scaled to zero. istioctl analyze reports this as IST0173, and the message says outright that it may lead to 503 UH.
If the endpoints are listed but marked unhealthy, check outlierDetection in the DestinationRule. With aggressive settings such as consecutive5xxErrors: 1 and maxEjectionPercent: 100, a short burst of errors can eject every pod at once. The service then returns 503 UH for the whole baseEjectionTime, even after the pods recover.
Step 5: UF and URX, the connection failed
UF means the proxy could not open a connection to the upstream. URX means Envoy gave up because it hit the retry limit. They often travel together, because by default Istio retries a failed request twice, and connect-failure is one of the default retry conditions. A pod that refuses connections therefore produces a burst of failed attempts before the caller sees a 503.
The first thing to rule out is a TLS mismatch. A DestinationRule that sets trafficPolicy.tls.mode: DISABLE makes the client send plain text. If the server's PeerAuthentication is STRICT, the server sidecar rejects that connection. The same thing happens when automatic mTLS is turned off and a DestinationRule has no tls block at all, since the mode then defaults to DISABLE. The fix is to let auto mTLS do its job or to set the mode explicitly.
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: reviews
namespace: default
spec:
host: reviews.default.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
If TLS is not the problem, check the application. Make sure the container listens on the targetPort the Service declares. A container that listens on a different port, or that has not finished starting, refuses the connection, and Envoy reports that as UF. List the EndpointSlices of the Service to see which pod IPs and ports it points at, then compare them with what the container really opens.
kubectl get endpointslices -n default -l kubernetes.io/service-name=reviews -o wide
Step 6: UO, the circuit breaker tripped
UO stands for upstream overflow. The request was rejected before it left the client proxy because a limit from connectionPool in the DestinationRule was reached. This is the circuit breaker doing what you told it to do, so the question is whether the limits fit the real load.
The proxy stats show how often it happened. A rising upstream_rq_pending_overflow counter on the cluster confirms that requests are being rejected for lack of room in the pending queue.
kubectl exec deploy/productpage -c istio-proxy -n default -- \
pilot-agent request GET stats | grep "reviews" | grep pending
Values like maxConnections: 1 and http1MaxPendingRequests: 1 make good demos but break real services. Raise the limits to fit measured peak concurrency plus some headroom.
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: reviews
namespace: default
spec:
host: reviews.default.svc.cluster.local
trafficPolicy:
connectionPool:
tcp:
maxConnections: 200
http:
http1MaxPendingRequests: 100
Step 7: UC and DPE, protocol trouble
UC means the upstream connection was terminated while the request was in flight. DPE means the downstream request had an HTTP protocol error. Both often trace back to Istio treating a port as the wrong protocol.
Istio picks the protocol from the Service port. You set it with appProtocol, or with a port name like http-web or tcp-db (the <protocol>[-<suffix>] pattern). When both are set, appProtocol wins. Without either, the sidecar sniffs the traffic, and server-first protocols such as MySQL do not work with sniffing. istioctl analyze flags unnamed ports as IST0118. Declaring the protocol explicitly removes a whole class of surprises.
apiVersion: v1
kind: Service
metadata:
name: payments
namespace: default
spec:
selector:
app: payments
ports:
- name: http-api
port: 8080
appProtocol: http
- name: tcp-db
port: 5432
The Istio docs describe a specific UC case with headless services. A client calls a StatefulSet pod by its IP on an http port, so the Host header is that IP. The server sidecar finds no route for that host and fails with 503 UC. Calling the pod by its DNS name (web-0.nginx.default), sending a proper Host header, or declaring the port as tcp all fix it.
DPE also shows up, with a 400 rather than a 503, when an application sends HTTPS to a port declared as http, for example a ServiceEntry for an external API on port 443 with protocol: HTTP. Envoy tries to parse encrypted bytes as HTTP and fails. Declare the port as HTTPS (or TLS) instead.
Step 8: make sure the config reached the proxies
Everything above assumes the proxy runs the configuration you think it does. Istio pushes config in an eventually consistent way, so check that before you blame a resource.
istioctl proxy-status
SYNCED means the proxy acknowledged the last config istiod sent. STALE means istiod sent an update and got no acknowledgement, which usually points at a networking problem between the proxy and istiod, or at istiod itself being overloaded.
Propagation delay also explains 503s that appear for a few seconds during a change and then vanish. If a VirtualService that routes to a new subset reaches the proxies before the DestinationRule that defines it, Envoy refers to a cluster that does not exist yet. Applying both in one kubectl apply does not prevent this. Use make-before-break: add the subset to the DestinationRule first, wait a few seconds, then update the VirtualService. When removing a subset, reverse the order. That habit removes most of the transient 503s people see during canary rollouts.