# How to troubleshoot dropped traffic in Cilium with Hubble

> Find out why Cilium drops a pod's traffic: catch the drop with Hubble, read the drop reason, find the policy behind it and fix it.

Published: 2026-11-19
Author: k8wiz Team
URL: https://k8wiz.com/articles/troubleshoot-dropped-traffic-in-cilium-with-hubble

A pod can't reach a Service, the request hangs, and the application logs say nothing useful. On a cluster that runs Cilium, the packet was most likely dropped inside an eBPF program on one of the nodes. Your usual tools don't see that. There is no iptables rule to count, and `tcpdump` in the pod only shows packets leaving and never coming back.

Cilium records every drop it makes, together with a reason code, and Hubble lets you read those records. This guide walks through the order we use on real incidents: make sure Hubble can see the traffic, catch the drop, read the reason, then find the policy or Service behind it. The commands and field names come from the Cilium 1.20 source and documentation.

## Step 1: check that Hubble is running and reachable

Hubble runs inside each Cilium agent and keeps recent flows in a ring buffer. Hubble Relay collects those buffers from every node so you can query the whole cluster from your laptop. Start by checking that both are healthy.

```bash
cilium status
hubble status -P
```

The `-P` flag (`--port-forward`) makes the Hubble CLI open a port-forward to Hubble Relay on `localhost:4245` for you. Without it, run `cilium hubble port-forward` in another terminal first. A healthy `hubble status` looks like this:

```text
Healthcheck (via 127.0.0.1:4245): Ok
Current/Max Flows: 11917/12288 (96.98%)
Flows/s: 11.74
Connected Nodes: 3/3
```

Read two lines here. "Connected Nodes" should match your node count, or flows from the missing nodes will never show up. "Current/Max Flows" tells you how much history you have. The buffer is small (the Helm default for `hubble.eventBufferCapacity` is 4095 flows per node) and the oldest flows are overwritten first. On a busy node that can mean only a few seconds of history, so reproduce the problem while you watch rather than searching for something that happened an hour ago.

One more check saves time. Hubble only sees pods that Cilium manages. A pod with `hostNetwork: true`, or one started before Cilium was installed and never restarted, will not appear in flows at all.

## Step 2: catch the drop while you reproduce it

Open a stream of dropped flows for the namespace you care about, then trigger the failing request from a second terminal.

```bash
hubble observe -P --namespace shop --verdict DROPPED --follow
```

```bash
kubectl -n shop exec deploy/frontend -- curl -sS -m 5 http://payments:8080/health
```

Every line that appears is a flow Cilium dropped. The Cilium documentation shows output like this for a client blocked by an ingress policy:

```text
May  4 13:23:47.852: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
May  4 13:23:48.854: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
```

Each line holds a timestamp, the source and destination (pod names when Hubble can resolve them), the drop reason, the verdict and a summary. The repeated SYN with a one-second gap is the client's TCP stack retrying. That retry pattern is what a timeout looks like from the inside. By default Cilium drops policy-denied packets silently, so the client waits instead of getting an error.

If nothing shows up, widen the filter before you conclude that Cilium is innocent. Drop `--namespace` and use `--pod shop/frontend` (which matches both directions), or `--to-ip` with the Service or backend IP. The arrow tells you about direction: `->` is a request, `<-` a reply, and `<>` means Hubble couldn't tell.

## Step 3: read the drop reason

The text before `DROPPED` is the drop reason, and it decides where you look next. In JSON output the same value appears as `drop_reason_desc`, an enum name such as `POLICY_DENIED`. You can filter on it directly:

```bash
hubble observe -P --drop-reason-desc POLICY_DENIED --last 50
```

These are the reasons we see most often in practice, with their enum names from the Hubble flow API:

- `Policy denied` (`POLICY_DENIED`, code 133): the endpoint is in default-deny for that direction and no rule allows this traffic.
- `Policy denied by denylist` (`POLICY_DENY`, code 181): an explicit `ingressDeny` or `egressDeny` rule matched.
- `Service backend not found` (`SERVICE_BACKEND_NOT_FOUND`, code 158): the client hit a Service address that has no usable backend.
- `Stale or unroutable IP` (`STALE_OR_UNROUTABLE_IP`, code 151): the destination IP doesn't belong to a known endpoint, often a pod that was just deleted.
- `CT: Map insertion failed` (code 155): the connection tracking map is full.
- `Authentication required` (`AUTH_REQUIRED`, code 189): a policy asks for mutual authentication that hasn't completed.

The two policy reasons are worth telling apart. `POLICY_DENIED` means something is missing, so the fix is an allow rule. `POLICY_DENY` means a rule actively blocks the traffic, and adding allow rules won't help, because Cilium gives deny rules precedence over allow rules from any policy type, Kubernetes NetworkPolicy included.

## Step 4: find the policy behind a policy drop

A policy drop tells you what was blocked but not which policy is involved. Hubble's policy verdict events carry that information. They are a separate event type that records the policy decision for a connection.

```bash
hubble observe -P --namespace shop --type policy-verdict --print-policy-names --last 20
```

For policy verdict events the verdict column reads `ALLOWED`, `DENIED` or `AUDITED`. With `--print-policy-names`, Hubble appends the policies it could match to the decision, for example `DENIED BY block-metadata (CiliumNetworkPolicy)`. This relies on network policy correlation, which the Helm value `hubble.networkPolicyCorrelation.enabled` turns on by default.

For the common `Policy denied` case there is often no name to print, because no rule matched at all. Then look at the policies that put the pod into default-deny. Any policy that selects a pod and has an `ingress` section switches ingress to default-deny for that pod; the same goes for egress. List the candidates and check which ones select your pod's labels:

```bash
kubectl -n shop get ciliumnetworkpolicies,networkpolicies
kubectl get ciliumclusterwidenetworkpolicies
kubectl -n shop get pod -l app=payments --show-labels
```

The agent on the pod's node can confirm what it actually enforces. `cilium-dbg endpoint list` shows, for each endpoint, whether ingress and egress enforcement is on, in its `POLICY (ingress) ENFORCEMENT` and `POLICY (egress) ENFORCEMENT` columns.

```bash
NODE=$(kubectl -n shop get pod -l app=payments -o jsonpath='{.items[0].spec.nodeName}')
AGENT=$(kubectl -n kube-system get pods -l k8s-app=cilium --field-selector spec.nodeName=$NODE -o name)
kubectl -n kube-system exec $AGENT -c cilium-agent -- cilium-dbg endpoint list
```

A frequent surprise is an egress policy written for one destination that silently cuts off DNS. Once a pod has any egress rule, its lookups to kube-dns are blocked unless a rule allows them. In Hubble that shows up as dropped UDP flows to port 53, which you can spot with `--to-port 53`. If you use `toFQDNs`, our guide on [restricting pod egress with toFQDNs](/articles/restrict-pod-egress-to-external-domains-with-cilium-tofqdns) covers the DNS rule it needs.

Before you push a fix to production, you can test it without risk by turning on policy audit mode (the Helm value `policyAuditMode`). Traffic that would be denied is then forwarded and reported with the `AUDIT` verdict, so `hubble observe --verdict AUDIT` shows exactly what the policy would block.

## Step 5: handle Service and routing drops

`Service backend not found` points away from policy and toward the Service itself. Check that it has ready endpoints:

```bash
kubectl -n shop get endpointslices -l kubernetes.io/service-name=payments
kubectl -n shop get pods -l app=payments -o wide
```

An empty slice usually means the selector doesn't match the pod labels or the pods are failing readiness. There is a less obvious cause too. For a Service with `externalTrafficPolicy: Local` (or `internalTrafficPolicy: Local`), Cilium's datapath drops traffic with this reason when the node that receives it has no local backend, even if other nodes have healthy pods. The Helm value `serviceNoBackendResponse` defaults to `reject`, so the client typically gets an ICMP error and fails fast instead of hanging.

`Stale or unroutable IP` during a rollout is usually harmless: clients still hold connections or cached addresses for pods that are gone. If it keeps happening long after the rollout, look for an application that caches pod IPs instead of using the Service name.

## Step 6: get the full record as JSON

The compact output hides most of what Hubble knows. For a single flow you want to quote in a ticket, ask for JSON and pick the fields with `jq`:

```bash
hubble observe -P --verdict DROPPED --namespace shop --last 1 -o jsonpb \
  | jq '.flow | {time, node_name, drop_reason_desc, traffic_direction, src: .source.pod_name, dst: .destination.pod_name, IP}'
```

The record includes the node that dropped the packet, the security identities of both sides and the traffic direction. The node name matters when only some nodes misbehave, because the next step is usually that node's agent logs.

If Hubble Relay itself is broken, you can still see drops on one node from inside its agent. `cilium-dbg monitor` reads the same event stream before Hubble processes it:

```bash
kubectl -n kube-system exec $AGENT -c cilium-agent -- cilium-dbg monitor --type drop
```

Its lines look like `xx drop (Policy denied) to endpoint 25729, identity 261->264: 10.11.13.37 -> 10.11.101.61 EchoRequest`, with numeric endpoint IDs and identities that you can map back with `cilium-dbg endpoint list`.

## Step 7: keep an eye on drops after the fix

Ad hoc queries only help while someone is looking. The agent already exports `cilium_drop_count_total` with `reason` and `direction` labels, enabled by default, so a Prometheus alert on a rising rate of `reason="Policy denied"` catches the next broken policy before users report it. Hubble has its own drop metric too: add `drop` to `hubble.metrics.enabled` and it appears as `hubble_drop_total` with `reason` and `protocol` labels, plus optional source and destination context.

Keep Cilium on the latest patch release while you're at it. Cilium 1.20.5, released on 17 November 2026, fixes the source identity reported in drop notifications for traffic to a Service with no backends, which makes the Step 5 case easier to trace. It also fixes egress rules that combine `toServices` with `toPorts`: when the referenced Service didn't exist or had no backends, those rules allowed traffic to any destination on those ports.
