# How to debug OOMKilled containers in Kubernetes > Tell a container OOM kill from a node OOM and a kubelet eviction, find the real memory culprit, and fix limits without guessing. Published: 2026-11-12 Author: k8wiz Team URL: https://k8wiz.com/articles/debug-oomkilled-containers-in-kubernetes A pod restarts every few minutes and `kubectl get pods` shows `OOMKilled` in the status column, or just a climbing restart count. The quick fix is to double the memory limit and move on. Sometimes that's right. Often it only buys a few days, because the limit was never the problem. Kubernetes has three different ways of killing a workload for memory, and they leave different traces. This guide walks through telling them apart, finding out what actually used the memory, and choosing between a bigger limit, a runtime setting and a code fix. Everything here applies to current releases (Kubernetes 1.37 at the time of writing) on Linux nodes with cgroup v2. ## Three ways memory kills a pod The first is a container OOM kill. Each container with a memory limit gets its own cgroup, and the kernel enforces that limit. When the container's usage hits the limit and the kernel can't reclaim enough page cache, the kernel's OOM killer kills it. The pod stays on the node and the kubelet restarts the container according to the pod's `restartPolicy`. The second is a node OOM kill. Here no single container hit its limit, but the node as a whole ran out of memory before the kubelet could react. The kernel picks a victim across the whole machine, guided by an `oom_score_adj` value the kubelet assigns to every container based on its QoS class. The third is a kubelet eviction. The kubelet watches node memory and, when `memory.available` drops below a threshold (`100Mi` by default on Linux), it evicts whole pods to protect the node. An evicted pod is not restarted. Its phase becomes `Failed` and its controller creates a replacement, possibly on another node. The fix for each is different, so identify which one you have before changing anything. ## Step 1: read the last termination state The container's previous run is recorded in the pod status under `lastState`. `kubectl describe` shows it in readable form. ```bash kubectl describe pod api-7d9c6b8f4-x2klp -n shop ``` ```text State: Running Started: Thu, 12 Nov 2026 09:41:07 +0100 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Thu, 12 Nov 2026 09:33:52 +0100 Finished: Thu, 12 Nov 2026 09:41:05 +0100 Ready: True Restart Count: 6 Limits: memory: 512Mi Requests: cpu: 250m memory: 256Mi ``` Exit code 137 means the process died from signal 9 (`SIGKILL`), since shell convention reports a signal death as 128 plus the signal number. `SIGKILL` alone doesn't prove an OOM kill, because the kubelet also sends it when a container ignores `SIGTERM` past its grace period. The `OOMKilled` reason is the part that matters. The container runtime sets it when the kernel recorded an OOM kill in the container's cgroup. When a Deployment has many replicas, find all the affected containers at once with `jq`. ```bash kubectl get pods -A -o json | jq -r ' .items[] | .metadata.namespace as $ns | .metadata.name as $pod | .status.containerStatuses[]? | select(.lastState.terminated.reason == "OOMKilled") | "\($ns)/\($pod) \(.name) restarts=\(.restartCount)"' ``` If the reason is `OOMKilled` and the container sits on a limit, as above, you're dealing with a container OOM kill. Skip to step 3. ## Step 2: rule out evictions and node OOM kills Evicted pods look different. They don't restart; they sit in the `Failed` phase with the reason `Evicted`, and the message names the resource the node was short of. ```bash kubectl get pods -A --field-selector=status.phase=Failed kubectl get pod api-7d9c6b8f4-q8wzn -n shop -o jsonpath='{.status.reason}{"\n"}{.status.message}{"\n"}' ``` ```text Evicted The node was low on resource: memory. Threshold quantity: 100Mi, available: 84312Ki. Container api was using 1123044Ki, request is 256Mi, has larger consumption of memory. ``` That last sentence is useful. The kubelet ranks eviction candidates first by whether their usage exceeds their requests, then by priority, then by how far over their requests they are. A container using 1 GiB against a 256 MiB request is a natural first victim. Note also that node-pressure eviction ignores PodDisruptionBudgets, and with hard thresholds it uses a zero-second grace period. A node OOM kill leaves an event on the node rather than on the pod. The kubelet watches the kernel log and records a `SystemOOM` warning when the OOM killer fires outside any container's own limit. ```bash kubectl get events -A --field-selector=reason=SystemOOM kubectl describe node worker-3 | grep -A3 -i oom ``` ```text Warning SystemOOM 4m kubelet System OOM encountered, victim process: java, pid: 48211 ``` Both evictions and node OOM kills point at the same root cause. Pods on that node use far more memory than they requested, so the scheduler packed the node based on numbers that don't reflect reality. Raising limits won't help here. Raising requests to match real usage will. ## Step 3: measure what the container really uses Before you change a limit, find out how close normal usage runs to it. `kubectl top` reads from metrics-server, which reports the working set, an estimate of the memory that can't be freed under pressure. ```bash kubectl top pod -n shop -l app=api --containers ``` ```text POD NAME CPU(cores) MEMORY(bytes) api-7d9c6b8f4-x2klp api 212m 488Mi api-7d9c6b8f4-m4tqz api 187m 301Mi api-7d9c6b8f4-v7hcd api 204m 455Mi ``` A one-off reading tells you little. What you want is the shape over time, and that comes from your Prometheus or whatever scrapes cAdvisor, using `container_memory_working_set_bytes`. A steady climb that resets at each restart is a leak. A flat line with sharp spikes is a burst, usually a large request, a batch job or a cache warming up. A flat line just below the limit is a container that was simply sized too small. For a closer look, read the cgroup files from inside the container. On cgroup v2 they sit directly under `/sys/fs/cgroup`. ```bash kubectl exec -n shop api-7d9c6b8f4-x2klp -c api -- cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current kubectl exec -n shop api-7d9c6b8f4-x2klp -c api -- cat /sys/fs/cgroup/memory.events ``` ```text 536870912 511705088 low 0 high 0 max 1843 oom 6 oom_kill 6 oom_group_kill 6 ``` `memory.max` is the 512 MiB limit in bytes. In `memory.events`, `max` counts how often usage hit the limit and the kernel had to reclaim, `oom` counts how often reclaim failed, and `oom_kill` counts processes killed. A large `max` with few `oom` events means the container lives right at its limit and survives on page cache reclaim, which usually also shows up as poor latency. Remember that memory isn't only your heap. Files written to an `emptyDir` with `medium: Memory` live in tmpfs and count against the memory limit of the container that wrote them. Logs or temporary uploads spooled to a RAM-backed volume are a classic surprise. ## Step 4: know how the kernel picks what to kill On cgroup v2 nodes the kubelet sets `memory.oom.group` on each container's cgroup. When the OOM killer fires in that cgroup, it kills every process in the container together instead of picking one. This is a deliberate choice. A web server with worker processes would otherwise lose one worker and limp on in a broken state, with no restart and no `OOMKilled` status to tell you. If you run something that is designed to survive losing a child process, such as a supervisor that respawns workers, you can restore the old per-process behavior with the `singleProcessOOMKill` field in the kubelet configuration. It applies to the whole node, so most clusters should leave it at its default of `false`. ```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration singleProcessOOMKill: true ``` For node OOM kills, the victim depends on QoS. Containers in Guaranteed pods get an `oom_score_adj` of -997, BestEffort containers get 1000, and Burstable containers get a value between 2 and 999 that drops as their memory request grows relative to node capacity. So a Burstable container with a tiny request and a large footprint is the first to go when the node runs dry. ## Step 5: fix the runtime before the limit Many OOM kills come from a runtime that doesn't know it has a limit, or sizes itself as if it had the whole limit for the heap. Fix that first, because it changes what any limit means. Modern JDKs read the container's cgroup limit, but the default maximum heap is only 25% of it. Teams then set `-Xmx` close to the limit and forget that metaspace, thread stacks, direct buffers and the JIT live outside the heap. Setting the heap as a percentage keeps it proportional when you change the limit. ```yaml containers: - name: api image: registry.example.com/shop/api:2.4.0 env: - name: JAVA_TOOL_OPTIONS value: "-XX:MaxRAMPercentage=75.0" resources: requests: memory: 768Mi limits: memory: 768Mi ``` Go programs can set a soft limit with `GOMEMLIMIT`, which makes the garbage collector work harder as usage approaches it. Set it around 90% of the container limit so the collector reacts before the kernel does. Node.js has its own heap cap through `--max-old-space-size`, in megabytes. If memory climbs steadily across hours no matter the setting, you have a leak, and no limit will fix it. Take a heap dump or profile before the next kill and fix the code. ## Step 6: set requests and limits from real data Once the runtime is configured, size the container from what you measured. A reasonable starting point is a limit 20 to 30% above the peak working set you saw over a week of normal traffic, and a request at or close to typical usage. Setting the memory request equal to the limit is worth considering for anything important. Memory is not compressible like CPU, so a container that bursts above its request is borrowing memory the scheduler promised to someone else. Equal requests and limits (on CPU as well) make the pod Guaranteed, which gives it the lowest OOM priority and puts it last in line for eviction. The cost is that you pay for the peak all the time. On Kubernetes 1.33 and later you can change a running pod's resources in place through the `resize` subresource, which needs a `kubectl` client of at least 1.32. This is handy to confirm a new limit stops the kills before you change the Deployment. ```bash kubectl patch pod api-7d9c6b8f4-x2klp -n shop --subresource resize --patch \ '{"spec":{"containers":[{"name":"api","resources":{"requests":{"memory":"768Mi"},"limits":{"memory":"768Mi"}}}]}}' ``` The kubelet makes only a best-effort check when you lower a memory limit, so don't shrink a container that is already near its new value. And a pod-level resize is temporary by nature, since the next rollout recreates the pod from the Deployment template. Once the numbers hold, put them in the template. ```bash kubectl set resources deployment/api -n shop -c api --requests=memory=768Mi --limits=memory=768Mi ``` Finally, keep an alert on `OOMKilled` restarts rather than discovering them during an incident. A container that is OOM-killed once a week is telling you its limit sits right at the edge of normal traffic, and the next traffic spike will make it fail far more often.