# Agent Substrate explained: how it suspends and resumes AI agents on Kubernetes

> What Agent Substrate is, how its actors, workers and snapshots fit together, how suspend and resume work, and where it differs from running one Pod per agent.

Published: 2026-11-18
Author: k8wiz Team
URL: https://k8wiz.com/articles/agent-substrate-explained-suspend-and-resume-ai-agents-on-kubernetes

An AI agent spends most of its life waiting. It waits for the model to answer, for a tool to return, for a human to type the next message. On plain Kubernetes each agent is usually a Pod, and that Pod keeps its memory and its slot on a node the whole time, busy or not. With ten thousand mostly idle agents, you pay for ten thousand Pods that are doing nothing.

[Agent Substrate](https://github.com/agent-substrate/substrate) takes a different approach. It snapshots idle agents out of memory into object storage and restores them, in under half a second by the project's numbers, onto a small pool of Pods that are already running. It runs on standard Kubernetes and reached 1.0 in November 2026. This article covers the concepts, the suspend and resume path, how it compares with plain Kubernetes, and how to try it on a laptop.

If you'd rather click through the moving parts, the [interactive graph of Agent Substrate](/graphs/agent-substrate-actors-workers-snapshots) draws every component with a step-by-step walkthrough of one request.

## The problem in one picture

```text
Plain Kubernetes: one Pod per agent, all of them resident all the time

  agent-1   Pod  [##........................]   busy 8% of the time
  agent-2   Pod  [.....#..............#.....]
  agent-3   Pod  [...........##.............]
  ...
  agent-N   Pod  [.................#........]   N Pods = N x memory

Agent Substrate: N actors in a bucket, a few warm workers

  object storage   [a1] [a2] [a3] [a4] ... [aN]   suspended: no CPU, no RAM
                     |   resume      ^ suspend
                     v               |
  WorkerPool       [worker-1] [worker-2] [worker-3]   only busy actors here
```

Three things make agents a bad fit for one Pod each:

- **They are bursty.** Work comes in short spikes, and the gaps between them can last minutes or days.
- **They run untrusted code.** An agent that writes and runs code needs a real sandbox, so you can't pack several users into one process.
- **There are a lot of them.** One agent per user, per session or per task adds up fast. Kubernetes documents limits of 110 Pods per node and 150,000 Pods per cluster.

## What Agent Substrate is

- An open source runtime (Apache 2.0) for sandboxed, stateful, mostly idle workloads, installed into a Kubernetes cluster in the `ate-system` namespace. `ate` is the prefix you'll see on every binary, header and CLI.
- A low-opinion runtime. It runs any OCI image, so it works with ADK, LangChain, MCP servers and coding agents such as Claude Code without code changes. It isn't an agent framework.
- Built around one idea: map many **actors** (agent instances) onto few **workers** (warm Pods), and move actors in and out with snapshots.
- Already used underneath other projects. [kagent](https://github.com/kagent-dev/kagent), a CNCF Sandbox project, uses it to run its agents, and so does Google's [Agent Executor](https://github.com/google/ax).

The project's published numbers:

| Metric | Today | Long-term target |
| --- | --- | --- |
| Resume latency | under 500 ms | 100 ms at p95 |
| Wakeups per second, one cluster | over 500 | 1,000 |
| Actors per cluster, active and idle | millions | 1 billion |
| Actors per worker Pod (oversubscription) | 10x to 30x | |

The demo video multiplexes about 250 stateful actors onto 8 worker Pods. Treat all of these as the project's own figures. Measure with your own images, because a 4 GiB memory snapshot restores slower than a 100 MiB one.

## Core concepts

| Concept | What it is | Stored in | Closest Kubernetes idea |
| --- | --- | --- | --- |
| Actor | One agent instance, addressed as `<atespace>/<name>`. The unit that is suspended and resumed. | PostgreSQL | A Pod you could freeze and move to another node |
| ActorTemplate | Immutable blueprint: images, volumes, CPU and memory limits, snapshot settings | PostgreSQL | Pod template |
| WorkerPool | Warm capacity: N worker Pods of one sandbox class | etcd (CRD) | A Deployment of placeholder Pods |
| Worker | The control plane's record of one worker Pod, its capacity and the actors on it | PostgreSQL | A Node, as an actor sees it |
| Atespace | Name scope and authorization boundary for actors, templates and tags | PostgreSQL | Namespace |
| SandboxConfig | The runtime versions for one sandbox class, gVisor or microVM | etcd (CRD) | RuntimeClass |
| Snapshot | Memory, root filesystem and volumes of a suspended actor | Object storage (GCS or S3) | Nothing. A VolumeSnapshot covers disks only. |
| Golden snapshot | A snapshot taken once when a template is created, so new actors skip the boot | Object storage | A pre-warmed image |
| Tag | A named, immutable copy of a snapshot that other atespaces can use | Object storage | An image tag, but for running state |

The table's third column is the part that trips people up. Only `WorkerPool`, `SandboxConfig` and `CSIDriverConfig` are Kubernetes objects. Actors, templates and workers are rows in Substrate's own PostgreSQL database, so `kubectl get actors` returns nothing. You use the `kubectl ate` plugin instead.

The components:

- **`ate-api-server`.** The control plane, a gRPC API. It owns actor state, picks workers and runs resume and suspend as workflows.
- **`atecontroller`.** A normal Kubernetes controller that turns each WorkerPool into a Deployment.
- **`atelet`.** A DaemonSet on every node, Substrate's kubelet for actors. It runs, checkpoints and restores sandboxes and moves snapshots.
- **`ateworker`.** The process inside each worker Pod. It drives `runsc` (gVisor) or Cloud Hypervisor (microVM) and embeds `atunnel` for the actor's network.
- **`atenet-router`.** Envoy at the front door. It wakes actors on demand and routes requests to them.
- **`atenet-egress`.** The only way out for actor traffic. It enforces egress policy and injects credentials.

## Architecture

```text
            your platform  (sets ate-target-actor: <atespace>/<actor>)
                  |
                  v
 ate-system  +---------------+ ResumeActor +----------------+
             | atenet-router |------------>| ate-api-server |--> PostgreSQL
             | (Envoy)       |             | (gRPC)         |    actor rows
             +---------------+             +----------------+
                  |                                |
     mTLS CONNECT |                                | Restore / Checkpoint
        port 8443 v                                v
 each node   +----------------------------+  +-----------+
             | worker Pod (ateworker)     |<-| atelet    |--> object storage
             |  atunnel -> [sandbox: a1]  |  | DaemonSet |    snapshots
             |             [sandbox: a7]  |  +-----------+
             +----------------------------+
                  |
                  v  every outbound request
             atenet-egress  (policy, credential injection) --> internet
```

The split follows how often things change:

- **Kubernetes handles what changes rarely.** Nodes, worker Pods and their rollout, node autoscaling, the WorkerPool and SandboxConfig objects, RBAC on those objects.
- **Substrate handles what changes all the time.** Which actor runs where, and its state. Thousands of transitions per second would overload etcd and the kube-apiserver's watch machinery, so they go to PostgreSQL behind a synchronous API.

## How resume works

A request for a suspended actor goes through these steps:

1. Your platform sends an HTTP request to `atenet-router` with the header `ate-target-actor: ate-demo-counter/my-counter-1`. The `Host` header plays no part in choosing the actor.
2. The router holds the request and calls `ResumeActor` on `ate-api-server`.
3. The control plane locks the actor and claims room on a worker in a pool the template's `workerSelector` matches. No Pod is created and kube-scheduler isn't involved.
4. `atelet` on that worker's node gets the snapshot. It uses its local copy if it still has one, otherwise it downloads it from the bucket. On an actor's first run, that's the template's golden snapshot, if it has one.
5. `ateworker` restores the snapshot into a new sandbox. If the container has a `wakeupProbe`, `ResumeActor` waits until that endpoint returns 200.
6. The actor is `RUNNING`. The router opens an mTLS `CONNECT` tunnel to `atunnel` on the worker, which forwards the original request to the actor's port.

What happens when every worker is full:

- The router **parks** the request and retries the resume with backoff for up to 5 seconds (`--parked-request-budget`), because another actor usually suspends within milliseconds.
- After that it returns `503 actor <id> unavailable: no worker has room for the actor`.
- The parking lot holds 1,024 requests by default (`--parked-request-max`). Requests beyond that get the `503` right away.

## How suspend works

1. Something calls `SuspendActor`, your platform or `kubectl ate suspend actor`.
2. `ateworker` freezes the sandbox and writes a checkpoint.
3. `atelet` keeps the snapshot on the node and uploads it to object storage.
4. The sandbox is wiped and its room on the worker goes back to the pool.
5. The actor is `SUSPENDED`. Its next resume may land on any worker.

Who decides that an actor is idle? In 1.0, not Substrate. There is no idle timeout to configure. The API has a path for a worker to propose a suspend (`RequestActorSuspend`), but nothing calls it yet. Plan on your platform calling `SuspendActor` after an idle period it picks, for example when a chat session has had no message for two minutes.

Two suspend modes:

| | Asynchronous (default) | Synchronous (`--mode=sync`) |
| --- | --- | --- |
| `SuspendActor` returns | when the checkpoint is on the node's disk | after the upload to the bucket finishes |
| Worker freed | right away | right away |
| If the node dies before the upload | the actor resumes from its previous snapshot, the latest state is lost | nothing is lost |
| Use it for | chat sessions, scratch work | anything you can't redo |

The node keeps its local copy for `--snapshot-detach-delay`, so an actor that wakes up again soon resumes on the same node without a download.

### What a snapshot keeps

The template's `snapshotConfig.preferredFidelity` sets how much state survives. Each level includes the one before it.

| Fidelity | Keeps | On resume | Cost |
| --- | --- | --- | --- |
| `SNAPSHOT_FIDELITY_VOLUMES` | `durableDir` volume contents | containers boot fresh from the image | smallest snapshot |
| `SNAPSHOT_FIDELITY_ROOTFS` | volumes plus files written to the container filesystem | containers boot fresh, with their files back | medium |
| `SNAPSHOT_FIDELITY_MEMORY` (default) | all of the above plus processes and RAM | processes continue where they stopped | largest, and fastest to a working agent |

A `MEMORY` snapshot is tied to the template it came from and to the sandbox runtime version that took it. If no worker with that runtime has room, Substrate restores the `ROOTFS` state and cold-boots.

### The actor lifecycle

```text
          CreateActor
               |
               v
  +------> SUSPENDED ---------- DeleteActor ----> deleted
  |            |                                  (snapshots garbage collected)
  |            | ResumeActor  (a request, or kubectl ate resume)
  |            v
  |        RESUMING
  |            | restore or boot done
  |            v
  |         RUNNING --------- sandbox dies ------> CRASHED
  |            |                                      |
  |            | SuspendActor                         | RevertActor
  |            v                                      v
  +------- SUSPENDING                        SUSPENDED at the last
         checkpoint written                  durable snapshot
```

`RevertActor` also works on a `RUNNING` actor. It throws away whatever happened since the last durable snapshot, which is a handy undo when an agent wrecks its own environment.

## Design principles

- **Separate the actor from the Pod.** A worker Pod outlives many actors, and an actor outlives many Pods. Kubernetes never sees an actor start or stop.
- **Keep Kubernetes off the hot path.** Pods take seconds to schedule, pull and start. That's fine for a service that runs for days and too slow for an agent that runs for two seconds. So the hot path is a database update plus a restore into a Pod that already exists.
- **Always sandboxed.** Every actor runs in gVisor or a Cloud Hypervisor microVM with its own network namespace. Plain `runc` containers aren't supported at all.
- **No credentials in the sandbox.** Actors get no API keys or cloud tokens. The egress gateway adds credentials to approved outbound requests, so a prompt-injected agent has nothing to steal.
- **Default-deny networking.** Traffic comes in only through the router from authenticated callers. It goes out only through the egress gateway, with DNS filtered against the policy. Actors have no direct path to each other, the Substrate API or the Kubernetes API.
- **An identity for every actor.** Each running actor gets a SPIFFE X.509 certificate valid for one hour, and the egress gateway can mint a JWT for it on demand. The UID tells apart two actors that had the same name at different times.
- **Stay low-opinion.** No SDK. Anything that serves HTTP from an OCI image works. End-user authentication, agent registries and agent lineage belong to the layer above.

## Kubernetes vs Agent Substrate

| | Kubernetes, one Pod per agent | Agent Substrate |
| --- | --- | --- |
| Unit of work | Pod | Actor |
| Idle agent | holds its memory and its requests on a node | a snapshot in a bucket, no worker used |
| Getting to running | seconds: schedule, pull image, start container | under 500 ms: restore into a running Pod (project figure) |
| Scale to zero | possible with KEDA or Knative, but the process and its memory are gone | the normal state, with memory kept |
| State store | etcd, watch-based, eventually consistent | PostgreSQL, synchronous workflows |
| Scale | 110 Pods per node, 150,000 per cluster | up to 1,000 actors per worker by default, a target of 1 billion per cluster |
| Scheduler | kube-scheduler places Pods on Nodes | `ate-api-server` places actors on warm Workers |
| Node agent | kubelet | `atelet`, a DaemonSet |
| Sizing | `requests` and `limits` | `limits` only, fixed in the template |
| Isolation | `runc` containers share the host kernel | gVisor or microVM, always |
| Network default | any Pod can reach any Pod | default-deny, one way in, one way out |
| Addressing | Service DNS name and ClusterIP | `ate-target-actor: <atespace>/<actor>` header |
| Credentials | Secrets mounted into the Pod | injected at the egress gateway, never in the sandbox |
| Workload identity | ServiceAccount token | SPIFFE certificate per actor, JWT on demand |
| API and CLI | REST with watches, `kubectl` | gRPC, `kubectl ate` plugin |
| Authorization | RBAC | OpenFGA per atespace, plus Kubernetes RBAC for the CRDs |

### Where it looks like Kubernetes

If you know Kubernetes, most of the surface feels familiar:

- **Templates and instances.** An ActorTemplate is to an Actor what a Pod template is to a Pod. `command` and `args` override `ENTRYPOINT` and `CMD` exactly the way they do in a Pod.
- **Labels and selectors.** A template's `workerSelector.matchLabels` picks WorkerPools by their labels, like a node selector.
- **Namespacing.** Atespaces scope names the way namespaces do, `-a` works like `-n`, and `-A` lists across all of them.
- **Controllers.** `atecontroller` reconciles WorkerPools into Deployments, with conditions and events you read with `kubectl events --for workerpool/NAME`.
- **Security contexts.** `securityContext.capabilities.add` and `drop` use Kubernetes naming, such as `NET_BIND_SERVICE` without the `CAP_` prefix.
- **The CLI.** `kubectl ate get actors`, `-o yaml`, `create -f` and `logs -f` behave as they do in kubectl.

### Where it doesn't

- **No `kubectl apply` for actors.** Templates and actors go through `kubectl ate`, so GitOps tools like Argo CD and Flux can manage the WorkerPools but not the templates or actors.
- **Templates are immutable.** To change an image or the size, create a new template and point the suspended actor at it with `kubectl ate update actor NAME --template NEW`. The next resume keeps `durableDir` data and cold-boots.
- **Images need a digest.** `name@sha256:...`, never a tag.
- **Env vars are literal.** No `valueFrom`, no `envFrom`, no `$(VAR)` expansion.
- **Identity comes from files, not env vars.** A `systemInfo` volume projects the actor's name, atespace and UID as files. An env var would be frozen in the snapshot and wrong for every actor restored from a shared golden snapshot.
- **Probes run once per wakeup.** A `wakeupProbe` gates each resume and then stops. There's no liveness or readiness probe that keeps running.

## Isn't this just scale to zero?

Not quite. KEDA's HTTP add-on and Knative can also start a workload when a request arrives. The difference is what comes back:

- **Scale to zero** starts a new Pod from the image. Memory, open files and in-flight tool state are gone, and the start takes as long as a Pod start.
- **Substrate** restores the same process, with its memory and files, into a Pod that is already running.

For a stateless HTTP function, scale to zero is simpler and good enough. For an agent holding a long conversation, a loaded repository or a half-finished tool call, keeping the memory is the whole point.

## Features in 1.0

- **Three snapshot fidelities** (`VOLUMES`, `ROOTFS`, `MEMORY`) and golden snapshots per template.
- **Tags.** Name a suspended actor's snapshot, publish it to other atespaces, and start new actors from it with `kubectl ate create actor NAME --template T --tag TAG`. That's a fork of an agent at a known good point.
- **Revert** to the last durable snapshot.
- **Request parking** while the pool is full.
- **Ports other than 80** through HTTP `CONNECT` to `<actor>:<port>`. HTTP and HTTPS only for now.
- **Egress policies** per actor, with credential injection and optional TLS interception through a trust bundle.
- **Two sandbox classes.** gVisor runs on any Linux node. The microVM class needs `/dev/kvm` and enforces per-container limits, which gVisor accepts and ignores.
- **Observability.** OpenTelemetry metrics, traces with `--trace`, and `kubectl ate logs actor NAME -f`.
- **Storage choices.** Bundled PostgreSQL or Cloud SQL for state, GCS or S3 for snapshots. External CSI volumes are in preview.

## Limits to know before you commit

- **Idle detection is your job.** See the suspend section. Substrate won't suspend an actor on its own in 1.0.
- **No GPUs inside actors.** Device passthrough was removed while its API is redesigned.
- **HTTP only through the router.** Raw TCP on other ports isn't reachable yet.
- **No built-in actor-to-actor authorization.** Put an authenticating proxy in between, or handle it in your app.
- **Quotas and rate limits are still on the roadmap.** Nothing stops one atespace from using the whole pool.
- **Workers need trusted nodes.** Worker Pods run as root with `SYS_ADMIN` and `NET_ADMIN`. Put them on dedicated nodes.
- **Kubernetes versions.** The project supports the latest stable minor and the one before it.

## Try it on kind in ten minutes

You need Go, Docker and kubectl. On a host that will run more than about 1,000 sandboxes, raise the kernel neighbor table first (`net.ipv4.neigh.default.gc_thresh1/2/3` = 4096/16384/32768).

Install Substrate and the counter demo:

```bash
git clone https://github.com/agent-substrate/substrate && cd substrate
hack/create-kind-cluster.sh
export ATE_INSTALL_KIND=true

go run ./cmd/ate-setup deploy ate-system \
  --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client
go run ./cmd/ate-setup deploy demo counter
go install ./cmd/kubectl-ate
```

Create an actor and open a local proxy to it through the router:

```bash
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter
kubectl ate proxy actor my-counter-1 -a ate-demo-counter
```

In a second terminal, use it, suspend it, and use it again:

```bash
curl -X POST http://localhost:8000/
# preserved memory count: 1 | preserved file counter: 1 | ...

# STATE is ACTOR_STATE_RUNNING and WORKER POD names a worker
kubectl ate get actor my-counter-1 -a ate-demo-counter

kubectl ate suspend actor my-counter-1 -a ate-demo-counter

# STATE is ACTOR_STATE_SUSPENDED and WORKER POD is <none>
kubectl ate get actor my-counter-1 -a ate-demo-counter

curl -X POST http://localhost:8000/
# preserved memory count: 2 | preserved file counter: 2 | ...
```

The second `curl` woke the actor up, possibly on another worker Pod, and the in-memory counter carried on from 1. Run `kubectl ate get workers` to see each worker's actor count and its CPU and memory use. For a real cluster, the [GKE installation guide](https://docs.cloud.google.com/kubernetes-engine/ai-ml/install-overview-substrate) has a one-line installer.

### The two objects you write

The WorkerPool is a normal Kubernetes manifest:

```yaml
apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: counter
  namespace: ate-demo-counter
  labels:
    workload: counter            # what templates select on
spec:
  replicas: 3
  workerImage: ghcr.io/agent-substrate/substrate/ateworker-gvisor:v1.0.0
  sandboxClasses:
    - name: gvisor
  template:
    resources:
      limits:                    # capacity the worker's actors share
        cpu: "1"
        memory: 1Gi
      requests:                  # what kube-scheduler packs nodes by
        cpu: 250m
        memory: 1Gi
```

The ActorTemplate goes to the Substrate API with `kubectl ate create actor-template -f counter-template.yaml`, after `kubectl ate create atespace ate-demo-counter`:

```yaml
metadata:
  atespace: ate-demo-counter
  name: counter
workerSelector:
  matchLabels:
    workload: counter            # must match WorkerPool metadata.labels
containers:
  - name: counter
    image: registry.example.com/counter@sha256:<digest>
    command: ["/ko-app/counter"]
    wakeupProbe:
      httpGet:
        path: /readyz
        port: 80
    volumeMounts:
      - name: data
        mountPath: /home/counter
volumes:
  - name: data
    durableDir: {}
resources:
  limits:                        # sizes the sandbox; there is no requests field
    cpuMillicores: 1000
    memoryBytes: 536870912       # 512 MiB
snapshotConfig:
  preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
```

A worker takes an actor only while its remaining capacity covers the template's `limits`. Here only one 512 MiB actor fits on each 1 GiB worker, because the worker keeps part of its memory for itself and each actor adds some runtime overhead. For real density, give workers more memory and actors less.

## When to use it

Use Agent Substrate when:

- You run many agents per user, per session or per task, and most of them are idle at any moment.
- The agents run code you don't trust: coding agents, notebooks, generated scripts.
- Agents keep state you don't want to rebuild, such as a cloned repository, a loaded model context or a long conversation.
- You want MCP servers or tools sandboxed with their own identity and egress rules.

Stay with plain Deployments, or KEDA and Knative, when:

- The workload is busy most of the time. Suspending saves nothing.
- It's stateless. Scale to zero is simpler.
- It needs a GPU, or a non-HTTP protocol.
- You need GitOps for every instance, since actors aren't Kubernetes objects.

## Production checklist

- [ ] Create a GCS or S3 bucket for snapshots and set it at install.
- [ ] Choose async or sync suspend for each kind of agent, based on how much lost state you can tolerate.
- [ ] Write the idle policy in your platform. When does it call `SuspendActor`?
- [ ] Size WorkerPools from the templates' `limits`, then load test resume latency with your real image sizes.
- [ ] Put workers on dedicated nodes, and the control plane on its own with `--cordon-system-infra-components`.
- [ ] Decide between bundled PostgreSQL and Cloud SQL, and back it up. It holds every actor's state.
- [ ] Write an `EgressPolicy` per actor, and grant the credential provider only the namespaces whose Secrets it should read.
- [ ] Restrict `--ingress-allowed-spiffe-ids` on the router to the clients that may wake actors.
- [ ] Alert on the router's `ate.router.park.outcome` metric and on its `503`s. Both mean the pool is too small.

The [interactive Agent Substrate graph](/graphs/agent-substrate-actors-workers-snapshots) walks through the same request path one component at a time, with the WorkerPool manifest and the mistakes people make most often.
