k8wiz
Back to Articles

Ready to test your Kubernetes knowledge? Try our Kubernetes quiz!

Agent Substrate explained: how it suspends and resumes AI agents on Kubernetes

What Agent Substrate is, how its actors, workers and snapshots fit together, how suspend and resume work, and where it differs from running one Pod per agent.

20 min read
By k8wiz Team
  • agent-substrate
  • ai-agents
  • kubernetes
  • sandboxing
  • gvisor
  • microvm
  • suspend-resume
  • snapshots

An AI agent spends most of its life waiting. It waits for the model to answer, for a tool to return, for a human to type the next message. On plain Kubernetes each agent is usually a Pod, and that Pod keeps its memory and its slot on a node the whole time, busy or not. With ten thousand mostly idle agents, you pay for ten thousand Pods that are doing nothing.

Agent Substrate takes a different approach. It snapshots idle agents out of memory into object storage and restores them, in under half a second by the project's numbers, onto a small pool of Pods that are already running. It runs on standard Kubernetes and reached 1.0 in November 2026. This article covers the concepts, the suspend and resume path, how it compares with plain Kubernetes, and how to try it on a laptop.

If you'd rather click through the moving parts, the interactive graph of Agent Substrate draws every component with a step-by-step walkthrough of one request.

The problem in one picture

Plain Kubernetes: one Pod per agent, all of them resident all the time

  agent-1   Pod  [##........................]   busy 8% of the time
  agent-2   Pod  [.....#..............#.....]
  agent-3   Pod  [...........##.............]
  ...
  agent-N   Pod  [.................#........]   N Pods = N x memory

Agent Substrate: N actors in a bucket, a few warm workers

  object storage   [a1] [a2] [a3] [a4] ... [aN]   suspended: no CPU, no RAM
                     |   resume      ^ suspend
                     v               |
  WorkerPool       [worker-1] [worker-2] [worker-3]   only busy actors here

Three things make agents a bad fit for one Pod each:

  • They are bursty. Work comes in short spikes, and the gaps between them can last minutes or days.
  • They run untrusted code. An agent that writes and runs code needs a real sandbox, so you can't pack several users into one process.
  • There are a lot of them. One agent per user, per session or per task adds up fast. Kubernetes documents limits of 110 Pods per node and 150,000 Pods per cluster.

What Agent Substrate is

  • An open source runtime (Apache 2.0) for sandboxed, stateful, mostly idle workloads, installed into a Kubernetes cluster in the ate-system namespace. ate is the prefix you'll see on every binary, header and CLI.
  • A low-opinion runtime. It runs any OCI image, so it works with ADK, LangChain, MCP servers and coding agents such as Claude Code without code changes. It isn't an agent framework.
  • Built around one idea: map many actors (agent instances) onto few workers (warm Pods), and move actors in and out with snapshots.
  • Already used underneath other projects. kagent, a CNCF Sandbox project, uses it to run its agents, and so does Google's Agent Executor.

The project's published numbers:

MetricTodayLong-term target
Resume latencyunder 500 ms100 ms at p95
Wakeups per second, one clusterover 5001,000
Actors per cluster, active and idlemillions1 billion
Actors per worker Pod (oversubscription)10x to 30x

The demo video multiplexes about 250 stateful actors onto 8 worker Pods. Treat all of these as the project's own figures. Measure with your own images, because a 4 GiB memory snapshot restores slower than a 100 MiB one.

Core concepts

ConceptWhat it isStored inClosest Kubernetes idea
ActorOne agent instance, addressed as <atespace>/<name>. The unit that is suspended and resumed.PostgreSQLA Pod you could freeze and move to another node
ActorTemplateImmutable blueprint: images, volumes, CPU and memory limits, snapshot settingsPostgreSQLPod template
WorkerPoolWarm capacity: N worker Pods of one sandbox classetcd (CRD)A Deployment of placeholder Pods
WorkerThe control plane's record of one worker Pod, its capacity and the actors on itPostgreSQLA Node, as an actor sees it
AtespaceName scope and authorization boundary for actors, templates and tagsPostgreSQLNamespace
SandboxConfigThe runtime versions for one sandbox class, gVisor or microVMetcd (CRD)RuntimeClass
SnapshotMemory, root filesystem and volumes of a suspended actorObject storage (GCS or S3)Nothing. A VolumeSnapshot covers disks only.
Golden snapshotA snapshot taken once when a template is created, so new actors skip the bootObject storageA pre-warmed image
TagA named, immutable copy of a snapshot that other atespaces can useObject storageAn image tag, but for running state

The table's third column is the part that trips people up. Only WorkerPool, SandboxConfig and CSIDriverConfig are Kubernetes objects. Actors, templates and workers are rows in Substrate's own PostgreSQL database, so kubectl get actors returns nothing. You use the kubectl ate plugin instead.

The components:

  • ate-api-server. The control plane, a gRPC API. It owns actor state, picks workers and runs resume and suspend as workflows.
  • atecontroller. A normal Kubernetes controller that turns each WorkerPool into a Deployment.
  • atelet. A DaemonSet on every node, Substrate's kubelet for actors. It runs, checkpoints and restores sandboxes and moves snapshots.
  • ateworker. The process inside each worker Pod. It drives runsc (gVisor) or Cloud Hypervisor (microVM) and embeds atunnel for the actor's network.
  • atenet-router. Envoy at the front door. It wakes actors on demand and routes requests to them.
  • atenet-egress. The only way out for actor traffic. It enforces egress policy and injects credentials.

Architecture

            your platform  (sets ate-target-actor: <atespace>/<actor>)
                  |
                  v
 ate-system  +---------------+ ResumeActor +----------------+
             | atenet-router |------------>| ate-api-server |--> PostgreSQL
             | (Envoy)       |             | (gRPC)         |    actor rows
             +---------------+             +----------------+
                  |                                |
     mTLS CONNECT |                                | Restore / Checkpoint
        port 8443 v                                v
 each node   +----------------------------+  +-----------+
             | worker Pod (ateworker)     |<-| atelet    |--> object storage
             |  atunnel -> [sandbox: a1]  |  | DaemonSet |    snapshots
             |             [sandbox: a7]  |  +-----------+
             +----------------------------+
                  |
                  v  every outbound request
             atenet-egress  (policy, credential injection) --> internet

The split follows how often things change:

  • Kubernetes handles what changes rarely. Nodes, worker Pods and their rollout, node autoscaling, the WorkerPool and SandboxConfig objects, RBAC on those objects.
  • Substrate handles what changes all the time. Which actor runs where, and its state. Thousands of transitions per second would overload etcd and the kube-apiserver's watch machinery, so they go to PostgreSQL behind a synchronous API.

How resume works

A request for a suspended actor goes through these steps:

  1. Your platform sends an HTTP request to atenet-router with the header ate-target-actor: ate-demo-counter/my-counter-1. The Host header plays no part in choosing the actor.
  2. The router holds the request and calls ResumeActor on ate-api-server.
  3. The control plane locks the actor and claims room on a worker in a pool the template's workerSelector matches. No Pod is created and kube-scheduler isn't involved.
  4. atelet on that worker's node gets the snapshot. It uses its local copy if it still has one, otherwise it downloads it from the bucket. On an actor's first run, that's the template's golden snapshot, if it has one.
  5. ateworker restores the snapshot into a new sandbox. If the container has a wakeupProbe, ResumeActor waits until that endpoint returns 200.
  6. The actor is RUNNING. The router opens an mTLS CONNECT tunnel to atunnel on the worker, which forwards the original request to the actor's port.

What happens when every worker is full:

  • The router parks the request and retries the resume with backoff for up to 5 seconds (--parked-request-budget), because another actor usually suspends within milliseconds.
  • After that it returns 503 actor <id> unavailable: no worker has room for the actor.
  • The parking lot holds 1,024 requests by default (--parked-request-max). Requests beyond that get the 503 right away.

How suspend works

  1. Something calls SuspendActor, your platform or kubectl ate suspend actor.
  2. ateworker freezes the sandbox and writes a checkpoint.
  3. atelet keeps the snapshot on the node and uploads it to object storage.
  4. The sandbox is wiped and its room on the worker goes back to the pool.
  5. The actor is SUSPENDED. Its next resume may land on any worker.

Who decides that an actor is idle? In 1.0, not Substrate. There is no idle timeout to configure. The API has a path for a worker to propose a suspend (RequestActorSuspend), but nothing calls it yet. Plan on your platform calling SuspendActor after an idle period it picks, for example when a chat session has had no message for two minutes.

Two suspend modes:

Asynchronous (default)Synchronous (--mode=sync)
SuspendActor returnswhen the checkpoint is on the node's diskafter the upload to the bucket finishes
Worker freedright awayright away
If the node dies before the uploadthe actor resumes from its previous snapshot, the latest state is lostnothing is lost
Use it forchat sessions, scratch workanything you can't redo

The node keeps its local copy for --snapshot-detach-delay, so an actor that wakes up again soon resumes on the same node without a download.

What a snapshot keeps

The template's snapshotConfig.preferredFidelity sets how much state survives. Each level includes the one before it.

FidelityKeepsOn resumeCost
SNAPSHOT_FIDELITY_VOLUMESdurableDir volume contentscontainers boot fresh from the imagesmallest snapshot
SNAPSHOT_FIDELITY_ROOTFSvolumes plus files written to the container filesystemcontainers boot fresh, with their files backmedium
SNAPSHOT_FIDELITY_MEMORY (default)all of the above plus processes and RAMprocesses continue where they stoppedlargest, and fastest to a working agent

A MEMORY snapshot is tied to the template it came from and to the sandbox runtime version that took it. If no worker with that runtime has room, Substrate restores the ROOTFS state and cold-boots.

The actor lifecycle

          CreateActor
               |
               v
  +------> SUSPENDED ---------- DeleteActor ----> deleted
  |            |                                  (snapshots garbage collected)
  |            | ResumeActor  (a request, or kubectl ate resume)
  |            v
  |        RESUMING
  |            | restore or boot done
  |            v
  |         RUNNING --------- sandbox dies ------> CRASHED
  |            |                                      |
  |            | SuspendActor                         | RevertActor
  |            v                                      v
  +------- SUSPENDING                        SUSPENDED at the last
         checkpoint written                  durable snapshot

RevertActor also works on a RUNNING actor. It throws away whatever happened since the last durable snapshot, which is a handy undo when an agent wrecks its own environment.

Design principles

  • Separate the actor from the Pod. A worker Pod outlives many actors, and an actor outlives many Pods. Kubernetes never sees an actor start or stop.
  • Keep Kubernetes off the hot path. Pods take seconds to schedule, pull and start. That's fine for a service that runs for days and too slow for an agent that runs for two seconds. So the hot path is a database update plus a restore into a Pod that already exists.
  • Always sandboxed. Every actor runs in gVisor or a Cloud Hypervisor microVM with its own network namespace. Plain runc containers aren't supported at all.
  • No credentials in the sandbox. Actors get no API keys or cloud tokens. The egress gateway adds credentials to approved outbound requests, so a prompt-injected agent has nothing to steal.
  • Default-deny networking. Traffic comes in only through the router from authenticated callers. It goes out only through the egress gateway, with DNS filtered against the policy. Actors have no direct path to each other, the Substrate API or the Kubernetes API.
  • An identity for every actor. Each running actor gets a SPIFFE X.509 certificate valid for one hour, and the egress gateway can mint a JWT for it on demand. The UID tells apart two actors that had the same name at different times.
  • Stay low-opinion. No SDK. Anything that serves HTTP from an OCI image works. End-user authentication, agent registries and agent lineage belong to the layer above.

Kubernetes vs Agent Substrate

Kubernetes, one Pod per agentAgent Substrate
Unit of workPodActor
Idle agentholds its memory and its requests on a nodea snapshot in a bucket, no worker used
Getting to runningseconds: schedule, pull image, start containerunder 500 ms: restore into a running Pod (project figure)
Scale to zeropossible with KEDA or Knative, but the process and its memory are gonethe normal state, with memory kept
State storeetcd, watch-based, eventually consistentPostgreSQL, synchronous workflows
Scale110 Pods per node, 150,000 per clusterup to 1,000 actors per worker by default, a target of 1 billion per cluster
Schedulerkube-scheduler places Pods on Nodesate-api-server places actors on warm Workers
Node agentkubeletatelet, a DaemonSet
Sizingrequests and limitslimits only, fixed in the template
Isolationrunc containers share the host kernelgVisor or microVM, always
Network defaultany Pod can reach any Poddefault-deny, one way in, one way out
AddressingService DNS name and ClusterIPate-target-actor: <atespace>/<actor> header
CredentialsSecrets mounted into the Podinjected at the egress gateway, never in the sandbox
Workload identityServiceAccount tokenSPIFFE certificate per actor, JWT on demand
API and CLIREST with watches, kubectlgRPC, kubectl ate plugin
AuthorizationRBACOpenFGA per atespace, plus Kubernetes RBAC for the CRDs

Where it looks like Kubernetes

If you know Kubernetes, most of the surface feels familiar:

  • Templates and instances. An ActorTemplate is to an Actor what a Pod template is to a Pod. command and args override ENTRYPOINT and CMD exactly the way they do in a Pod.
  • Labels and selectors. A template's workerSelector.matchLabels picks WorkerPools by their labels, like a node selector.
  • Namespacing. Atespaces scope names the way namespaces do, -a works like -n, and -A lists across all of them.
  • Controllers. atecontroller reconciles WorkerPools into Deployments, with conditions and events you read with kubectl events --for workerpool/NAME.
  • Security contexts. securityContext.capabilities.add and drop use Kubernetes naming, such as NET_BIND_SERVICE without the CAP_ prefix.
  • The CLI. kubectl ate get actors, -o yaml, create -f and logs -f behave as they do in kubectl.

Where it doesn't

  • No kubectl apply for actors. Templates and actors go through kubectl ate, so GitOps tools like Argo CD and Flux can manage the WorkerPools but not the templates or actors.
  • Templates are immutable. To change an image or the size, create a new template and point the suspended actor at it with kubectl ate update actor NAME --template NEW. The next resume keeps durableDir data and cold-boots.
  • Images need a digest. name@sha256:..., never a tag.
  • Env vars are literal. No valueFrom, no envFrom, no $(VAR) expansion.
  • Identity comes from files, not env vars. A systemInfo volume projects the actor's name, atespace and UID as files. An env var would be frozen in the snapshot and wrong for every actor restored from a shared golden snapshot.
  • Probes run once per wakeup. A wakeupProbe gates each resume and then stops. There's no liveness or readiness probe that keeps running.

Isn't this just scale to zero?

Not quite. KEDA's HTTP add-on and Knative can also start a workload when a request arrives. The difference is what comes back:

  • Scale to zero starts a new Pod from the image. Memory, open files and in-flight tool state are gone, and the start takes as long as a Pod start.
  • Substrate restores the same process, with its memory and files, into a Pod that is already running.

For a stateless HTTP function, scale to zero is simpler and good enough. For an agent holding a long conversation, a loaded repository or a half-finished tool call, keeping the memory is the whole point.

Features in 1.0

  • Three snapshot fidelities (VOLUMES, ROOTFS, MEMORY) and golden snapshots per template.
  • Tags. Name a suspended actor's snapshot, publish it to other atespaces, and start new actors from it with kubectl ate create actor NAME --template T --tag TAG. That's a fork of an agent at a known good point.
  • Revert to the last durable snapshot.
  • Request parking while the pool is full.
  • Ports other than 80 through HTTP CONNECT to <actor>:<port>. HTTP and HTTPS only for now.
  • Egress policies per actor, with credential injection and optional TLS interception through a trust bundle.
  • Two sandbox classes. gVisor runs on any Linux node. The microVM class needs /dev/kvm and enforces per-container limits, which gVisor accepts and ignores.
  • Observability. OpenTelemetry metrics, traces with --trace, and kubectl ate logs actor NAME -f.
  • Storage choices. Bundled PostgreSQL or Cloud SQL for state, GCS or S3 for snapshots. External CSI volumes are in preview.

Limits to know before you commit

  • Idle detection is your job. See the suspend section. Substrate won't suspend an actor on its own in 1.0.
  • No GPUs inside actors. Device passthrough was removed while its API is redesigned.
  • HTTP only through the router. Raw TCP on other ports isn't reachable yet.
  • No built-in actor-to-actor authorization. Put an authenticating proxy in between, or handle it in your app.
  • Quotas and rate limits are still on the roadmap. Nothing stops one atespace from using the whole pool.
  • Workers need trusted nodes. Worker Pods run as root with SYS_ADMIN and NET_ADMIN. Put them on dedicated nodes.
  • Kubernetes versions. The project supports the latest stable minor and the one before it.

Try it on kind in ten minutes

You need Go, Docker and kubectl. On a host that will run more than about 1,000 sandboxes, raise the kernel neighbor table first (net.ipv4.neigh.default.gc_thresh1/2/3 = 4096/16384/32768).

Install Substrate and the counter demo:

git clone https://github.com/agent-substrate/substrate && cd substrate
hack/create-kind-cluster.sh
export ATE_INSTALL_KIND=true

go run ./cmd/ate-setup deploy ate-system \
  --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client
go run ./cmd/ate-setup deploy demo counter
go install ./cmd/kubectl-ate

Create an actor and open a local proxy to it through the router:

kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter
kubectl ate proxy actor my-counter-1 -a ate-demo-counter

In a second terminal, use it, suspend it, and use it again:

curl -X POST http://localhost:8000/
# preserved memory count: 1 | preserved file counter: 1 | ...

# STATE is ACTOR_STATE_RUNNING and WORKER POD names a worker
kubectl ate get actor my-counter-1 -a ate-demo-counter

kubectl ate suspend actor my-counter-1 -a ate-demo-counter

# STATE is ACTOR_STATE_SUSPENDED and WORKER POD is <none>
kubectl ate get actor my-counter-1 -a ate-demo-counter

curl -X POST http://localhost:8000/
# preserved memory count: 2 | preserved file counter: 2 | ...

The second curl woke the actor up, possibly on another worker Pod, and the in-memory counter carried on from 1. Run kubectl ate get workers to see each worker's actor count and its CPU and memory use. For a real cluster, the GKE installation guide has a one-line installer.

The two objects you write

The WorkerPool is a normal Kubernetes manifest:

apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: counter
  namespace: ate-demo-counter
  labels:
    workload: counter            # what templates select on
spec:
  replicas: 3
  workerImage: ghcr.io/agent-substrate/substrate/ateworker-gvisor:v1.0.0
  sandboxClasses:
    - name: gvisor
  template:
    resources:
      limits:                    # capacity the worker's actors share
        cpu: "1"
        memory: 1Gi
      requests:                  # what kube-scheduler packs nodes by
        cpu: 250m
        memory: 1Gi

The ActorTemplate goes to the Substrate API with kubectl ate create actor-template -f counter-template.yaml, after kubectl ate create atespace ate-demo-counter:

metadata:
  atespace: ate-demo-counter
  name: counter
workerSelector:
  matchLabels:
    workload: counter            # must match WorkerPool metadata.labels
containers:
  - name: counter
    image: registry.example.com/counter@sha256:<digest>
    command: ["/ko-app/counter"]
    wakeupProbe:
      httpGet:
        path: /readyz
        port: 80
    volumeMounts:
      - name: data
        mountPath: /home/counter
volumes:
  - name: data
    durableDir: {}
resources:
  limits:                        # sizes the sandbox; there is no requests field
    cpuMillicores: 1000
    memoryBytes: 536870912       # 512 MiB
snapshotConfig:
  preferredFidelity: SNAPSHOT_FIDELITY_MEMORY

A worker takes an actor only while its remaining capacity covers the template's limits. Here only one 512 MiB actor fits on each 1 GiB worker, because the worker keeps part of its memory for itself and each actor adds some runtime overhead. For real density, give workers more memory and actors less.

When to use it

Use Agent Substrate when:

  • You run many agents per user, per session or per task, and most of them are idle at any moment.
  • The agents run code you don't trust: coding agents, notebooks, generated scripts.
  • Agents keep state you don't want to rebuild, such as a cloned repository, a loaded model context or a long conversation.
  • You want MCP servers or tools sandboxed with their own identity and egress rules.

Stay with plain Deployments, or KEDA and Knative, when:

  • The workload is busy most of the time. Suspending saves nothing.
  • It's stateless. Scale to zero is simpler.
  • It needs a GPU, or a non-HTTP protocol.
  • You need GitOps for every instance, since actors aren't Kubernetes objects.

Production checklist

  • Create a GCS or S3 bucket for snapshots and set it at install.
  • Choose async or sync suspend for each kind of agent, based on how much lost state you can tolerate.
  • Write the idle policy in your platform. When does it call SuspendActor?
  • Size WorkerPools from the templates' limits, then load test resume latency with your real image sizes.
  • Put workers on dedicated nodes, and the control plane on its own with --cordon-system-infra-components.
  • Decide between bundled PostgreSQL and Cloud SQL, and back it up. It holds every actor's state.
  • Write an EgressPolicy per actor, and grant the credential provider only the namespaces whose Secrets it should read.
  • Restrict --ingress-allowed-spiffe-ids on the router to the clients that may wake actors.
  • Alert on the router's ate.router.park.outcome metric and on its 503s. Both mean the pool is too small.

The interactive Agent Substrate graph walks through the same request path one component at a time, with the WorkerPool manifest and the mistakes people make most often.

Test yourself

Ready to test your Kubernetes knowledge? Try our Kubernetes quiz!