# Agent Substrate: actors, warm workers and snapshots on Kubernetes

> How Agent Substrate runs many sandboxed agents on a few warm Pods: a request for a suspended actor reaches the router, the control plane picks a worker, atelet restores the snapshot, and suspend writes it back to object storage.

Interactive graph: https://k8wiz.com/graphs/agent-substrate-actors-workers-snapshots

Agent Substrate splits an agent in two. The `Actor` is the agent's identity and state, kept as a row in PostgreSQL and a snapshot in object storage. The worker is a Pod that is already running and waits for an actor to restore into it. Kubernetes only ever sees the workers.

Click a resource to see its role. The walkthrough follows one request for a suspended actor, then the suspend that frees the worker again.

## Resources

### Your platform agent app or gateway

outside the cluster

Whatever sits in front of Substrate: your API, kagent, an MCP gateway. It authenticates end users, picks the actor, and sets the `ate-target-actor: <atespace>/<actor>` header. Substrate doesn't authenticate your end users.

### Deployment atenet-router

comes with an add-on, apps/v1, namespace ate-system

Envoy with an `ext_proc` processor. It reads `ate-target-actor`, asks the control plane to resume that actor, then opens an mTLS `CONNECT` tunnel to the worker that got it. When every worker is busy it parks the request for up to 5 seconds (`--parked-request-budget`) instead of returning `503`.

### Deployment ate-api-server

comes with an add-on, apps/v1, namespace ate-system

The Substrate control plane, a gRPC API. It owns the actor lifecycle (create, resume, suspend, revert, delete), picks a worker with room, and runs each resume or suspend as a workflow. It authorizes every call per atespace with an embedded OpenFGA model. `kubectl ate` talks to it.

### StatefulSet postgres

comes with an add-on, apps/v1, namespace ate-system

The state store for everything that changes often: atespaces, actor templates, actors, workers and their assignments. These rows would be too many and change too fast for etcd. The install can use Cloud SQL instead of this bundled database.

### ActorTemplate ate-demo-counter/counter

you write it

The blueprint: digest-pinned images, `command`, literal env vars, `durableDir` volumes, CPU and memory `limits`, snapshot fidelity, and a `workerSelector` that matches WorkerPool labels. It is immutable and lives in the Substrate database, not in Kubernetes. Create it with `kubectl ate create actor-template -f template.yaml`. Creating it boots the workload once and saves a golden snapshot that new actors start from.

### Actor ate-demo-counter/my-counter-1

you write it

One agent instance, addressed as `<atespace>/<name>`. It starts `SUSPENDED` and moves through `RESUMING`, `RUNNING` and `SUSPENDING`. Its record holds the worker it runs on, if any, and the snapshot it resumes from. Create it with `kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter`.

### SandboxConfig gvisor-default

comes with an add-on, ate.dev/v1

Cluster-scoped list of sandbox runtime versions for one class, here gVisor's `runsc`. The installer creates one per class and marks it the class default. Nodes download the binaries from it, so they are not baked into the worker image.

### WorkerPool counter

you write it, ate.dev/v1, namespace ate-demo-counter

Warm capacity. `replicas` worker Pods of one sandbox class, each with `limits` that set how much the actors placed on it may book together. Its `metadata.labels` are what an ActorTemplate's `workerSelector` matches, not the Pod template's labels.

### Deployment counter

created by a controller, apps/v1, namespace ate-demo-counter

Created by `atecontroller` from the WorkerPool. Ordinary Kubernetes from here on: kube-scheduler places the worker Pods, and node autoscalers see them like any other Pods.

### Pod ×3 counter-…

created by a controller, v1, namespace ate-demo-counter

A worker: the `ateworker` process with `atunnel` inside it, running and waiting. It hosts actors, each in its own gVisor or microVM sandbox with a private network namespace, up to its capacity and `--max-actors` (default 1000). Pod and actor lifetimes are separate: a worker outlives many actors, and an actor outlives many workers.

### Worker one row per worker Pod

created by a controller

The control plane's record of a worker Pod: `ACTIVE`, `UNAVAILABLE` or `DRAINING`, its capacity, what is allocated, and one `ActorAssignment` per hosted actor. `kubectl ate get workers` lists them; `kubectl get` never will.

### DaemonSet atelet

comes with an add-on, apps/v1, namespace ate-system

The node supervisor, Substrate's kubelet for actors. It pulls images, builds OCI bundles, tells `ateworker` to run, checkpoint or restore a sandbox, and moves snapshots to and from object storage through a plugin sidecar. It keeps the latest snapshot on the node for a while, so the next resume there skips the download.

### Object storage gs:// or s3:// bucket

outside the cluster

Where snapshots live while actors are suspended: golden snapshots, each actor's last snapshot, and tags. Only the plugin sidecars of `atelet` and `ate-api-server` touch it. Sandboxes never do.

### Deployment atenet-egress

comes with an add-on, apps/v1, namespace ate-system

The only way out. Each actor's outbound traffic is tunneled here, checked against its `EgressPolicy`, and given credentials on the way out, so API keys never enter the sandbox. If the policy check fails or isn't configured, the traffic is dropped.

## How the resources connect

- Your platform agent app or gateway → Deployment atenet-router: ate-target-actor header (traffic)
- Deployment atenet-router → Deployment ate-api-server: ResumeActor (traffic)
- Deployment ate-api-server → StatefulSet postgres: actor, worker rows (references)
- Actor ate-demo-counter/my-counter-1 → ActorTemplate ate-demo-counter/counter: actorTemplate (references)
- ActorTemplate ate-demo-counter/counter → WorkerPool counter: workerSelector (selects)
- WorkerPool counter → SandboxConfig gvisor-default: sandboxClasses (references)
- WorkerPool counter → Deployment counter: atecontroller creates (creates)
- Deployment counter → Pod ×3 counter-…: creates (creates)
- Worker one row per worker Pod → Pod ×3 counter-…: maps to (references)
- Worker one row per worker Pod → Actor ate-demo-counter/my-counter-1: ActorAssignment (references)
- Deployment ate-api-server → DaemonSet atelet: Restore / Checkpoint (traffic)
- DaemonSet atelet → Object storage gs:// or s3:// bucket: download / upload snapshot (traffic)
- DaemonSet atelet → Pod ×3 counter-…: RestoreWorkload (traffic)
- Deployment atenet-router → Pod ×3 counter-…: mTLS CONNECT :8443 (traffic)
- Pod ×3 counter-… → Deployment atenet-egress: all outbound traffic (traffic)

## How it works

1. **Set up capacity once.** You apply a WorkerPool. `atecontroller` turns it into a Deployment, and the worker Pods start and register as Workers. This is the slow, rare part, and Kubernetes handles it like any rollout.
2. **Define the agent, create an instance.** `kubectl ate create actor-template` stores the template and takes a golden snapshot. `kubectl ate create actor` adds a row in `SUSPENDED` state. Nothing runs yet, and an actor that never gets a request never uses a worker.
3. **A request arrives for a suspended actor.** Your platform sends the request with `ate-target-actor: ate-demo-counter/my-counter-1`. The router holds it and calls `ResumeActor`.
4. **The control plane picks a warm worker.** `ate-api-server` locks the actor, finds a worker in a matching pool with room for the template's `limits`, and records the assignment. No Pod is created and kube-scheduler isn't involved.
5. **atelet restores the snapshot.** `atelet` on that node fetches the snapshot, from its local copy if it still has one, or else from object storage. Then `ateworker` restores it into a fresh sandbox. With `MEMORY` fidelity, the processes continue where they stopped. The project's goal for this whole path is under 500 ms.
6. **The request reaches the actor.** The router opens an mTLS `CONNECT` tunnel to `atunnel` on the worker, which forwards the request to the actor's port. Anything the actor sends out goes through `atenet-egress`.
7. **Suspend frees the worker.** Something calls `SuspendActor`: your platform, or `kubectl ate suspend actor`. `ateworker` checkpoints the sandbox, `atelet` keeps a copy on the node and uploads it to the bucket, and the sandbox is wiped, which gives its room on the worker back to the pool. The next request may resume the actor on a different worker.

## Common mistakes

- `kubectl get actors` returns nothing. Actors, Workers and ActorTemplates live in Substrate's PostgreSQL, not etcd. Use `kubectl ate get …`. Only WorkerPool and SandboxConfig are CRDs.
- Suspend is asynchronous by default: it returns once the snapshot is on the node's disk, before the upload. If the node dies in between, the actor comes back from its previous snapshot. Use `kubectl ate suspend actor NAME --mode=sync` when losing that state matters.
- An ActorTemplate's `workerSelector` matches the WorkerPool's own `metadata.labels`. Labels under `spec.template.labels` only end up on the Pods, and actors stay unplaceable.
- Worker capacity comes from the pool's `limits`, while kube-scheduler packs nodes by `requests`. An actor whose template `limits` don't fit in a worker's remaining capacity is never placed, however idle the node looks.
- Templates are immutable. To change an image or the size, create a new template and run `kubectl ate update actor NAME --template NEW` on the suspended actor. Its next resume keeps the `durableDir` data and cold-boots, so memory and root filesystem writes are lost.
- Container images in a template must be pinned by digest (`name@sha256:…`). Env vars take literal values only: no `valueFrom`, no `envFrom`, no `$(VAR)` expansion. Keep credentials out of them anyway, and let the egress gateway inject them.

## The manifests you write

```yaml
apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: counter
  namespace: ate-demo-counter
  labels:
    workload: counter
spec:
  replicas: 3
  workerImage: ghcr.io/agent-substrate/substrate/ateworker-gvisor:v1.0.0
  sandboxClasses:
    - name: gvisor      # or microvm, which needs /dev/kvm on the node
  template:
    resources:
      limits:           # worker capacity shared by its actors
        cpu: "1"
        memory: 1Gi
      requests:         # what kube-scheduler packs nodes by
        cpu: 250m
        memory: 1Gi
```
