apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
name: counter
namespace: ate-demo-counter
labels:
workload: counter
spec:
replicas: 3
workerImage: ghcr.io/agent-substrate/substrate/ateworker-gvisor:v1.0.0
sandboxClasses:
- name: gvisor # or microvm, which needs /dev/kvm on the node
template:
resources:
limits: # worker capacity shared by its actors
cpu: "1"
memory: 1Gi
requests: # what kube-scheduler packs nodes by
cpu: 250m
memory: 1GiKubernetes graph
Agent Substrate: actors, warm workers and snapshots on Kubernetes
How Agent Substrate runs many sandboxed agents on a few warm Pods: a request for a suspended actor reaches the router, the control plane picks a worker, atelet restores the snapshot, and suspend writes it back to object storage.
Agent Substrate splits an agent in two. The Actor is the agent's identity and state, kept as a row in PostgreSQL and a snapshot in object storage. The worker is a Pod that is already running and waits for an actor to restore into it. Kubernetes only ever sees the workers.
Click a resource to see its role. The walkthrough follows one request for a suspended actor, then the suspend that frees the worker again.
Interactive graph
Click a resource to see what it does, or follow the setup step by step.
How it works
All 14 resources and what they do
- Your platform agent app or gateway
- Whatever sits in front of Substrate: your API, kagent, an MCP gateway. It authenticates end users, picks the actor, and sets the
ate-target-actor: <atespace>/<actor>header. Substrate doesn't authenticate your end users. - Deployment atenet-routerapps/v1
- Envoy with an
ext_procprocessor. It readsate-target-actor, asks the control plane to resume that actor, then opens an mTLSCONNECTtunnel to the worker that got it. When every worker is busy it parks the request for up to 5 seconds (--parked-request-budget) instead of returning503. - Deployment ate-api-serverapps/v1
- The Substrate control plane, a gRPC API. It owns the actor lifecycle (create, resume, suspend, revert, delete), picks a worker with room, and runs each resume or suspend as a workflow. It authorizes every call per atespace with an embedded OpenFGA model.
kubectl atetalks to it. - StatefulSet postgresapps/v1
- The state store for everything that changes often: atespaces, actor templates, actors, workers and their assignments. These rows would be too many and change too fast for etcd. The install can use Cloud SQL instead of this bundled database.
- ActorTemplate ate-demo-counter/counter
- The blueprint: digest-pinned images,
command, literal env vars,durableDirvolumes, CPU and memorylimits, snapshot fidelity, and aworkerSelectorthat matches WorkerPool labels. It is immutable and lives in the Substrate database, not in Kubernetes. Create it withkubectl ate create actor-template -f template.yaml. Creating it boots the workload once and saves a golden snapshot that new actors start from. - Actor ate-demo-counter/my-counter-1
- One agent instance, addressed as
<atespace>/<name>. It startsSUSPENDEDand moves throughRESUMING,RUNNINGandSUSPENDING. Its record holds the worker it runs on, if any, and the snapshot it resumes from. Create it withkubectl ate create actor my-counter-1 -a ate-demo-counter --template counter. - SandboxConfig gvisor-defaultate.dev/v1
- Cluster-scoped list of sandbox runtime versions for one class, here gVisor's
runsc. The installer creates one per class and marks it the class default. Nodes download the binaries from it, so they are not baked into the worker image. - WorkerPool counterate.dev/v1
- Warm capacity.
replicasworker Pods of one sandbox class, each withlimitsthat set how much the actors placed on it may book together. Itsmetadata.labelsare what an ActorTemplate'sworkerSelectormatches, not the Pod template's labels. - Deployment counterapps/v1
- Created by
atecontrollerfrom the WorkerPool. Ordinary Kubernetes from here on: kube-scheduler places the worker Pods, and node autoscalers see them like any other Pods. - Pod counter-…v1
- A worker: the
ateworkerprocess withatunnelinside it, running and waiting. It hosts actors, each in its own gVisor or microVM sandbox with a private network namespace, up to its capacity and--max-actors(default 1000). Pod and actor lifetimes are separate: a worker outlives many actors, and an actor outlives many workers. - Worker one row per worker Pod
- The control plane's record of a worker Pod:
ACTIVE,UNAVAILABLEorDRAINING, its capacity, what is allocated, and oneActorAssignmentper hosted actor.kubectl ate get workerslists them;kubectl getnever will. - DaemonSet ateletapps/v1
- The node supervisor, Substrate's kubelet for actors. It pulls images, builds OCI bundles, tells
ateworkerto run, checkpoint or restore a sandbox, and moves snapshots to and from object storage through a plugin sidecar. It keeps the latest snapshot on the node for a while, so the next resume there skips the download. - Object storage gs:// or s3:// bucket
- Where snapshots live while actors are suspended: golden snapshots, each actor's last snapshot, and tags. Only the plugin sidecars of
ateletandate-api-servertouch it. Sandboxes never do. - Deployment atenet-egressapps/v1
- The only way out. Each actor's outbound traffic is tunneled here, checked against its
EgressPolicy, and given credentials on the way out, so API keys never enter the sandbox. If the policy check fails or isn't configured, the traffic is dropped.
Common mistakes
kubectl get actorsreturns nothing. Actors, Workers and ActorTemplates live in Substrate's PostgreSQL, not etcd. Usekubectl ate get …. Only WorkerPool and SandboxConfig are CRDs.- Suspend is asynchronous by default: it returns once the snapshot is on the node's disk, before the upload. If the node dies in between, the actor comes back from its previous snapshot. Use
kubectl ate suspend actor NAME --mode=syncwhen losing that state matters. - An ActorTemplate's
workerSelectormatches the WorkerPool's ownmetadata.labels. Labels underspec.template.labelsonly end up on the Pods, and actors stay unplaceable. - Worker capacity comes from the pool's
limits, while kube-scheduler packs nodes byrequests. An actor whose templatelimitsdon't fit in a worker's remaining capacity is never placed, however idle the node looks. - Templates are immutable. To change an image or the size, create a new template and run
kubectl ate update actor NAME --template NEWon the suspended actor. Its next resume keeps thedurableDirdata and cold-boots, so memory and root filesystem writes are lost. - Container images in a template must be pinned by digest (
name@sha256:…). Env vars take literal values only: novalueFrom, noenvFrom, no$(VAR)expansion. Keep credentials out of them anyway, and let the egress gateway inject them.
The manifests you write
Everything above that is marked "You write it", in one file. The rest comes from add-ons and controllers.
Keep going
More graphs
- Deployment, Service and Ingress: how a request reaches your Pods
- Gateway API: GatewayClass, Gateway and HTTPRoute with a canary split
- PVC, StorageClass and CSI: how a cloud disk ends up in your Pod
- RBAC: how a ServiceAccount gets its permissions
- cert-manager with Let's Encrypt: Certificate, CertificateRequest, Order and Challenge
- A private CA with cert-manager and trust-manager
- Istio traffic management: Gateway, VirtualService, DestinationRule and policies
- Flux dependencies: Git and OCI sources, Kustomizations and HelmReleases
- Argo CD: ApplicationSet, AppProject and the Applications it generates
- Inside an OCI image: tag, index, manifests, config and layers
- OCI artifacts: Helm charts, Flux artifacts and image volumes
- A Helm release: chart, values, revisions and the objects Helm owns
- Kustomize: a base, two overlays, a component and generated ConfigMaps
- A StatefulSet database: headless Service, per-Pod volumes, snapshots and restore
- Autoscaling end to end: KEDA, the HPA it creates and Karpenter adding nodes
- Prometheus Operator: ServiceMonitor, PrometheusRule and the path to an alert
- External Secrets Operator: from a cloud secret manager to a Pod, with rotation
- Kyverno CEL policies: ValidatingPolicy, MutatingPolicy, GeneratingPolicy and exceptions
- Cilium egress to a domain: CiliumNetworkPolicy, the DNS proxy and toFQDNs
- Istio ambient mode: ztunnel, waypoint proxies and authorization without sidecars
- Argo Rollouts canary: traffic weights on an HTTPRoute and analysis from Prometheus