AWS Builder Center

Builder Center

Agent Sandboxes on AWS: Optimizing Sandbox Checkout Latency with Warm Pools on EKS Fargate

Agent Sandboxes on AWS: Optimizing Sandbox Checkout Latency with Warm Pools on EKS Fargate

Combining EKS Fargate's serverless isolation with the kubernetes-sigs/agent-sandbox operator for fast, operationally simple AI agent compute

Series: Agent Sandboxes on AWS (5 articles)

  1. 1
    Agent Sandboxes on AWS: AgentCore, Lambda MicroVMs, and EKS
  2. …
  3. 3
    Agent Sandboxes on AWS: Optimizing Sandbox Checkout Latency with Warm Pools on EKS Fargate This article
  4. 4
    Agent Sandboxes on AWS: Implementing Programmatic Tool Calling with Firecracker on Lambda MicroVMs

Anuj Dewangan

Principal Solutions Architect, Amazon Web Services

Contents

  1. Introduction
  2. Why Fargate for Agent Sandboxes
  3. Warm Pools with kubernetes-sigs/agent-sandbox
  4. Architecture and API Flow
  5. Performance
  6. Cost
  7. Tradeoffs
  8. Conclusion

Introduction

In our previous post on secure agent sandboxes , we explored three approaches to running untrusted AI-generated code on Amazon EKS: Fargate, gVisor, and Kata Containers. Each provides strong isolation for the Programmatic Tool Calling (PTC) pattern — where an LLM generates code that executes in a sandbox and invokes tools at runtime.
This post focuses on the Fargate approach and addresses a practical deployment concern: how do you combine Fargate's serverless operational model with fast sandbox provisioning? The answer is warm pools — pre-provisioned idle sandboxes that are ready to serve requests the moment they arrive. We use the kubernetes-sigs/agent-sandbox operator to manage this warm pool declaratively, giving us Kubernetes-native lifecycle management on top of Fargate's serverless compute.

Why Fargate for Agent Sandboxes

EKS Fargate is a serverless compute engine where each pod runs in a dedicated, AWS-managed environment. For agent sandboxes, this provides several operational benefits:
No node management. There are no EC2 instances to provision, patch, or scale. No AMIs to maintain. No operating system updates to roll out. AWS manages the compute infrastructure entirely — you define pod specs, Fargate runs them.
No capacity planning. You don't need to decide how many nodes to run, what instance types to use, or how to bin-pack pods onto nodes. Fargate allocates resources per-pod on demand. There is no concept of a "full node" or a scheduling failure due to insufficient capacity (within service limits).
AWS-managed isolation. Each Fargate pod runs in its own dedicated environment with a separate kernel. Pods cannot share memory, observe each other's processes, or interact at the kernel level. The isolation mechanism is managed entirely by AWS — there is no runtime to install, configure, or update.
Pay-per-pod pricing. You pay for the vCPU and memory your pods consume, billed per second. When pods are terminated, billing stops. You never pay for unused node capacity.
Security without complexity. The pod security model on Fargate doesn't require custom AMIs, RuntimeClasses, or runtime installations. Standard Kubernetes security contexts (non-root, drop capabilities, no privilege escalation) work natively. The isolation boundary is provided by the platform without any configuration.

Warm Pools with kubernetes-sigs/agent-sandbox

The kubernetes-sigs/agent-sandbox project is a Kubernetes SIG Apps initiative that provides custom resources for managing isolated, stateful workloads. It implements a controller pattern with three CRDs relevant to warm pool management:

SandboxTemplate

Defines the pod specification for sandbox instances. This is the blueprint — what container image to run, what resources to allocate, what security context to apply. Templates are reusable across multiple warm pools and claims.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxTemplate
metadata:
name: code-sandbox
namespace: sandbox
spec:
networkPolicyManagement: Unmanaged
envVarsInjectionPolicy: Allowed
podTemplate:
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
containers:
- name: driver
image: <ecr-repo>/sandbox-driver:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]

SandboxWarmPool

Maintains a configurable number of pre-warmed sandbox instances based on a template. The operator provisions pods proactively and replenishes the pool automatically when instances are consumed.
1
2
3
4
5
6
7
8
9
10
11
apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxWarmPool
metadata:
name: code-sandbox-pool
namespace: sandbox
spec:
replicas: 3
sandboxTemplateRef:
name: code-sandbox
updateStrategy:
type: Recreate
The replicas field controls how many idle sandboxes are maintained. When a sandbox is claimed, the pool drops below target and the operator provisions a replacement. The updateStrategy: Recreate ensures that when the template changes, stale pods are replaced with fresh ones.

SandboxClaim

Claims a sandbox from the warm pool. This is the checkout mechanism — creating a SandboxClaim causes the operator to bind an available pre-warmed sandbox to the claim. Once bound, the claim's status includes the pod IP for direct connectivity.
1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxClaim
metadata:
name: session-abc123
namespace: sandbox
spec:
sandboxTemplateRef:
name: code-sandbox
lifecycle:
shutdownTime: "2026-06-02T12:30:00Z"
shutdownPolicy: Delete
warmpool: default
The lifecycle.shutdownTime provides a TTL — the operator automatically deletes the sandbox when the time is reached, even if the client doesn't explicitly release it. The shutdownPolicy: Delete ensures the claim and its associated pod are fully cleaned up.

Operator Lifecycle

The operator runs as a single controller deployment and manages the full lifecycle:
  1. Provision — watches the SandboxWarmPool and ensures replicas idle pods exist at all times
  2. Bind — when a SandboxClaim is created, selects an available pod from the matching warm pool and binds it to the claim
  3. Expose — populates the claim's status with the bound sandbox's pod IPs
  4. Terminate — when the claim is deleted (or its TTL expires), terminates the bound pod
  5. Replenish — detects the pool is below target and provisions a replacement pod

Architecture and API Flow

The following diagram shows the warm pool lifecycle — from pre-provisioning through checkout, execution, and replenishment. Compare this to Figure 5 in the first post, which shows the traditional Job-based lifecycle where every request pays the full cold-start cost.
The key difference: steps 1-3 complete in under 5 seconds because the pod is already running. In the Job-based flow (Figure 5 in the first post), the equivalent steps require Fargate to provision a new environment, attach an ENI, and pull the container image — adding 30-45 seconds before the orchestrator can connect.

API Differences: Traditional vs SIG CRDs

The key architectural difference from the approach described in our first post is how sandbox lifecycle is managed. Previously, the orchestrator created Kubernetes Jobs directly and polled for pod readiness. With the SIG operator, the orchestrator creates declarative resources and the operator handles the mechanics.
Traditional approach (K8s Jobs):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 1. Create a Job
batch_v1.create_namespaced_job(namespace="sandbox", body=job_spec)

# 2. Poll for pod readiness (30-45s on Fargate)
while not pod_ready:
pods = core_v1.list_namespaced_pod(label_selector=f"job-name={job_name}")
for pod in pods.items:
if pod.status.phase == "Running" and pod_is_ready(pod):
pod_ip = pod.status.pod_ip
break
time.sleep(1)

# 3. Connect and execute
driver = TCPConnection(pod_ip, 8080)

# 4. Delete the Job
batch_v1.delete_namespaced_job(name=job_name, namespace="sandbox")
Each request pays the full Fargate cold-start cost — environment provisioning, ENI attachment, and container image pull.
SIG CRD approach (SandboxClaim from warm pool):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 1. Create a SandboxClaim — operator binds to pre-warmed pod
custom_api.create_namespaced_custom_object(
group="extensions.agents.x-k8s.io",
version="v1alpha1",
namespace="sandbox",
plural="sandboxclaims",
body=claim_spec,
)

# 2. Poll claim status for pod IP (< 5s — pod is already running)
while not pod_ip:
claim = custom_api.get_namespaced_custom_object(...)
pod_ips = claim.get("status", {}).get("sandbox", {}).get("podIPs", [])
if pod_ips:
pod_ip = pod_ips[0]
break
time.sleep(1)

# 3. Connect and execute (same TCP/NDJSON protocol)
driver = TCPConnection(pod_ip, 8080)

# 4. Delete the SandboxClaim — operator terminates pod and replenishes pool
custom_api.delete_namespaced_custom_object(...)
The cold-start cost is absorbed by the warm pool — pods are already running when the claim arrives. The operator handles termination and replenishment in the background.

What Stays the Same

The sandbox container, communication protocol, and security model are identical to the first post's implementation:
  • Same container image (distroless, non-root, drop ALL capabilities)
  • Same NDJSON protocol over TCP for code execution and tool relay
  • Same call_tool() mechanism for bidirectional communication
  • Same pod security context (no service account token, no privilege escalation)
  • Same VPC network architecture (private subnets, no internet, ECR via PrivateLink)
The SIG operator adds a management layer on top — it doesn't change what runs inside the sandbox or how the orchestrator communicates with it.

Performance

With the warm pool pattern, sandbox checkout — from creating a SandboxClaim to receiving a pod IP — completes in under 5 seconds on Fargate. The pod is already running and healthy when the claim is created; the operator simply binds the claim to an available pod and exposes its IP in the status.
Pool replenishment happens asynchronously. When a sandbox is consumed, the operator detects the pool is below target and provisions a replacement. On Fargate, replenishment takes 30-45 seconds (the standard Fargate provisioning time). During this window, the pool operates with reduced capacity.

Cost

Fargate pricing is per-pod: vCPU-seconds and GB-seconds of memory, billed per second with a 1-minute minimum. Warm pool pods incur cost while idle — they are running pods that haven't been claimed yet. For a pool of 3 sandboxes at 0.25 vCPU / 512 MiB each, the idle cost is approximately $0.015/hour total. This is the price of instant availability.
Right-sizing replicas in the SandboxWarmPool is the primary cost lever. A smaller pool saves money but increases the probability of exhaustion under load. A larger pool ensures availability but costs more during quiet periods. The operator's replicas field can be adjusted at runtime without redeployment — making it straightforward to tune based on observed demand patterns.

Tradeoffs

Fargate's serverless model simplifies operations significantly, but it comes with constraints that may affect certain workloads:
NetworkPolicy is not enforced. Fargate pods do not run a CNI DaemonSet. Kubernetes NetworkPolicy resources can be applied to the namespace, but they are not enforced at the pod level. Network isolation relies entirely on VPC security groups attached to the pod's ENI. This means two sandbox pods can potentially communicate with each other if the security group allows it (e.g., via a self-referencing inbound rule). For workloads that require strict pod-to-pod network isolation enforced at the kernel level, gVisor or Kata Containers on EC2 with the VPC CNI Network Policy Agent is the appropriate choice.
Ephemeral storage is capped at 20 GiB. Fargate pods receive a maximum of 20 GiB of ephemeral storage (configurable up to 200 GiB with additional cost). For sandbox workloads that need large temporary datasets or substantial working directories, this may be insufficient. EC2-backed pods with EBS volumes or instance store provide higher storage capacity.
vCPU and memory limits. Fargate pods are limited to a maximum of 4 vCPU and 30 GiB memory per pod. For most agent sandbox workloads (Python code execution with tool calls), these limits are more than sufficient. For compute-intensive workloads (ML inference, large data processing), EC2 provides access to larger instance types.
No DaemonSet support. Fargate does not run DaemonSets. Observability agents, log collectors, and security monitoring tools that typically deploy as DaemonSets must instead run as sidecar containers within the pod. This increases per-pod resource consumption and adds complexity to the pod spec.
No GPU support. Fargate does not provide GPU-attached compute. Workloads requiring GPU acceleration (ML inference, rendering) must run on EC2 node groups with GPU instance types.
No custom runtimes. Fargate runs a fixed container runtime. You cannot use gVisor, Kata Containers, or other custom RuntimeClasses on Fargate. The isolation mechanism is AWS-managed and not configurable.
Pool exhaustion. If all warm pool members are consumed before replenishment completes, subsequent claims experience the full Fargate cold-start provisioning time. Sizing the pool appropriately for expected concurrency is important for consistent performance.

Conclusion

EKS Fargate combined with the kubernetes-sigs/agent-sandbox warm pool operator provides a serverless sandbox architecture with fast checkout and minimal operational overhead. Fargate handles compute isolation and resource management. The SIG operator handles pool lifecycle — provisioning, claiming, terminating, and replenishing. The orchestrator focuses exclusively on application logic: querying the LLM, creating claims, executing code, and relaying tool calls.
This approach eliminates the operational burden of node management, AMI maintenance, capacity planning, and scaling decisions. It trades those concerns for Fargate's constraints — no NetworkPolicy enforcement, storage and compute caps, no DaemonSets. For workloads that fit within those boundaries, the result is a production-ready sandbox fleet that requires no infrastructure maintenance and scales with demand.
For workloads that require NetworkPolicy enforcement, GPU support, or custom runtimes, our first post covers gVisor and Kata Containers on EC2 as alternative approaches with different tradeoff profiles.

Series: Agent Sandboxes on AWS (5 articles)

  1. 1
    Agent Sandboxes on AWS: AgentCore, Lambda MicroVMs, and EKS
  2. …
  3. 3
    Agent Sandboxes on AWS: Optimizing Sandbox Checkout Latency with Warm Pools on EKS Fargate This article
  4. 4
    Agent Sandboxes on AWS: Implementing Programmatic Tool Calling with Firecracker on Lambda MicroVMs
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article