Building AgentGarage on AWS
We built a behavioural-coverage system for tool-using agents: map exercised paths, predict failures with Bedrock, replay them in AgentCore sandboxes, and promote only confirmed bugs to permanent evals. This is how the AWS pieces fit together and what we learned shipping it.
Applied AI | Hackathons @ OpenAI, Anthropic, Microsoft, YCombinator
Autonomous agents pick tools in a loop and branch on what they observe. They often run with broad permissions and thin guardrails. The tests teams write cover a slice of the paths those agents can actually reach.
We built AgentGarage for that gap. It maps which behaviours an agent has exercised, finds reachable state–action pairs it never took, predicts outcomes on those paths with a world model, ranks which predictions deserve a sandbox first, replays the risky ones against the real agent in isolation, and keeps confirmed failures as permanent evals.
The rule we refused to bend: a predicted failure is a hypothesis until the real agent reproduces it. No path gets marked failed on a model's word.
Who it's for
Teams shipping tool-using agents. Our public demo is an autonomous AWS SRE remediator for a checkout service. It gets a goal and five tools (health, logs, deploy history, rollback, verify) with no fixed script. The branching is real, so the behaviour map is real.
Private mode accepts traces from any agent that emits OTLP. Sandbox runs execute your agent container in AgentCore Runtime from an ECR image you register at deploy time.
Status pipeline
Every path moves through four statuses:
- Observed — ran in production (or demo traffic), stored as a redacted trace
- Predicted — world-model step guessed an outcome on an unexplored pair
- Verified — the real agent reproduced the failure inside an isolated sandbox
- Protected — compiled into a permanent YAML eval and re-run on new versions
Only a code verifier moves Predicted → Verified. Only the eval compiler, given a failed rule log, moves Verified → Protected.
End-to-end flow
- The agent runs and exports spans over OTLP.
- An ECS Fargate collector receives them. Kinesis buffers. A redact Lambda strips secrets and PII on an allowlist.
- Redacted traces land in S3. Session rows land in DynamoDB.
- A graph builder writes state edges to Neptune Serverless.
- A scan lists legal state–action pairs with no observed run (unexplored paths).
- We predict outcomes on those pairs and store them as Predicted. We wanted a dedicated world-model API here. We could not secure a key in time, so we used Amazon Bedrock (Nova) as an LLM trajectory / path finder for that step instead.
- Jev (TypeSafe AI) ranks Predicted scenarios so we spend AgentCore budget on the highest-risk paths first.
- Step Functions starts one AgentCore Runtime microVM per scenario: restore start state, inject one fault, run the real agent loop, return snapshot + spans.
- A verifier Lambda checks fixed invariants in code. Broken rule → Verified. This Lambda never runs agent code and never calls the world-model step.
- The eval compiler writes YAML to S3 and indexes it in DynamoDB → Protected.
- The Amplify dashboard lets you inspect gaps, sandboxes, traces, and re-run protected evals against newer agent versions.
Trust split
We treated this as a product constraint, not a slogan:
- The world-model step proposes. It cannot write evals.
- The sandbox executes the agent in a Firecracker microVM (VPC, no internet, killed after). It cannot write evals.
- The verifier applies code-level invariants. It cannot run agents and cannot reach the world-model step.
- Bedrock also narrates and reasons inside the agent loop. It cannot change statuses.
Skip this split and you get a model grading another model. That is not coverage.
Build it: packaging and open stack
We kept two deploy paths from one codebase:
- AWS CDK (Python nested stacks) for the full customer package (
mode=demoormode=private)
- AWS SAM + CloudFormation for the live team demo, deployed through GitHub Actions
- Amplify Hosting for the Next.js dashboard (
amplify.yml, zip deploy when needed)
- Strands Agents as the reference agent framework for the SRE demo image
CDK taught us how to package a multi-service system: network, storage, graph, intake, compute, AgentCore, orchestration, API, frontend. SAM kept the public demo lighter so we could iterate without redeploying Neptune and AgentCore on every push.
Ship it: AWS services we actually ran
Job | Service |
|---|---|
Dashboard hosting | Amplify (us-east-1; Amplify isn't in ap-south-2) |
Auth | Cognito |
HTTP entry | API Gateway |
APIs + workers | Lambda (API, graph, simulate, rank, replay/verify, eval compile) |
Sandbox sequence | Step Functions |
Trace-driven wakeups | EventBridge |
Traces, sandbox logs, evals | S3 (append-only) |
Sessions, jobs, eval index, agent registry | DynamoDB |
Behaviour graph | Neptune Serverless |
Trajectory / path finding + agent reasoning | Amazon Bedrock (Nova) |
Predicted-scenario ranking | Jev (TypeSafe AI) |
Isolated agent replay | Bedrock AgentCore Runtime |
OTLP buffer | Kinesis |
Collector only (never agents) | ECS Fargate |
Agent images | ECR |
Keys / demo creds | Secrets Manager |
Logs / alarms | CloudWatch |
Encryption + network | KMS, VPC |
One sharp edge: our API and data plane live in ap-south-2, while Amplify sits in us-east-1. Env wiring and CDN cache invalidation after green deploys bit us more than once.
What broke in the demo path (on purpose)
The planted SRE bug: after a rollback timeout, the agent retries without an
operation_token / recheck and double-rollbacks past last-known-good. That path shows up as unexplored, gets a Predicted outcome from the Bedrock path finder, gets ranked by Jev, then Verified only when AgentCore reproduces it and the verifier's rules fire. After that it becomes a Protected eval that should fail on the broken agent version and pass after the fix.That loop is the product. The dashboard is how you watch it.
What we learned
This hackathon was a good chance to learn a complete AWS stack. We left with a clearer mental model of how the popular services connect when you actually have to ship them together.
Specifics:
- CDK — packaging different kinds of server-side pieces (Lambdas, VPC services, AgentCore runtime, Amplify) in one app, and keeping demo vs private modes from diverging into two products
- Agent observability — OTLP spans as the contract; redaction before storage; graph edges as the thing you query, not raw logs
- World models — we planned for a dedicated world-model API; when we could not get a key, Bedrock Nova stood in as an LLM trajectory path finder. Still useful for proposing unexplored failures. Still dangerous if you let it decide pass/fail
- Ranking with Jev — we wired Jev from TypeSafe AI to order Predicted scenarios before burning AgentCore sandboxes
- Frontend — graph-first UI (behaviour map, unexplored paths, sandboxes, permanent evals) on Next.js + Amplify, Cognito login, live API data
The lesson we keep repeating to ourselves: isolate the agent, replay with a real fault, verify in code. Models propose. Code promotes.
Links
- Live demo: https://main.d34rv48johwdk6.amplifyapp.com
If you're building agents on AWS and your test suite only covers the happy tool chain, try mapping the paths you never ran. That list is usually longer than you want.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article