
Test your coding agents locally with AWS Emulator
Coding agents wait on AWS deployments to test their calls. See where AWS Emulator fits between mocks and real AWS, and what still has to run on AWS.
Series: A study on AI in the product lifecycle (4 articles)
- …
- 3Test your coding agents locally with AWS Emulator This article
How much of your coding agent's time goes to waiting on AWS? In part 1 of this series, I cited research putting code authoring at 25 to 35% of the idea-to-launch span , with review, provisioning, deployment, and waiting taking the rest. An agent that has to deploy a stack before it can test an AWS call pays that wait on every attempt. AWS Emulator , a local endpoint for the AWS APIs that needs no AWS account, removes the deployment step from that loop. My view is that it belongs in the agent's loop as a middle test layer between mocks and real AWS, and that you should judge it by measured attempts to merge. In this article, I cover how the emulator works, how close it is to real AWS, where it fits in the agent loop and the pipeline, what still has to run on AWS, and how to measure whether it helps.
Where does a local emulator fit in your tests?
Real AWS still decides whether your AWS Identity and Access Management (IAM) policies allow a call and how a service behaves at your quotas and load. A contributor to AWS Blocks who ran its mocks and the emulator against the same tests describes the emulator as a third layer between the mocks and the sandbox .
Layer | What it tells you | What it misses |
|---|---|---|
Mocks and stubs | Whether your own logic works | Any AWS behavior you didn't write into the mock |
AWS Emulator | Whether your AWS calls, errors, and multi-service workflows behave as they would on AWS | IAM policy evaluation, Region and account separation, quotas, load, and some data plane behavior |
Real AWS in a sandbox account | Whether the whole application works, securely and at scale | Fast, isolated attempts that cost nothing |
How AWS Emulator works
AWS generates the emulator's services from the same published API models that the AWS software development kits (SDKs) use, so requests and errors have the same shape as they do on AWS . Each copy you start gets its own port, keys, and in-memory state, so several agents or test jobs can run at the same time without seeing each other's resources. When an operation isn't implemented yet, the emulator returns an HTTP 501 Not Implemented error, so the test fails right away. Your code doesn't change beyond the endpoint, and a coding agent can set the emulator up for you through an aws-emulator skill . The source is released to the public domain under the CC0 dedication, so anyone can reuse it, apart from vendored code under the Apache 2.0 license. There is no service fee.
How close is the emulator to real AWS?
A fidelity test is an SDK workflow that AWS has verified against real AWS. It passes only when the emulator matches every response, including the status code and the exact error code and message. AWS publishes a scores file on each GitHub release and keeps the test suite public, so you can check the numbers yourself. I counted two of those files, the launch release (v1.0.322) and v1.0.338 , which AWS published on November 12, 2026, less than two days later.
At launch, 56.5% of 42,937 tests passed, and by v1.0.338, 59.7% of 43,548 passed. AWS added 612 tests and removed one in between, so the fairer comparison uses the 42,856 tests present and scored in both releases, where the pass rate moved from 56.6% to 60.1%. Per service, scores in v1.0.338 run from above 90% to below 20%, and AWS says control plane APIs, the ones that create and describe resources, match AWS most closely . Figure 1 counts tests that call only that service.

Figure 1. Fidelity tests passing by service in AWS Emulator v1.0.338
DynamoDB's score fell from 77.3% to 71.1% between the two releases because AWS added 437 DynamoDB tests. On the 1,044 DynamoDB tests present in both releases, it rose from 77.3% to 80.7%. Read the scores for the services your application calls, compare the same tests over time, and quote any number with its release. AWS itself calls each score a snapshot that can go down even while the emulator gets better . With CloudFormation at 16.0%, try your own stacks before you rely on local deployments of infrastructure as code (IaC). AWS notes that fidelity varies by service for IaC tools and for data plane APIs, the ones that read and write the data inside a resource.
Where does it go in the agent loop and the pipeline?
Start with the agent's own loop. Part 1 described the one-shot level, where a spec produces a mergeable pull request without revision. An agent that runs its AWS integration tests before it opens a pull request has a better chance of getting there. Consider a Kiro steering rule that integration tests run against the emulator before a task is complete. Pair it with a Kiro hook that clears real AWS credentials from the shell, starts the emulator, and runs the tests when a task finishes. AWS hasn't published this pattern, so try it with one team first.
In your continuous integration (CI) pipeline, consider one emulator per AWS CodeBuild build. AWS hasn't published a CodeBuild guide for this either. A CodeBuild build runs with the credentials of its service role, so give the emulator build a role with no permissions beyond writing its logs. In an independent test of the launch release by DevOps Daily , the JavaScript SDK's Amazon SQS client followed the queue URL the emulator returned and called real AWS. With a minimal role, an unintended call to real AWS can't access your resources. Pin the version your pipeline installs, since AWS published two releases on November 12 alone. Telemetry is on by default , so decide whether you want it in your builds. A later pipeline stage then deploys to a sandbox account for everything the next section covers.

Figure 2. Where each kind of test runs
What still has to run on AWS
The emulator doesn't evaluate IAM policies and resource-based policies the way AWS does, and AWS says not to use it to test whether your application is secure . The DevOps Daily test shows what that limit looks like in practice. It matched AWS in 27 of 32 permission cases, and every miss allowed a request that AWS refuses. In one case, a user with no policies assumed an admin role and created an IAM user with it. Keep the permission checks and penetration testing from the Build and test phase of part 2 in a deployed test environment.
Each emulator also acts as a single account with no cross-account or cross-Region behavior, so run one emulator per Region you need to test. AWS hasn't tested it for long-running use or load . Test service quotas, throttling, and eventual consistency there too, since none of the sources I reviewed covers them.
What does an agent-built emulator say about trust?
AWS says autonomous agents wrote all of the emulator's code and that its team hasn't written or reviewed any of it . Separate groups of agents maintain the tests, find issues, and fix them. Every change passes a merge gate in which other agents review it against coding standards and for security and check that fidelity didn't regress. AWS says that without this gate it wouldn't trust what its agents build.
This is the autonomous level from part 1's Principle 4, with fidelity tests acting as the evaluations that decide what merges. AWS also started with an advantage most teams lack, a published specification in its API models and a reference system in AWS itself. The case also tests the rule I set in part 1, that an agent should widen its scope only when its error rate falls below a measured human baseline. AWS publishes no human comparison, and about 40% of the tests in v1.0.338 still fail.
I think an agent can own work before it matches a person when its failures are loud, public, and cheap. Here an unsupported call returns a 501, open gaps sit in a public folder in the repository, and a wrong answer costs a failed test. The same test applies to your own agents, and it rarely passes where a wrong answer reaches a customer.
How do you know it's helping?
Part 1 argued for measuring the whole system, and the emulator deserves the same treatment. Have the engineering lead for one repository tag pull requests whose integration tests ran against the emulator. Then compare them with the rest for a quarter on attempts to reach a merged pull request, review wait, and change failure rate. If attempts fall and change failure rate holds, the emulator is removing waiting. If change failure rate rises, a gap in fidelity or in your real AWS stage is letting problems through. Rerun the fidelity count for your services each month, because it tells you which tests you can trust locally.
To start, install the skill in one repository, pick the two highest-scoring services among those your application calls most, and add one emulator step to its pipeline. When your coding agents test an AWS call today, do they run against mocks, a shared development account, a sandbox account per developer, or nothing until the pipeline?
Further reading
- Part 1: Measure AI development speed across the product lifecycle (AWS Builder Center)
- Part 2: DevOps agents and SDLC security on AWS (AWS Builder Center)
- Test your applications against AWS locally with AWS Emulator (AWS News Blog)
- Fidelity in AWS Emulator (AWS Emulator User Guide)
- How we built the agents that built AWS Emulator (About Amazon)
About the author
Michael Purpura is a Senior Global Private Equity Solutions Architect at AWS. He works with private equity firms and their portfolio companies to structure AI adoption around measurable business outcomes.
Series: A study on AI in the product lifecycle (4 articles)
- …
- 3Test your coding agents locally with AWS Emulator This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article