Spec-Based vs Traffic-Based API Regression Testing on AWS
Compare spec-based and traffic-based API regression testing on AWS, using API Gateway OpenAPI exports, recorded traffic, and CI pipelines in CodePipeline.
Developer Advocate
A typical API on AWS isn’t really one thing. There’s an API Gateway endpoint up front, a Lambda
Spec-Based vs Traffic-Based API Regression Testing on AWS: When to Generate Tests from Your OpenAPI Spec and When to Record Real Traffic
A typical API on AWS isn’t really one thing. There’s an API Gateway endpoint up front, a Lambda function or an ECS service doing the work, a DynamoDB table holding state, an SQS queue passing jobs along, and probably a third-party API or two that someone wired in last year. When a deploy changes what that API returns, the regression could have started in any of those layers.
That’s why API regression testing on AWS tends to start with a slightly bigger question than “which tool?” It’s really about where your expected behavior comes from. One option is the OpenAPI spec, which describes what the API is supposed to do. The other is real traffic, which shows what it actually does. Both are useful, they catch different problems, and knowing which one fits a given service saves a lot of wasted effort.
Why API regression testing on AWS is tricky
The managed services are part of the challenge. DynamoDB, SQS and EventBridge are great to build on, but tests that call them for real need credentials, network access and data that stays put between runs. Shared test tables get modified by other pipelines. Queues fill up with messages from someone else’s job. A regression test that fails because a staging table changed overnight teaches people to ignore red builds.
Then there’s the sheer number of services. A team might own a dozen APIs across Lambda and ECS, each with its own deploy cadence, each calling the others. A change in one service’s response can break three others, and nobody finds out until a downstream pipeline fails or a customer notices.
So whichever approach you choose has to deal with two things: deciding what “correct” looks like, and keeping AWS dependencies from making results unreliable.
Spec-based testing: starting from the OpenAPI definition
If your API runs behind API Gateway, you may already have a spec without realizing it. REST APIs can export their definition from a deployed stage as OpenAPI 2.0 or 3.0, either from the console or the CLI, and HTTP APIs export as OpenAPI 3.0. Teams that define their APIs spec-first, or with infrastructure as code, often keep a maintained spec in the repo too.
Spec-based tools read that definition and generate requests on their own. Schemathesis is a good example. It works through every documented operation, sends a wide spread of inputs including strange ones, like empty strings, huge numbers and unexpected types, and checks whether responses match what the spec promises.
What this approach catches well:
- Schema violations: a response that’s missing a required field, or returns the wrong type.
- Crashes on unusual input: a 500 error where a 400 should have come back.
- Undocumented behavior: status codes or content types the spec never mentions.
What it misses is just as important. A spec describes shape, not meaning, so a total that’s suddenly calculated wrong still passes, as long as it’s still a number. It knows nothing about real usage patterns, like which endpoints matter most or which parameter combinations clients actually send. And it doesn’t model your dependencies. The test hits the API, and whatever DynamoDB or the third-party API returns behind it is up to the environment.
There’s one AWS-specific catch worth knowing. With Lambda proxy integrations, API Gateway passes requests straight to the function and often has no response models defined. The exported spec may describe requests in detail and responses barely at all. That leaves spec-based tests with very little to check on the response side, unless someone fills those schemas in.
Traffic-based testing: starting from what actually happened
Traffic-based testing flips the source around. Instead of asking what the spec says, it records real requests hitting a service and the responses that came back, then replays them after every change. Any difference in a response gets flagged.
The more useful setups also record what the service did behind the scenes while handling each request: its calls to DynamoDB, SQS or a third-party API. During replay, those calls are answered from the recording instead of going out to AWS. That’s what makes runs repeatable. The test no longer depends on a staging table staying unchanged or a partner sandbox being up. Since AWS SDK calls are signed HTTPS requests under the hood, they can be captured like any other outbound call, although signature headers and timestamps change every time and need to be treated as noise.
Keploy is one example of this approach. It records traffic at the Linux kernel level using eBPF, so the service needs no code changes, and it saves outbound calls as mocks for replay. Kernel-level capture does need a host that allows it, so on AWS, teams usually record by running the service in a container locally, in CI or on EC2, rather than inside Lambda or Fargate, which don’t allow that level of access.
What traffic-based testing catches well:
- Real behavior changes: a field that disappeared, a value calculated differently, a status code that shifted, even when the schema still looks valid.
- Realistic edge cases: the odd parameters and half-filled payloads real clients send, which nobody would have scripted.
- Changes in downstream calls: a refactor that suddenly reads a different DynamoDB key or skips a queue message.
And what it misses: it can only replay what was recorded. An input nobody has sent yet won’t be in the baseline. Endpoints that barely get any traffic during the recording window may barely get tested. Recordings from real environments also need scrubbing, so personal data never ends up stored in a repo.
When to use which
Situation on AWS | Spec-based | Traffic-based |
|---|---|---|
New API, little or no real traffic yet | Strong fit | Weak, nothing to record |
Mature API with years of real usage | Useful for edge cases | Strong fit |
Lambda proxy integrations with no response models | Limited | Strong fit |
Heavy use of DynamoDB, SQS or third-party APIs | Doesn’t handle dependencies | Strong fit with recorded mocks |
Catching schema and contract breaks | Strong fit | Catches them as response diffs |
Finding crashes from unusual inputs | Strong fit | Only for inputs seen before |
In short, spec-based testing explores what could be sent. Traffic-based testing protects what is already being sent.
Using both in one CI pipeline
The two approaches fit together nicely, and the pipeline doesn’t need to be complicated.
A sensible first stage is a spec diff. Export the OpenAPI definition for the change and compare it with the version on the main branch. If a field was removed or a type changed, the pipeline can flag it before any test even runs. It’s fast and cheap, and it catches the most obvious contract breaks.
Next, run the traffic-based replay against the new build, with AWS dependencies served from recordings. This is the step that catches behavior changes the spec can’t describe. Because nothing calls real AWS services, it’s also quick and stable enough to run on every pull request.
Spec-based fuzzing tends to fit best as a separate stage, often running against a deployed test stage in API Gateway, since it needs a live endpoint and can take longer. Plenty of teams run it nightly rather than on every change.
This works the same way in CodePipeline, with CodeBuild running each stage, or in GitHub Actions, using OIDC to assume an AWS role for the export and deploy steps. Keep any credentials the tests need in Secrets Manager, never in the recordings or the repo.
Tools
A few tools cover the pieces described above. They’re examples of each approach, not a ranking.
- Keploy : traffic-based testing that records API calls and their downstream dependencies, then replays them with mocks.
- Schemathesis : spec-based testing that generates requests from an OpenAPI or GraphQL definition.
- oasdiff : compares two OpenAPI specs and reports breaking changes, which suits the spec diff stage.
- Pact : consumer-driven contract testing, useful when several teams depend on the same internal APIs.
Where to start
Pick one service where a regression would actually hurt, ideally an API behind API Gateway with at least one AWS dependency. Export its spec and look at how complete it is, especially the response schemas. A detailed spec makes spec-based testing worth adding straight away. A thin one, which is common with Lambda proxy integrations, is a strong hint to start with recorded traffic instead.
Either way, add the spec diff to the pipeline early. It costs almost nothing and tends to catch a breaking change within the first few weeks, which makes the case for the rest of the setup much easier.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article