I asked an AI agent the same question 11 times. It got it wrong 5 times — in 4 different ways.
The problem nobody talks about When an AI agent gives a wrong answer, what do you do? Most of us run it again. But an AI model doesn't give the same answer twice. The second run might be right. Or wrong in a different way. The mistake you wanted to study is gone. Logs don't help much either. They show what the agent said. They don't show which step the wrong answer came from. So I built Replay — a flight recorder and a rewind button for AI agents.
Built at Bharat Builds: First Commit (WeMakeDevs × AWS). Live demo: https://3hy4acxf74.execute-api.ap-south-1.amazonaws.com · Code: https://github.com/GaMeBoY-21/replay
The problem nobody talks about
When an AI agent gives a wrong answer, what do you do?
Most of us run it again. But an AI model doesn't give the same answer twice. The second run might be right. Or wrong in a different way. The mistake you wanted to study is gone.
Logs don't help much either. They show what the agent said. They don't show which step the wrong answer came from.
So I built Replay — a flight recorder and a rewind button for AI agents.
The experiment
I gave an agent one simple accounts job:
"Reconcile invoice INV-2291 from Meridian Supplies and report the total in USD."
The agent had 8 tools: read the invoice, read the line items, read the bank details, look up the vendor, write to a worksheet, convert currency, and submit.
The catch: the invoice doesn't say its currency. The total is 41,000. That could be rupees or dollars. The only clue is in the bank details — HDFC Bank, Baner, Pune. An Indian bank. So it's rupees, and the right answer is ₹41,000 × 0.012 = $492.
I ran the real qwen2.5:14b model 11 times, with the same task, same data, same tools. Nothing scripted.
Result | Runs | What the agent did |
|---|---|---|
✅ $492 | 6 | Picked INR |
❌ $41,000 | 1 | Picked USD — "based on standard business practices" |
❌ $29,930 | 2 | Picked CAD — "inferred" |
❌ $44,280 | 1 | Picked EUR — "default currency for vendor" (a vendor it couldn't even find) |
❌ no total | 1 | Wrote the literal text <result_of_get_invoice_header.currency> as the currency |
5 wrong out of 11. In 4 different ways.
And here's the part that surprised me most: even most of the right answers gave made-up reasons. One said it read the currency from
invoice.header.currency — a field that doesn't exist. Only one run gave the real reason: "based on bank information provided by vendor."The AI's explanation of itself can't be trusted. The recording can.
What Replay does
1. Record
While the agent runs, Replay saves everything that can change between runs: every model reply, every tool call and result, every value written to or read from the agent's memory.
2. Replay
Play the run back from the recording. The model is not called at all — each step gets its saved answer. Same run, every time.
- Recording a run with the model: 70–166 seconds
- Replaying it: 7.8 milliseconds
3. Trace — "where did this wrong number come from?"
Click the wrong answer. Replay follows it backwards through what each step read and wrote:
$41,000 answer ↑ came from the convert step, which READ invoice.currency ↑ which was WRITTEN as "USD" at step 10 ↑ by the model's decision just before it
It doesn't check steps one by one. Every value remembers where it came from, so it jumps straight to the cause. It also shows what the agent had but didn't use: the HDFC Pune bank details were read at step 4 and ignored.

4. Fork — "prove it"
Take the wrong run. Keep every step before the bad decision exactly as recorded. Swap in one different decision (taken from a real good run, not typed by hand). Then let the real model continue from there.
Result: the same model, from the same place, reaches $492.
5. Diff
Put the wrong run and the fork side by side. They're identical up to one step, and every difference comes after it. That's how you know that one step was the cause.

Bonus: circuit breakers
Agents can loop or go too deep. Replay has limits (loops, depth, budget, time). When one trips, the run stops cleanly — and because it's recorded, you can resume it later with a higher limit instead of starting over.
How it runs on AWS
Browser → API Gateway (HTTP API) → Lambda "api" → reads recordings │ DynamoDB "events" (the recordings, append-only) │ DynamoDB Streams: "a new step was saved" ▼ Lambda "projector" → DynamoDB "views" (the list of runs)
S3: any single step bigger than 100 KB IAM (least privilege per function) · CloudWatch logs · all built with AWS CDK
Why these:
- DynamoDB — each step is an append-only item. A conditional write means two writers can never claim the same step number.
- Streams + a projector Lambda — the runs list is rebuilt from the recording, so the list page stays fast and never scans every step.
- Lambda + API Gateway — nothing runs when nobody is clicking. Cold start was about 3 s; warm requests about 0.2–0.5 s.
- S3 — DynamoDB items max out at 400 KB, and a model reply can be big.
- CDK — one command builds it, one command deletes it.
The deployed version is replay-only: it has no model, so it can't run new forks — but every recorded run can be opened, replayed, traced and compared. CloudFront is already in the stack behind a flag; my new account was waiting on AWS verification, so for now the site is served through API Gateway.
Things I learned building on Strands Agents SDK
I used the Strands Agents SDK as the agent framework. To make runs replayable, I had to handle a few things that fail quietly:
- Tools run in parallel by default. Parallel results come back in random order, which breaks step numbering. I switched to the sequential tool executor.
- Every message gets a random
tracking_id. A perfect replay still "fails" an equality check on that field. I remove it before comparing.
- SQLite + threads. Strands runs tool functions on a worker thread. SQLite complained, Strands turned the error into a normal "tool failed" result, and the agent kept going. Writes could vanish without any crash.
The lesson: check what the agent knew, not just what it said.
Honest limits
- Tested on agents of 11–22 steps, not thousands.
- The recordings were made with qwen2.5:14b running locally through Ollama.
- The AWS version replays; forking live needs a model.
Thank you
Big thanks to WeMakeDevs and AWS for Bharat Builds, to the AWS team for the Strands workshop, and to Polaris School of Technology for hosting us in Bengaluru.
Try it: https://3hy4acxf74.execute-api.ap-south-1.amazonaws.com — start with the wrong run, traced:
/runs/qwen-00?trace=output Code: https://github.com/GaMeBoY-21/replay AI tools used: Claude Code helped write the code, tests and infrastructure under my direction. qwen2.5:14b (via Ollama) was the agent under test.
#BharatBuilds #AWS #StrandsAgents #AIAgents
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article