When “Succeeded” Still Means Wrong: Building CounterFlow to Catch Duplicate Charges on AWS
A serverless workflow can complete successfully and still leave the business in the wrong state. We built CounterFlow on AWS to reproduce a duplicate-payment retry bug, identify exactly where correctness breaks, and prove an idempotent fix under the same failure.
A payment workflow can complete successfully while charging a customer twice.
That sounds counterintuitive, but it is exactly the kind of failure scenario that must be handled in distributed systems.
Imagine a check-out service charges ₹999 successfully - the payment commit was accepted, but before the service can reply with success, the request expires.
The workflow engine sees a failure and will retry.
If the payment operation is not idempotent, the retry can create another ₹999 charge.
The infrastructure may indicate:
SUCCEEDED
but the business outcome would be:
1 ORDER → 2 CHARGES
₹999 expected → ₹1,998 charged
This disconnect between execution success and business correctness was the driving force behind building CounterFlow.
We built CounterFlow as a part of the four-day AWS x WeMakeDevs First Commit hackathon to help backend engineers deliberately reproduce such failure scenarios, understand precisely where correctness breaks, and verify that a fix survived the same failure.
The problem we wanted to solve
Modern backend systems are increasingly dependent on retries.
This is helpful behavior, due to inevitable temporary failures:
network timeouts,
service interruptions,
delayed responses,
transient database errors,
lost acknowledgements.
But there is an important distinction though:
A failed request does not necessarily mean the operation itself failed.
A payment may have already been committed before the caller sees a timeout.
If the caller blindly retries, the same logical action can happen twice.
For CounterFlow, we modeled a simple checkout workflow:
CreateOrder
↓
ReserveInventory
↓
ChargePayment
↓
ConfirmOrder
The interesting failure scenario is during
ChargePayment .We deliberately inject a failure after the payment side effect has already happened.
Conceptually:
Charge ₹999
↓
Payment committed
↓
Acknowledgement lost
↓
Workflow sees failure
↓
Retry
This is very different from failing before the charge had happened.
The side effect already existed.
Designing the Buggy experiment
We want this failure to be deterministic, not random.
So CounterFlow defines a fault plan similar to:
Type: AFTER_SIDE_EFFECT_TIMEOUT
Target: ChargePayment
Attempt: 1
On the first attempt:
PaymentCharged
↓
InjectedFailure
The workflow believes the operation failed and will retry.
Our intentionally Buggy implementation will generate a new payment identity on retry.
The result will be:
Attempt 1 → PaymentCharged
Attempt 1 → InjectedFailure
Attempt 2 → PaymentCharged
From the workflow engine's perspecitve, the retry eventually succeeded.
From the customer's perspective, something much worse happened.
They were charged twice.
Why AWS Step Functions was central to the experiment
AWS Step Functions orchestrates the checkout.
This was important, since we did not want to build a frontend that pretended a retry occurred.
The retry happens in the real serverless workflow.
Step Functions executes:
CreateOrder
→ ReserveInventory
→ ChargePayment
→ ConfirmOrder
and its retry behavior reproduces the exact condition we wanted to study.
This also taught us an important lesson:
Orchestration engines are great at answering "Did the workflow finish?"
They do not automatically answer "Did the business remain correct?"
That second question requires knowledge of the business rules.
That is where CounterFlow's invariant checker comes in.
Turning a business rule into an invariant
Our payment rule is simple:
An order must never have more than one committed charge.
We represent that as:
ChargeAtMostOnce
For an active order:
committedCharges <= 1
And once the order is confirmed:
committedCharges == 1
This check is deterministic application logic.
We deliberately did not use an LLM to decide whether a run passed or failed.
Correctness should not depend on interpretation when a business rule can be stated precisely.
In the Buggy execution, CounterFlow observe:
1 ORDER → 2 CHARGES
and produce:
Workflow: COMPLETED
Invariant: VIOLATED
That became one of the most important UI decisions in the project.
We deliberately show the execution state and business correctness separately.
Because they are not the same thing.
Finding where correctness first breaks
Simply stating:
FAILED
was not enough.
We wanted CounterFlow to answer a much more useful question:
What is the first event after which the system becomes incorrect?
So we introduced the concept of a first failing prefix.
Consider this trace:
CreateOrder
ReserveInventory
PaymentCharged
InjectedFailure
PaymentCharged
ConfirmOrder
CounterFlow evaluates the invariant, on the trace as it grows.
After the first payment:
charges = 1
PASS
After the injected failure:
charges = 1
PASS
After the retry commits another payment:
charges = 2
FAIL
Therefore the second
PaymentCharged event became the first violating event.Rather than forcing an engineer to inspect raw logs, CounterFlow highlights precisely where the system had crossed from valid to invalid state.
This changed the way we thought about observability.
Logs tell you what happened.
Correctness evidence should tell you when what happened became wrong.
Building the fix with DynamoDB idempotency
The fix was not to remove retries.
Retries are useful.
The fix was to make the payment operation safe to retry.
In our Fixed implementation, we use an order-scoped payment identity, plus an Amazon DynamoDB conditional write.
The first attempt creates the payment.
When the retry occurs, the application detects that the logical payment already exists.
Instead of producing another charge:
PaymentCharged
the retry produces:
PaymentReused
The Fixed trace becomes:
Attempt 1 → PaymentCharged
Attempt 1 → InjectedFailure
Attempt 2 → PaymentReused
The retry still existed.
The duplicate side effect did not.
The final result:
1 ORDER → 1 CHARGE
Expected: ₹999
Charged: ₹999
Overcharge: ₹0
ChargeAtMostOnce: PASSED
This was one of the clearest lessons of the project:
Idempotency does not stop retries.
It makes retries safe.
Making the comparison fair
Once we had Buggy and Fixed implementations, another problem arose.
If we test them using different failures, the comparison is not very strong.
So CounterFlow stores the resolved fault-plan snapshot used for the original run.
We also calculate a SHA-256 hash for that stored plan.
When the user selects Compare with Fixed, CounterFlow does not resolve a new fault configuration.
It replays the exact stored fault.
Conceptually:
Same workflow scenario
Same fault type
Same target
Same attempt
Same stored fault snapshot
Only the implementation changes
That gives us a much stronger experiment.
In our final acceptance run:
| | Buggy | Fixed |
| ------------------- | -----: | -----: |
| Payment attempts | 2 | 2 |
| Committed charges | 2 | 1 |
| Invariant | FAILED | PASSED |
| Customer overcharge | ₹999 | ₹0 |
The failure remained constant.
The implementation changed.
This is what made the result convincing.
How AWS services fit together
CounterFlow is built as a serverless AWS application.
AWS Step Functions
Step Functions orchestrates the checkout workflow, and performs the retry behavior that makes our experiment possible.
AWS Lambda
We used Lambda functions for the individual workflow operations:
CreateOrder
ReserveInventory
ChargePayment
ConfirmOrder
We also used Lambda for the Control API and invariant-checking logic.
Amazon DynamoDB
DynamoDB stores:
simulated business state,
payment records,
trace evidence,
run metadata,
stored fault information.
It also provides the conditional-write behavior used in our idempotent Fixed payment flow.
Amazon API Gateway
API Gateway exposes the CounterFlow APIs used by the frontend, including operations for:
POST /runs
GET /runs/{runId}
GET /runs/{runId}/trace
POST /runs/{runId}/compare
Amazon CloudWatch
CloudWatch helped us inspect Lambda behavior and debug the deployed workflow.
Amazon S3
The React frontend is deployed using S3 static website hosting.
AWS SAM and CloudFormation
The infrastructure is defined as code using AWS SAM.
Our Lambda functions, DynamoDB table, API Gateway configuration, Step Functions state machine, permissions, environment configuration, and other infrastructure can all be deployed reproducibly from the project configuration.
This became especially useful during a four-day hackathon where the infrastructure was changing often.
What deployment taught us
One of the big surprises was that getting the application working locally was only part of the job.
Our permanent deployment unearthed issues that didn't appear in local development.
CORS
Our browser needed to send a custom:
X-Demo-Token
header.
API Gateway therefore needed that header explicitly allowed.
The API could work perfectly from a direct request, but would fail from a browser.
Authenticated polling
The frontend starts a workflow and then polls for its status.
We discovered that authentication was not only needed when the run was created; the polling requests also needed to include the token.
Browser compatibility
Our frontend originally used:
crypto.randomUUID()
for comparison request tokens.
The S3 HTTP deployment did not provide the exact browser environment that we had assumed, so we added a compatible fallback.
These were all small problems individually, but they reinforced an important lesson:
Deployment is part of testing.
A system is not complete simply because its code runs locally.
The browser, API, permissions, cloud infrastructure, and workflow all need to work together.
What we liked about AWS
The strongest part of the AWS experience was how naturally the services mapped to the architecture.
Step Functions took care of orchestration.
Lambda handled compute.
DynamoDB took care of state and idempotency.
API Gateway exposed the system.
SAM gave us a chance to define everything as code.
For a small team building quickly, that meant that we could create a distributed workflow without managing servers.
Step Functions was particularly valuable as retry behavior was visible and configurable, rather than buried inside of application code.
DynamoDB conditional writes helped demonstrate an abstract concept - idempotency - directly.
What we would improve
The biggest challenge was debugging across service boundaries.
When one browser action touches:
API Gateway
→ Lambda
→ Step Functions
→ multiple Lambdas
→ DynamoDB
→ CloudWatch
Finding the exact failure can require jumping between several tools.
A more unified correlation view across API requests, Step Functions executions, Lambda logs, and application-level run IDs, would have made debugging serverless applications much easier.
We also found CORS configuration easy to get subtly wrong.
Better validation for browser-facing API Gateway configurations would help teams detect those problems before deployment.
What four days changed for us
We started CounterFlow thinking primarily about retry behavior.
We finished thinking much more deeply about correctness.
The most important lesson we learned was:
A successful retry is not necessarily a correct recovery.
This changed the way we think about distributed systems.
We also learned that good observability is not only infrastructure metrics.
Sometimes the most useful signal is a business invariant like:
A customer must not be charged twice.
And we learned that if you want to prove a fix, reproducing the same failure conditions matters just as much as showing that the new implementation passes.
The final outcome
CounterFlow's Buggy implementation demonstrates:
Workflow: COMPLETED
Invariant: VIOLATED
1 ORDER → 2 CHARGES
Expected: ₹999
Charged: ₹1,998
Overcharge: ₹999
The Fixed implementation receives the same fault, but produces:
Workflow: COMPLETED
Invariant: PRESERVED
1 ORDER → 1 CHARGE
Expected: ₹999
Charged: ₹999
Overcharge: ₹0
The retry stayed.
The duplicate charge did not.
And that is ultimately what CounterFlow is about:
Breaking workflows intentionally, capturing evidence of what happens, and proving that the fix survives the same failure before production has to discover it for us.
Project links
GitHub:
https://github.com/devaganesh-source/counterflow Live CounterFlow deployment:
http://counterflow-demo-web-618042349623.s3-website-us-east-1.amazonaws.com Built with: AWS Step Functions, AWS Lambda, Amazon DynamoDB, Amazon API Gateway, Amazon CloudWatch, Amazon S3, AWS IAM, AWS SAM, and AWS CloudFormation.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article