AWS Builder Center

When “Succeeded” Still Means Wrong: Building CounterFlow to Catch Duplicate Charges on AWS

A serverless workflow can complete successfully and still leave the business in the wrong state. We built CounterFlow on AWS to reproduce a duplicate-payment retry bug, identify exactly where correctness breaks, and prove an idempotent fix under the same failure.

A payment workflow can complete successfully while charging a customer twice.
That sounds counterintuitive, but it is exactly the kind of failure scenario that must be handled in distributed systems.
Imagine a check-out service charges ₹999 successfully - the payment commit was accepted, but before the service can reply with success, the request expires.
The workflow engine sees a failure and will retry.
If the payment operation is not idempotent, the retry can create another ₹999 charge.
The infrastructure may indicate:
SUCCEEDED

but the business outcome would be:
1 ORDER → 2 CHARGES
₹999 expected → ₹1,998 charged

This disconnect between execution success and business correctness was the driving force behind building CounterFlow.
We built CounterFlow as a part of the four-day AWS x WeMakeDevs First Commit hackathon to help backend engineers deliberately reproduce such failure scenarios, understand precisely where correctness breaks, and verify that a fix survived the same failure.



The problem we wanted to solve

Modern backend systems are increasingly dependent on retries.
This is helpful behavior, due to inevitable temporary failures:
network timeouts,
service interruptions,
delayed responses,
transient database errors,
lost acknowledgements.
But there is an important distinction though:
A failed request does not necessarily mean the operation itself failed.
A payment may have already been committed before the caller sees a timeout.
If the caller blindly retries, the same logical action can happen twice.
For CounterFlow, we modeled a simple checkout workflow:
CreateOrder
↓
ReserveInventory
↓
ChargePayment
↓
ConfirmOrder

The interesting failure scenario is during  ChargePayment .
We deliberately inject a failure after the payment side effect has already happened.
Conceptually:
Charge ₹999
↓
Payment committed
↓
Acknowledgement lost
↓
Workflow sees failure
↓
Retry

This is very different from failing before the charge had happened.
The side effect already existed.



Designing the Buggy experiment

We want this failure to be deterministic, not random.
So CounterFlow defines a fault plan similar to:
Type: AFTER_SIDE_EFFECT_TIMEOUT
Target: ChargePayment
Attempt: 1

On the first attempt:
PaymentCharged
↓
InjectedFailure

The workflow believes the operation failed and will retry.
Our intentionally Buggy implementation will generate a new payment identity on retry.
The result will be:
Attempt 1 → PaymentCharged
Attempt 1 → InjectedFailure
Attempt 2 → PaymentCharged

From the workflow engine's perspecitve, the retry eventually succeeded.
From the customer's perspective, something much worse happened.
They were charged twice.



Why AWS Step Functions was central to the experiment

AWS Step Functions orchestrates the checkout.
This was important, since we did not want to build a frontend that pretended a retry occurred.
The retry happens in the real serverless workflow.
Step Functions executes:
CreateOrder
→ ReserveInventory
→ ChargePayment
→ ConfirmOrder

and its retry behavior reproduces the exact condition we wanted to study.
This also taught us an important lesson:
Orchestration engines are great at answering "Did the workflow finish?"
They do not automatically answer "Did the business remain correct?"
That second question requires knowledge of the business rules.
That is where CounterFlow's invariant checker comes in.



Turning a business rule into an invariant

Our payment rule is simple:
An order must never have more than one committed charge.
We represent that as:
ChargeAtMostOnce

For an active order:
committedCharges <= 1

And once the order is confirmed:
committedCharges == 1

This check is deterministic application logic.
We deliberately did not use an LLM to decide whether a run passed or failed.
Correctness should not depend on interpretation when a business rule can be stated precisely.
In the Buggy execution, CounterFlow observe:
1 ORDER → 2 CHARGES

and produce:
Workflow: COMPLETED
Invariant: VIOLATED

That became one of the most important UI decisions in the project.
We deliberately show the execution state and business correctness separately.
Because they are not the same thing.



Finding where correctness first breaks

Simply stating:
FAILED

was not enough.
We wanted CounterFlow to answer a much more useful question:
What is the first event after which the system becomes incorrect?
So we introduced the concept of a first failing prefix.
Consider this trace:
CreateOrder
ReserveInventory
PaymentCharged
InjectedFailure
PaymentCharged
ConfirmOrder

CounterFlow evaluates the invariant, on the trace as it grows.
After the first payment:
charges = 1
PASS

After the injected failure:
charges = 1
PASS

After the retry commits another payment:
charges = 2
FAIL

Therefore the second  PaymentCharged  event became the first violating event.
Rather than forcing an engineer to inspect raw logs, CounterFlow highlights precisely where the system had crossed from valid to invalid state.
This changed the way we thought about observability.
Logs tell you what happened.
Correctness evidence should tell you when what happened became wrong.



Building the fix with DynamoDB idempotency

The fix was not to remove retries.
Retries are useful.
The fix was to make the payment operation safe to retry.
In our Fixed implementation, we use an order-scoped payment identity, plus an Amazon DynamoDB conditional write.
The first attempt creates the payment.
When the retry occurs, the application detects that the logical payment already exists.
Instead of producing another charge:
PaymentCharged

the retry produces:
PaymentReused

The Fixed trace becomes:
Attempt 1 → PaymentCharged
Attempt 1 → InjectedFailure
Attempt 2 → PaymentReused

The retry still existed.
The duplicate side effect did not.
The final result:
1 ORDER → 1 CHARGE
Expected: ₹999
Charged: ₹999
Overcharge: ₹0
ChargeAtMostOnce: PASSED

This was one of the clearest lessons of the project:
Idempotency does not stop retries.
It makes retries safe.



Making the comparison fair

Once we had Buggy and Fixed implementations, another problem arose.
If we test them using different failures, the comparison is not very strong.
So CounterFlow stores the resolved fault-plan snapshot used for the original run.
We also calculate a SHA-256 hash for that stored plan.
When the user selects Compare with Fixed, CounterFlow does not resolve a new fault configuration.
It replays the exact stored fault.
Conceptually:
Same workflow scenario
Same fault type
Same target
Same attempt
Same stored fault snapshot
Only the implementation changes

That gives us a much stronger experiment.
In our final acceptance run:
| | Buggy | Fixed |
| ------------------- | -----: | -----: |
| Payment attempts | 2 | 2 |
| Committed charges | 2 | 1 |
| Invariant | FAILED | PASSED |
| Customer overcharge | ₹999 | ₹0 |
The failure remained constant.
The implementation changed.
This is what made the result convincing.



How AWS services fit together

CounterFlow is built as a serverless AWS application.

AWS Step Functions

Step Functions orchestrates the checkout workflow, and performs the retry behavior that makes our experiment possible.

AWS Lambda

We used Lambda functions for the individual workflow operations:
CreateOrder
ReserveInventory
ChargePayment
ConfirmOrder

We also used Lambda for the Control API and invariant-checking logic.

Amazon DynamoDB

DynamoDB stores:
simulated business state,
payment records,
trace evidence,
run metadata,
stored fault information.
It also provides the conditional-write behavior used in our idempotent Fixed payment flow.

Amazon API Gateway

API Gateway exposes the CounterFlow APIs used by the frontend, including operations for:
POST /runs
GET /runs/{runId}
GET /runs/{runId}/trace
POST /runs/{runId}/compare


Amazon CloudWatch

CloudWatch helped us inspect Lambda behavior and debug the deployed workflow.

Amazon S3

The React frontend is deployed using S3 static website hosting.

AWS SAM and CloudFormation

The infrastructure is defined as code using AWS SAM.
Our Lambda functions, DynamoDB table, API Gateway configuration, Step Functions state machine, permissions, environment configuration, and other infrastructure can all be deployed reproducibly from the project configuration.
This became especially useful during a four-day hackathon where the infrastructure was changing often.



What deployment taught us

One of the big surprises was that getting the application working locally was only part of the job.
Our permanent deployment unearthed issues that didn't appear in local development.

CORS

Our browser needed to send a custom:
X-Demo-Token

header.
API Gateway therefore needed that header explicitly allowed.
The API could work perfectly from a direct request, but would fail from a browser.

Authenticated polling

The frontend starts a workflow and then polls for its status.
We discovered that authentication was not only needed when the run was created; the polling requests also needed to include the token.

Browser compatibility

Our frontend originally used:
crypto.randomUUID()

for comparison request tokens.
The S3 HTTP deployment did not provide the exact browser environment that we had assumed, so we added a compatible fallback.
These were all small problems individually, but they reinforced an important lesson:
Deployment is part of testing.
A system is not complete simply because its code runs locally.
The browser, API, permissions, cloud infrastructure, and workflow all need to work together.



What we liked about AWS

The strongest part of the AWS experience was how naturally the services mapped to the architecture.
Step Functions took care of orchestration.
Lambda handled compute.
DynamoDB took care of state and idempotency.
API Gateway exposed the system.
SAM gave us a chance to define everything as code.
For a small team building quickly, that meant that we could create a distributed workflow without managing servers.
Step Functions was particularly valuable as retry behavior was visible and configurable, rather than buried inside of application code.
DynamoDB conditional writes helped demonstrate an abstract concept - idempotency - directly.



What we would improve

The biggest challenge was debugging across service boundaries.
When one browser action touches:
API Gateway
→ Lambda
→ Step Functions
→ multiple Lambdas
→ DynamoDB
→ CloudWatch

Finding the exact failure can require jumping between several tools.
A more unified correlation view across API requests, Step Functions executions, Lambda logs, and application-level run IDs, would have made debugging serverless applications much easier.
We also found CORS configuration easy to get subtly wrong.
Better validation for browser-facing API Gateway configurations would help teams detect those problems before deployment.



What four days changed for us

We started CounterFlow thinking primarily about retry behavior.
We finished thinking much more deeply about correctness.
The most important lesson we learned was:
A successful retry is not necessarily a correct recovery.
This changed the way we think about distributed systems.
We also learned that good observability is not only infrastructure metrics.
Sometimes the most useful signal is a business invariant like:
A customer must not be charged twice.

And we learned that if you want to prove a fix, reproducing the same failure conditions matters just as much as showing that the new implementation passes.



The final outcome

CounterFlow's Buggy implementation demonstrates:
Workflow: COMPLETED
Invariant: VIOLATED
1 ORDER → 2 CHARGES
Expected: ₹999
Charged: ₹1,998
Overcharge: ₹999

The Fixed implementation receives the same fault, but produces:
Workflow: COMPLETED
Invariant: PRESERVED
1 ORDER → 1 CHARGE
Expected: ₹999
Charged: ₹999
Overcharge: ₹0

The retry stayed.
The duplicate charge did not.
And that is ultimately what CounterFlow is about:
Breaking workflows intentionally, capturing evidence of what happens, and proving that the fix survives the same failure before production has to discover it for us.



Project links

GitHub:
 https://github.com/devaganesh-source/counterflow 
Live CounterFlow deployment:
 http://counterflow-demo-web-618042349623.s3-website-us-east-1.amazonaws.com 
Built with: AWS Step Functions, AWS Lambda, Amazon DynamoDB, Amazon API Gateway, Amazon CloudWatch, Amazon S3, AWS IAM, AWS SAM, and AWS CloudFormation.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article