
Two billing counters, one bad connection, and the bug that erases a sale
How a hackathon inventory-sync engine uses vector clocks, a PN-Counter CRDT, and SQS FIFO ordering on AWS to guarantee two offline point-of-sale counters can never silently overwrite each other's sales — plus a real concurrency bug a dedicated hardening pass caught before launch.
The problem I kept seeing
Small shops in India increasingly run more than one billing counter — a
second register during rush hour, or an owner checking stock from their
own phone — sharing a single, often flaky internet connection. The
popular digital-ledger apps market "offline mode" and "multi-device
sync," but dig into what they actually mean and it's the safe case: edit
later, when you're back online, one device at a time. None of them
advertise the case that actually breaks small retail — two devices,
both offline at the same time, both editing the same shared number.
That case matters because the naive fix is actively dangerous. If both
counters queue their sales locally and just push a full snapshot on
reconnect, whichever one syncs last wins — and silently erases the
other counter's sales from the record. Nobody notices until the stock
count is wrong and a customer is told an item is in stock when it isn't.
That's not a networking problem. Retries and offline queues already
solve "the connection is unreliable." The actual problem is: two
independent devices changed the same number while neither could see the
other's change, and both need to reconnect to exactly one correct,
shared truth. That's a distributed-systems problem, and it has a
well-known, mathematically correct answer — I just hadn't had a reason
to actually build one before.
The "aha" — stock isn't a number, it's two counters
My first instinct was to treat stock as a single value protected by
optimistic locking. That falls apart the moment two counters are
concurrently offline: there's no "current version" either of them can
check against, because neither has seen the other's write.
The fix is a PN-Counter — a small, specific CRDT (conflict-free
replicated data type). Instead of storing one overwritable number, stock
is derived from two internal, grow-only counters:
stock = base_stock + total_increments − total_decrementsRestocks add to
increments , sales add to decrements , and each is
itself a map of { counter_id: amount } . Merging two counters' updates
is just addition — which means the merge is commutative and
associative by construction. It doesn't matter which counter
reconnects first, or in what order their individual sales arrive; the
final total is the same. I didn't just assert that — I wrote a
property-based test that throws arbitrary orderings of concurrent
increments/decrements at the merge function and checks the total is
identical every time, which is a meaningfully stronger claim than an
example-based unit test with one hand-picked order.Not every field needs this level of rigor, though — and knowing which
ones do is its own lesson. A price or a shelf-location isn't count-like;
it's replaced, not accumulated. For those, vector clocks determine
whether an incoming write cleanly follows the current state or is
genuinely concurrent with it, and disjoint field edits (price vs. shelf
location) merge without conflict automatically. The one case that gets
no automatic resolution is two concurrent writes to the same field
with different values — a real price disagreement. There, the system
never guesses. It flags the item
needs_review , keeps both values
visible, and — the part I was most unsure would actually help — calls
Amazon Bedrock to generate a short, plain-language explanation of
the disagreement for whoever resolves it. Bedrock is deliberately kept
advisory-only: it explains, it never decides.The architecture, and why each piece is load-bearing
A write makes one trip down this pipeline, and the live push back to
every open counter is a separate hop off the same event:
- StockSync Counter (React, IndexedDB offline queue) sends
POST /transactions— queued locally first, so a flaky connection never blocks a sale.
- API Gateway (REST) receives it and hands off to Lambda.
- Lambda
write-intakedoes the idempotency check, then enqueues.
- SQS FIFO, grouped by
MessageGroupId = item_id— not by which counter sent it — so every write touching one item is processed in strict order, no matter how many counters are concurrently active.
- Lambda
conflict-resolverdoes the actual merge: vector-clock comparison, PN-Counter arithmetic for stock, field-level merge for everything else. It commits the record, its audit-log entry, and its idempotency claim together viaTransactWriteItems, and — only on a genuine same-field conflict — asks Amazon Bedrock for a plain-language explanation, non-blocking.
- Amazon DynamoDB is the single source of truth the whole pipeline converges on.
- A DynamoDB Streams event off that same write fans out through
API Gateway's WebSocket API as a live push (
WSS) back to every other open counter — so a second screen sees the new stock count arrive without ever polling for it.
The detail that matters most and is easiest to get wrong: the SQS FIFO
queue's
MessageGroupId is the inventory item's ID, not the sending
counter's ID. Group by counter instead, and two counters editing the
same item race each other in parallel Lambda invocations — which
reintroduces the exact bug this project exists to prevent. Grouping by
the affected resource, not the sender, is what guarantees every write
touching one item is processed strictly in order, regardless of how many
counters are concurrently active.The record update, its audit-log entry, and its idempotency claim commit
together via
TransactWriteItems , so a crash mid-write can never leave
the audit trail silently missing what actually happened to the stock
count — a real bug I hit and fixed with an earlier draft that used
sequential PutItem calls instead.A bug a hardening pass actually caught
Late in the build, I ran a dedicated edge-case pass against a 25-item
catalog of concurrency, network, and security scenarios I'd written down
during design. One of them caught a real bug: the write-intake Lambda
processed a client's batch of queued offline transactions with
Promise.all , which is fine across different items but wrong when two
transactions in the same batch target the same item — their
SendMessageCommand calls could race each other and land in SQS out of
the client's original order, silently defeating the exact ordering
guarantee the architecture is built around. The fix was small — group a
batch's transactions by item and process each item's transactions
sequentially, while still parallelizing across different items — but
finding it required treating "we handle concurrency correctly" as a
claim to actively try to break, not something to assume once the happy
path demo works.What I'd tell someone starting this
If two clients can genuinely edit the same shared value while both are
offline, reach for a CRDT before reaching for locking or last-write-wins
— optimistic concurrency control assumes someone has seen someone else's
current version, and that assumption is exactly what's false in the
offline-and-concurrent case. And build the "break it on purpose" pass
into the plan from day one, not as an afterthought after the demo
works — the bug it caught here was invisible in every rehearsal until I
went looking for it deliberately.
The bigger shape of the problem
I didn't set out to build a lending product, but building the audit
trail correctly produced something adjacent to one almost for free: a
timestamped, tamper-evident record of a shop's actual sales activity.
Most small Indian retailers have no such record today — they run on
cash and memory — which is a real part of why formal credit is so hard
for them to access. I'm not claiming StockSync does anything with that
now. But a shop's StockSync history would be a far more honest input to
a creditworthiness decision than anything most lenders can see about
that shop today, and that's worth having in view even at hackathon
scale.
Full source, architecture docs, and the edge-case catalog are in the
repo: see the README for the complete breakdown and demo
video link.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article