A million agents, 100 units, zero oversells: engineering around Aurora DSQL's optimistic concurrency
I pointed one million AI agents at 100 units of inventory on Aurora DSQL. Exactly 100 sold – zero oversold, zero double-charged, zero errors, a ledger balanced to the penny. The catch: getting there meant un-learning everything a lock-based database taught me. A war-story write-up on engineering around DSQL's optimistic concurrency (SQLSTATE 40001) – sharding inventory and budget so nothing is a hot row, the bug that capped me at 2 commits/sec, and proving correctness with one SQL query.
I pointed one million AI agents at 100 units of inventory.
Exactly 100 sold. Zero oversold. Zero double-charged. Zero errors. A double-entry ledger that
balanced to the penny.
balanced to the penny.
The interesting part isn't that number — it's that getting there on Aurora DSQL meant un-learning
almost everything a lock-based database had taught me. This is the engineering write-up: the war
story, the bug that only showed up at scale, and why DSQL is the right database for transactions under
a high-contention agent swarm.
almost everything a lock-based database had taught me. This is the engineering write-up: the war
story, the bug that only showed up at scale, and why DSQL is the right database for transactions under
a high-contention agent swarm.
I built ZeroRace for the #H0Hackathon (Hack the Zero Stack with Vercel v0 + AWS Databases) —
an agent-safe checkout kernel on Aurora DSQL (truth core) + DynamoDB (read/event plane),
deployed on Vercel via the AWS Marketplace integration with OIDC (no static credentials).
Live: https://zerorace.vercel.app · Code: https://github.com/kaviyakumar23/zerorace
an agent-safe checkout kernel on Aurora DSQL (truth core) + DynamoDB (read/event plane),
deployed on Vercel via the AWS Marketplace integration with OIDC (no static credentials).
Live: https://zerorace.vercel.app · Code: https://github.com/kaviyakumar23/zerorace
What you'll get out of this:
- Why DSQL's optimistic concurrency makes the naive design worse — and forces a much better one
- How a single hot row capped me at 2 commits/sec, and the fix that unlocked a million
- Why correctness here is a
CHECKconstraint and aSELECT, not application logic you have to trust - The DSQL constraints (40001, no
FOREIGN KEY, async indexes, noTRUNCATE) and how each shaped the schema
First, why this is a database problem at all
By 2026, agents can shop and pay — ChatGPT Instant Checkout, Mastercard Agent Pay, Visa Intelligent
Commerce. Everyone's building the agent's identity and payment credentials. Almost nobody is
fixing the thing underneath: merchant backends were never built for software that buys at machine
speed. Agents retry hard, run in parallel, and hit you from multiple regions at once. At that speed,
naive checkout oversells the last unit, double-charges on a timeout-and-retry, and drifts past spend
limits.
Commerce. Everyone's building the agent's identity and payment credentials. Almost nobody is
fixing the thing underneath: merchant backends were never built for software that buys at machine
speed. Agents retry hard, run in parallel, and hit you from multiple regions at once. At that speed,
naive checkout oversells the last unit, double-charges on a timeout-and-retry, and drifts past spend
limits.
That's not an AI problem. It's a concurrency and consistency problem — exactly the kind of thing a
database is supposed to own. So I built the correctness layer as a database product, on the one engine
whose model actually fits it.
database is supposed to own. So I built the correctness layer as a database product, on the one engine
whose model actually fits it.
Start with the database, because the database is the design
Aurora DSQL is PostgreSQL-compatible, but it does not behave like single-writer Postgres. It uses
optimistic concurrency control (OCC) with snapshot isolation: no row locks. It lets transactions
run and detects write conflicts at commit time, returning
wrote the same row from the same snapshot? One commits; the other gets 40001 and has to retry.
optimistic concurrency control (OCC) with snapshot isolation: no row locks. It lets transactions
run and detects write conflicts at commit time, returning
SQLSTATE 40001. Two transactions thatwrote the same row from the same snapshot? One commits; the other gets 40001 and has to retry.
And AWS is blunt in the docs: don't concentrate writes on a single key. Spread them across random
keys.
keys.
That one sentence dictated the entire architecture. Bring a lock-based mental model to DSQL and the
obvious design gets worse under load. Design for OCC and it holds correctness at a million agents
— and gets faster as concurrency climbs.
Here's how that played out.
The product, in one transaction
ZeroRace is a checkout kernel.
that atomically: (1) checks an idempotency registry, (2) claims a unit of sharded inventory, (3) debits
a sharded spend mandate, (4) writes a balanced double-entry ledger, (5) emits a transactional outbox
row. All of it commits, or none of it does.
POST /v1/purchases/commit is one idempotent ACID transactionthat atomically: (1) checks an idempotency registry, (2) claims a unit of sharded inventory, (3) debits
a sharded spend mandate, (4) writes a balanced double-entry ledger, (5) emits a transactional outbox
row. All of it commits, or none of it does.
No saga. No two-phase dance. No compensating writes. And correctness is structural — a
constraint plus a conditional
CHECKconstraint plus a conditional
UPDATE … WHERE row-count, not advisory logic I have to trust.
The split that defines the system — Aurora DSQL is the strongly-consistent truth core; DynamoDB is the read/event plane, fed by a transactional outbox.
Designing out the obvious hot row
A single
agents decrementing one row is a 40001 conflict storm. So the 100 units are pre-split across 64
random-keyed
inventory(available_qty = 100) row is exactly the hot key DSQL warns about — a millionagents decrementing one row is a 40001 conflict storm. So the 100 units are pre-split across 64
random-keyed
inventory_buckets, and the claim shuffles candidates and conditionally decrements:1
2
3
UPDATE inventory_buckets
SET available_qty = available_qty - $qty
WHERE bucket_id = $id AND available_qty >= $qty; -- the DB refuses to go negative: no oversell, everNo oversell is structural:
impossible. A 40001 just means "retry, pick another bucket." Idempotency is keyed on
snapshot instead of double-committing.
CHECK (available_qty >= 0) plus that predicate make unit 101 of 100impossible. A 40001 just means "retry, pick another bucket." Idempotency is keyed on
(merchant_id, idempotency_key) under a UNIQUE index, so aggressive retries replay a storedsnapshot instead of double-committing.
Felt complete. Then I ran the swarm.
The number that hurt
First honest run — 10,000 agents vs 100 units:
1
committed 100 · commits/sec 2.0 · conflict-exhausted 303100 sold, zero oversold — correct. But 2 commits per second, and 303 agents that exhausted
their retries? For a system whose whole pitch is "engineered for DSQL," that's a tell. I'd sharded
inventory and left a different hot row — every winning commit still did this:
their retries? For a system whose whole pitch is "engineered for DSQL," that's a tell. I'd sharded
inventory and left a different hot row — every winning commit still did this:
1
UPDATE mandates SET remaining_minor = remaining_minor - $amt WHERE mandate_id = $id AND ...;One mandate row, updated by all 100 winning commits. Under OCC those serialize — each conflicts
with the last, retries, the backoff piles up, and the unlucky tail gives up. I'd removed the inventory
hot row and walked straight into a budget hot row.
with the last, retries, the backoff piles up, and the unlucky tail gives up. I'd removed the inventory
hot row and walked straight into a budget hot row.
The fix: shard the money too
Same trick, applied to dollars. A
keys. Mandate validity (active? in window?) stays a single read — reads never conflict under OCC.
Only the spend draws from a random budget shard, with a conditional decrement identical to inventory:
mandate_buckets table splits the spend limit across randomkeys. Mandate validity (active? in window?) stays a single read — reads never conflict under OCC.
Only the spend draws from a random budget shard, with a conditional decrement identical to inventory:
1
2
3
UPDATE mandate_buckets
SET remaining_minor = remaining_minor - $amt
WHERE bucket_id = $id AND remaining_minor >= $amt;
The whole fix in one picture: stop concentrating writes on a single key.
Now there's no hot row anywhere — not in stock, not in budget. A scaling sweep confirmed correctness
holds as contention climbs, and it actually runs faster at higher concurrency as the sold-out
attempts drain in parallel:
holds as contention climbs, and it actually runs faster at higher concurrency as the sold-out
attempts drain in parallel:
| concurrency | committed | OCC retries | conflict-exhausted |
|---|---|---|---|
| 20 | 100 | 148 | 0 |
| 50 | 100 | 314 | 0 |
| 100 | 100 | 323 | 0 |
| 200 | 100 | 330 | 0 |
Scaling sweep chart: committed stays flat at exactly 100 across every concurrency level while OCC retries rise with contention and conflict-exhausted stays at zero
A non-zero retry count isn't a flaw to hide — it's the proof you designed for DSQL's model.
Then I ran a million
The obvious question for a database crowd: does it hold at scale? I re-seeded to 100 units and fired
1,000,000 agents through the same kernel — a 10,000-to-1 contention ratio:
1,000,000 agents through the same kernel — a 10,000-to-1 contention ratio:
1
2
attempts 1,000,000 · committed 100 · out-of-stock 999,900
OCC retries 585 · conflict-exhausted 0 · errors 0Exactly 100 sold. Zero oversold. Zero errors across a million real commit calls.
The tell is the retry count: 585 — basically the same ~530 I saw at 25,000. Contention is
front-loaded. It happens among the ~100 winners racing for buckets while stock drains; after that,
every remaining attempt is a clean, non-conflicting decline.
front-loaded. It happens among the ~100 winners racing for buckets while stock drains; after that,
every remaining attempt is a clean, non-conflicting decline.
OCC retries track the 100 commits, not the swarm size. The design doesn't degrade as the horde
grows — it just declines the losers faster. That's the whole payoff of sharding for OCC.
Correctness proven, not claimed
Here's the part I trust most. Every guarantee reduces to one read-only query against the live
cluster, wrapped as
cluster, wrapped as
pnpm prove:1
2
3
4
oversells = 0
duplicate_settlements = 0
ledger_drift_minor = 0
mandate_breaches = 0
Animated terminal: pnpm prove runs against the live Aurora DSQL cluster and prints oversells 0, duplicate_settlements 0, ledger_drift_minor 0, mandate_breaches 0
Not dashboard counters — a
negative (oversell), any idempotency key bound to two settlements (dupe), any
limit (breach). Clone the repo and run it yourself. That's the difference between "trust me" and
correctness you can check.
SELECT over the actual rows: a HAVING that finds any bucket drivennegative (oversell), any idempotency key bound to two settlements (dupe), any
SUM(signed_amount_minor) <> 0 over the double-entry ledger (drift), and any mandate spent past itslimit (breach). Clone the repo and run it yourself. That's the difference between "trust me" and
correctness you can check.
Three things DSQL taught me the hard way
No
in
per-txn row cap. (My million-agent run wrote ~2M rows; cleaning it up was batched deletes, in chunks.)
TRUNCATE, and a per-transaction row limit. Re-seeding between runs blew up: declines pile upin
purchase_intents, and a single DELETE FROM touches them all in one transaction — past DSQL'sper-txn row cap. (My million-agent run wrote ~2M rows; cleaning it up was batched deletes, in chunks.)
Async indexes are a correctness gate, not a perf detail. The UNIQUE indexes on the idempotency keys
are what structurally guarantee dedupe. DSQL builds indexes asynchronously (
you must gate on
commit.
are what structurally guarantee dedupe. DSQL builds indexes asynchronously (
CREATE INDEX ASYNC), soyou must gate on
ACTIVE before any traffic — otherwise two first-time same-key requests can bothcommit.
No
(cross-tenant access returns
FOREIGN KEYs. Referential integrity — and tenant isolation in the multi-tenant layer(cross-tenant access returns
403) — lives in the service layer plus periodic audit queries.There was also a deadlock my own adversarial review caught before it ever hit the demo: on a
unique-violation race the kernel re-read the idempotency registry on a second pooled connection while
holding the first. Under same-key contention with N ≥ pool size, every slot waits on a connection that
never frees. Fixed by reusing the held connection — a bug that would only have surfaced live, on stage.
unique-violation race the kernel re-read the idempotency registry on a second pooled connection while
holding the first. Under same-key contention with N ≥ pool size, every slot waits on a connection that
never frees. Fixed by reusing the held connection — a bug that would only have surfaced live, on stage.
"Why not single-writer Postgres?" — two regions, one ledger
This is the question a database audience asks, and DSQL is the honest answer. I peered a real
multi-region cluster — Tokyo
logical database with two strongly-consistent write endpoints. A write committed through Tokyo is
instantly readable through Seoul. Strong, not eventual.
multi-region cluster — Tokyo
ap-northeast-1 + Seoul ap-northeast-2, witness in Osaka — onelogical database with two strongly-consistent write endpoints. A write committed through Tokyo is
instantly readable through Seoul. Strong, not eventual.
Then I killed a region mid-swarm. Failover Tokyo → Seoul split settlements 10 on Tokyo / 90 on
Seoul, total exactly 100, 0 oversold — and reconciliation still passed every invariant, both
endpoints folded into one consistent ledger. A single-writer Postgres cannot serve correct concurrent
writes from two active regions. That is the structural reason DSQL is the rail, not a buzzword.
Seoul, total exactly 100, 0 oversold — and reconciliation still passed every invariant, both
endpoints folded into one consistent ledger. A single-writer Postgres cannot serve correct concurrent
writes from two active regions. That is the structural reason DSQL is the rail, not a buzzword.
Multi-region failover: Tokyo and Seoul as two active-active write endpoints with an Osaka witness; Tokyo fails mid-swarm, traffic shifts to Seoul, settlements split 10 / 90, total exactly 100, zero oversold
The AWS + Vercel wiring, briefly
Deployment uses the AWS Marketplace integration with OIDC federation: Vercel issues an OIDC token,
the app trades it for short-lived AWS credentials via STS, and the DSQL node-postgres connector mints
time-bound IAM auth tokens from those — no password or static key anywhere in the codebase. (One
real gotcha: under a burst, many fresh connections each resolving credentials at once triggered a
the app trades it for short-lived AWS credentials via STS, and the DSQL node-postgres connector mints
time-bound IAM auth tokens from those — no password or static key anywhere in the codebase. (One
real gotcha: under a burst, many fresh connections each resolving credentials at once triggered a
getaddrinfo EBUSY DNS storm against STS. Fix: memoize the OIDC credentials process-wide.)Multi-tenancy comes (almost) for free
Because correctness lives in the schema and the kernel, extending it to many tenants is mostly a tenant_id on ownership — referential integrity and isolation enforced in the service layer (no FOREIGN KEYs). The same query that proves one merchant's invariants proves all of them at once. That's the argument for building on a database instead of around one.
The self-serve console and playground: create your own merchants, mandates, and sharded inventory, then drive your own scenario and watch the four invariants hold at zero
Key takeaways
- DSQL makes you design contention out instead of locking it away. "Don't concentrate writes on a single key" turned out to be the whole architecture, not a footnote.
- Shard the hot rows — all of them. I sharded inventory and still got bitten by a hot budget row. Find every single-key write under contention.
- Make correctness a
SELECT. If your invariants reduce to one query over the real rows, you can prove them instead of asserting them. - Surface OCC retries, don't hide them. A bounded, non-zero 40001 count is evidence you built for the engine — and it tracks your commits, not your traffic.
- Strong consistency across active-active regions is a capability you can't fake with replicas — and it's the honest answer to "why not single-writer Postgres."
The agentic-commerce wave needs a correct rail underneath it. That's a database problem, and Aurora DSQL is unusually well-suited to it.
Try it / reproduce it
The repo is open; the proof runs in a few commands against your own DSQL cluster (
pnpm demo:reset && pnpm swarm && pnpm prove).- Code: https://github.com/kaviyakumar23/zerorace
- Live demo: https://zerorace.vercel.app
Built for the #H0Hackathon (Hack the Zero Stack with Vercel v0 + AWS Databases). #H0Hackathon
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article