There is a good chance your multi-tenant agent platform is correct. You isolated tenant data, you tagged requests with a tenant ID, you put a rate limit at the edge, you externalized per-tenant behaviour into configuration instead of forking code. Those are the right moves, they're well documented, and at a hundred tenants they work.
They keep working right up until they don't — and the failures, when they arrive, share a family resemblance. They are not agent problems. They are distributed-systems problems: capacity as an invisible, account-scoped shared resource; fairness for work whose cost you can't price; degradation for state you can't afford to drop; a control plane nobody decided to build. AWS solved every one of these for its own services, and wrote the answers down publicly in the Builders' Library and Well-Architected.
This series is about the gap between those two bodies of knowledge: the published multi-tenant agent guidance that makes you correct at 100 tenants, and the published distributed-systems guidance that makes you survivable at 10,000. Five posts, one failure mode each, each pairing what breaks with the pattern that fixes it. We have watched teams cross that gap the expensive way; this is the cheaper route.
The clearest version of this looked like a good week. A platform team had just onboarded their largest customer to date, and the graphs said everything was fine: CPU flat, no throttles, autoscaling barely awake. What they didn't have a graph for was concurrent sessions — and one customer's overnight batch integration had quietly taken most of them. New users couldn't start a conversation. Existing ones were unaffected, which is why it took nine minutes to notice and another forty to believe.
Nothing in that story is about agents. It's a capacity model denominated in the wrong unit, a shared resource with no fairness policy, and a failure that was invisible in steady state. Those are distributed-systems problems, and the answers are already written down.
What makes this different from generic multi-tenant scaling
Classic multi-tenant scaling rests on three assumptions: requests have roughly bounded cost, work can be retried cheaply, and behaviour changes only when code deploys. Agent workloads break all three. That's the whole series in one table.
The assumption | Why agents break it | Post |
|---|
Capacity is throughput | Sessions are long-lived, stateful and hold fixed compute; concurrency, not RPS, is what runs out — and quotas are flat per-account ceilings | 1 |
You can price work at admission | Tokens, tool calls and reasoning turns are only known after the run; check-then-allow has nothing to check | 2 |
The noisy neighbour is another tenant | A runaway loop or prompt-injected agent spends on a tenant's behalf — the offending traffic never crosses your edge | 2 |
Rejected work is cheap to redo | Shedding a session destroys accumulated reasoning, tool results and workspace — and the retry costs more than what you shed | 3 |
Config is settings | Prompts, model IDs, tool schemas and guardrails are the behaviour, and they're read on every session start | 4, 5 |
Deploys are code | A prompt edit changes what every agent does, without passing through your pipeline or tripping your alarms | 5 |
The five posts
1. Your agent platform scales in sessions, not requests. Your capacity model is denominated in the wrong unit. Concurrency is arrival rate times duration, quotas are flat per-account ceilings, and rearranging components inside one account creates nothing. Make the account-sized cell your unit of scale, size it by whichever quota binds first, and scale-test one cell to reason about the fleet.
2. Admitting work you can't price. No managed component allocates capacity across your tenants, and for agents the classic check-then-allow model can't work. Admit on a forecast per request class, reconcile against actuals, hold reservations in one request-scoped context so rejections unwind cleanly — and budget the delegation tree, not just the request.
3. Park, don't shed. The mature response to saturation isn't dropping work, it's suspending it. Degrade down an explicit ladder — serve, queue, park, shed — with checkpoints you've actually restored and resumes that are bounded. And treat session churn as a capacity problem, because a cold start inside a tight deadline is the deadline.
4. Your prompt store is a control plane. If the serving path reads it, it's a control plane — and control planes fail more often than the data planes they configure. The test is concrete: can a new session start while your config service is down? Serve behaviour from per-cell snapshots, let the smaller service poll, do constant work.
5. A bad prompt is a bad deploy. Cells give you independent failure domains for compute; a global prompt table gives you one blast radius for behaviour. Behaviour changes ride the same staged waves, bake gates and automatic rollback as code — with gates on distributions rather than error rates, because a bad prompt never throws a stack trace.
How to read the series
Each post is self-contained and follows the same shape: the failure towards 10,000, why the published pattern stops working, how builders at extreme scale solve it, what agents add on top, and three to five design rules you can apply. The rules are the point — everything else is justification. Where a pattern costs something, I say so: none of these are free, and the tradeoffs matter as much as the mechanisms.
Every source I cite is public. Where a number is involved — a quota, a limit, a default — it's linked to documentation and dated, because those move and quotas are core at scale.
That said, let's start with the first bottleneck, agent capacity in part 1 . 
Loading article