SAS01 Capacity: Your agent platform scales in sessions, not requests
Capacity planning built on requests per second will report headroom while the platform is already full. This walks the arithmetic that actually governs agent capacity, explains why service quotas are flat per-account ceilings that no amount of internal architecture can raise, and shows how to size an account-based cell by whichever quota binds first — then scale-test one cell to reason about the rest.
Series: Secrets of Agentic Scale (6 articles)
- 2SAS01 Capacity: Your agent platform scales in sessions, not requests This article
1. Capacity planning changes on the path to 10,000
The capacity plan was good. You measured throughput under load, found the platform handled 400 requests per second (RPS) comfortably, and sized for 250 with headroom to spare. Growth was projected, dashboards were built, autoscaling was configured. Then you onboarded the tenant that pushed you past 3,000 concurrent users, and the platform stopped accepting new work at roughly 80 RPS.
Nothing in your capacity model predicted 80. CPU was fine. Memory was fine. The autoscaling group was not even at half stretch. What you had actually run out of was sessions — and sessions are not something your RPS model had an opinion about.
The follow-up discovery was worse. Someone suggested splitting the workload across two runtimes to double capacity. It changed nothing: both runtimes drew from the same account-level session pool. Someone else suggested sharding the busiest tenants onto their own runtime. Also nothing. The ceiling wasn't in the architecture — it was in the account, and no amount of rearranging components inside that account produced a single additional session.
Then, a week later, a different wall at a different number, and this time it wasn't sessions at all. It was a quota on a downstream service nobody had inventoried, because it sat on the serving path without anyone deciding that it should.
2. Why the published pattern stops working
Two assumptions are load-bearing in almost every capacity plan, and agent platforms break both.
Assumption one: throughput is the unit of capacity. Requests per second is a good denominator when requests are short and stateless. It becomes meaningless when the thing you are serving is a session — long-lived, stateful, holding fixed compute for its entire life, whether that's ninety seconds or eight hours. What limits you is not how many requests arrive per second but how many sessions are alive at once, and that number is governed by Little's law: concurrency = arrival rate x duration. Three hundred new sessions a minute, each living ten minutes, is three thousand concurrent sessions — a number your RPS dashboard will never show you. Double the average session duration and you have doubled your capacity consumption without a single extra request.
Assumption two: architecture creates capacity. In a world of horizontally scalable stateless services, adding components adds throughput. But agent runtime quotas are flat per-account ceilings: session limits, tokens per minute, memory operations. Splitting one workload into three runtimes inside one account divides the same pool three ways; it does not enlarge it. This is why multi-account is a scaling decision and not only a security one — the account is where new capacity actually comes from ( cell-based architecture ).
There's a third, quieter problem: the binding constraint is rarely the one you're watching. An agent turn touches a session slot, then model tokens, then memory or checkpoint writes, then a fan-out of tool calls — each with its own account-scoped ceiling, several of them on services you think of as storage or configuration rather than as serving-path dependencies. Your real capacity is the first ceiling that binds, and the published guidance won't tell you which one it is because it depends on the shape of your traffic. Quota values also move — providers raise defaults — so any number you hard-code into a capacity model has a shelf life ( AgentCore quotas ).
At 100 tenants the binding ceiling is invisible, because you never get close enough to touch it. At 10,000 tenants it is the only number that matters — and we have watched more than one team add runtimes to a saturated account twice over before anyone opened the quota page.
Walk one agent turn end to end and the ceilings it touches are easy to list — and easy to under-inventory:
Step in the turn | Resource claimed | Scope of the ceiling | Usually inventoried? |
|---|---|---|---|
Start session | session slot | account | sometimes |
Reason | model tokens per minute | account / model | yes |
Checkpoint state | memory / storage writes | account | rarely — sits on the serving path |
Call tools | tool invocations, gateway auth | account / per-integration | rarely |
Delegate | 1 session per sub-agent | account (same pool) | almost never |
3. How XXL ISVs build for this
Builders who operate past a single account's ceilings converge on the same answer: make the capacity boundary an explicit architectural unit, then replicate it. AWS calls this cell-based architecture, and it's documented in detail. There are four moves, and none of them are free.
The cell is the unit of scale. A cell is a complete, independent instance of your stack, sized so that we can reason about it and test it as a whole. We scale by adding cells rather than by growing one ( What is a cell-based architecture? ). For agent platforms the natural cell boundary is the account, because that is where the quotas live: one account-sized cell is one capacity pool. What this costs is operational surface — every cell is another account to deploy to, monitor, patch and pay for — so cells are worth their overhead only once a single account genuinely constrains us.
Size the cell by its binding quota, then test one cell to destruction. This is the move that makes the whole approach tractable. AWS's own guidance for cellular services is to load-test a single cell to its limit and use that measured number to reason about the fleet, rather than attempting to test at full scale ( resilience and availability on Amazon ECS ). For an agent cell: measure sessions per tenant under real traffic, derive concurrent sessions from arrival rate × duration, then divide the cell's binding quota by that figure. We now have a defensible tenants-per-cell number — and a way to recompute it when quotas or traffic shape change. The caveat is that the number is only as good as the traffic we measured, so it needs revisiting as the product changes.
Place tenants with shuffle sharding. Assigning tenants to cells at random means any single tenant's overload strands only the tenants that share its cell, and no two tenants share the same combination of cells. This is the standard mitigation for noisy-neighbour blast radius ( shuffle sharding ), and it composes with the bulkhead thinking in Well-Architected ( REL10-BP04 ). The tradeoff is that tenant placement becomes a thing we own and occasionally have to change, and moving a tenant between cells means moving their state.
Keep the router thin and avoid global state. The cell-based guidance is explicit about minimizing anything shared across cells: a global tenant registry that every session must consult reintroduces the single point of failure cells were meant to remove. We route by tenant, keep placement data replicated per cell, and let cells operate independently — which does mean accepting that placement data is eventually consistent.
The payoff is what our cold open was missing. When a tenant floods their cell, that cell saturates and the other nine keep serving. Capacity growth becomes an operation we can plan — add a cell, gain a known number of tenants — instead of a wall we rediscover each time we reach it.
4. The agent delta
Cells and shuffle sharding are established practice. Agents change the arithmetic you put into them.
Sessions, not requests, are what you're partitioning. Cell sizing for a classic service starts from throughput. For an agent platform it starts from concurrency: arrival rate × session duration, against the account's session ceiling. That makes session duration a capacity lever — an idle-timeout policy is a capacity decision, and a feature that keeps sessions open longer consumes capacity even when nothing is happening.
A supervisor request is not one session — it's a tree. When an agent delegates, the request occupies its own session plus one for each sub-agent, and the supervisor's slot is held for as long as the longest branch runs. So a topology with a supervisor and four specialists can consume five session slots for the duration of the slowest one. Cell capacity has to be computed against the tree, not the request.
Splitting sub-agents across accounts doesn't relieve the orchestrator. It's tempting to give sub-agents their own account to escape the shared pool. That works for the sub-agents — and does nothing for the supervisor pool, which is the one that fills first because supervisor slots are held longest. Orchestrators need sharding across cells too.
Downstream quotas sit on the serving path. Because agents checkpoint state as they reason, memory and storage write limits are not background concerns — they're in the availability equation for a live turn. Inventory them as serving dependencies, not as data infrastructure.
5. Design rules
- Denominate capacity in concurrent sessions. Agent sessions are long-lived and stateful, with fixed per-session resources. Our sizing model is arrival rate × session duration (Little's law), not RPS — request math will tell us we have headroom while the session pool is exhausted.
- Treat the account as the unit of scale. Quotas are flat per-account ceilings; architecture inside one account creates no new capacity. Only account boundaries mint new pools — make the account-sized cell your scaling unit, and shuffle-shard tenants across cells so one saturated cell strands few tenants.
- Inventory every quota on the agent path. Walk one request end to end — session slot, model tokens-per-minute, memory writes, tool-call fan-out — and list each account-scoped ceiling it touches. Your real limit is the first one that binds, and it's usually a downstream quota nobody has looked at.
- Size a cell by its binding quota, then scale-test one cell. Derive a cell's tenant capacity from its session quota discounted by whichever downstream quota binds first, against measured sessions-per-tenant. Test one cell to destruction and reason about N — never load-test the fleet.
- Count the whole session tree. A supervisor request consumes 1+N sessions, and the supervisor's slot is held for the longest branch. Separate accounts relieve sub-agent pools but never the orchestrator's — shard orchestrators across cells too.
6. Key takeaways + next in series
- Denominate agent capacity in concurrent sessions — arrival rate × duration — not requests per second. Session duration is a capacity lever.
- Quotas are flat per-account ceilings: rearranging components inside an account creates no capacity. Make the account-sized cell your scaling unit and shuffle-shard tenants across cells.
- Inventory every account-scoped ceiling on the agent path, size the cell by whichever binds first, and scale-test one cell to reason about N.
Series: Secrets of Agentic Scale (6 articles)
- 2SAS01 Capacity: Your agent platform scales in sessions, not requests This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article