SAS02 Fairness: Admit on a forecast, reconcile on actuals
An agent request's cost — tokens, tool calls, sub-agent sessions — is only known after it runs, so the check-then-allow rate limiting that protects classic SaaS has nothing to check. Here is the fairness layer that works instead: per-tenant token buckets, size-class queues, a request-scoped context that unwinds cleanly on rejection, and budgets on the whole delegation tree rather than the request.
Series: Secrets of Agentic Scale (6 articles)
- …
- 3SAS02 Fairness: Admit on a forecast, reconcile on actuals This article
1. Fairness in the age of agents
The page arrives at 09:14 on a Monday: p99 latency has doubled across your agent platform, and not one request was throttled. Every rate-limit dashboard is green. Your tenants are not.
The postmortem finds a single tenant — not even a big one. Their workflow asked a supervisor agent to reconcile a quarter's invoices, and the supervisor did what supervisors do: it decomposed the task and delegated. Each sub-agent spawned its own session, made its own model calls, wrote its own checkpoints. One innocuous request fanned out into hundreds of sessions and burned through the shared token-per-minute quota that every other tenant's chat turns depend on. Your edge rate limiter counted it as exactly one request, well within the tenant's plan.
Three weeks later it happens again, differently: a tenant wires your platform into a batch integration and floods you with ten thousand legitimate requests overnight. Again, each request is individually fine. Again, everyone else pays.
Two different attackers — a flood of cheap requests, and one expensive request — and the same root cause: you admitted work whose cost you could not know, and you had no way to be fair about what happened after admission.
2. Why the published pattern stops working
The published pattern for multi-tenant fairness is admission control at the edge: per-tenant API keys, usage plans, rate and burst limits. It works because of an assumption so quiet you've probably never said it out loud — a request is a reasonable proxy for work. Count requests, and you've approximately counted cost.
Agent workloads break that assumption three ways.
First, cost is a post-hoc fact. An agent request's consumption — tokens, tool calls, reasoning turns, sub-agent sessions — is only known after the run completes. Two requests to the same endpoint can differ in cost by three orders of magnitude: a one-turn chat reply and a deep-research task look identical at the front door. Check-then-allow logic cannot price what it is checking.
Second, one request claims many pools, sequentially. After admission, an agent request consumes a session slot, then model tokens, then memory writes, then N tool invocations — each a separate quota, each claimed at a different moment, mostly downstream of your gate. A rate limiter at the edge guards one door of a building with five.
Third, the noisy neighbour can be your own code. Classic fairness protects tenants from each other's traffic. But a runaway reasoning loop, or a prompt-injected agent, spends on a tenant's behalf — the offending traffic never crosses your edge at all.
And the part that catches most teams out: no managed component allocates capacity across your tenants. The building blocks exist — fair queues, usage plans, per-tenant keys — but the allocation policy, the budgets, and the reconciliation are yours to build. The Well-Architected agentic AI lens says as much: tenant-aware isolation and throttling is your responsibility to implement ( AGENTPERF07-BP02 ).
At 100 tenants you can absorb an unfair request — there is slack, and someone notices before it spreads. At 10,000 tenants unfairness is your steady state, and we have yet to meet a platform at that size that didn't end up building this layer itself.
3. How XXL ISVs build for this
Builders operating at extreme multi-tenant scale converge on the same architecture, and AWS has documented it publicly across the Builders' Library and Well-Architected. There are four ideas we need, and each one costs us something worth knowing about up front.
Give every tenant its own token bucket — never a global one. The Builders' Library article on fairness in multi-tenant systems describes admission control with a token bucket per client key: rate plus burst, refilled continuously, checked at admission ( Fairness in multi-tenant systems ). A single shared bucket means one tenant can spend everyone's budget — the failure from our cold open. Per-tenant buckets turn a flood into that tenant's own problem. The cost is state: we now hold a bucket per tenant and have to keep it available and reasonably fresh across every node that admits work, which is a real distributed-systems problem at 10,000 tenants.
Queue fairly, not just quickly. When work must wait, a single FIFO queue recreates the problem one layer down: a burst from one tenant sits in front of everyone else. Amazon SQS fair queues address this noisy-neighbour pattern directly, interleaving per tenant so one backlog doesn't become everyone's latency ( SQS fair queues ). We also want a second dimension for size, because agent tasks span seconds to hours and a short chat turn queued behind a deep-research run will wait for it. Separate queues per request class fix that, at the expense of more plumbing to operate and a routing decision we now have to get right.
Admit on an estimate, reconcile on actuals. Since true cost is only known afterwards, we forecast it per request class — chat turn, tool-heavy task, deep research — admit against the forecast, then settle the tenant's budget with measured consumption once the run ends. Persistent over- or under-estimates feed back into the next forecast. The tradeoff is that our budgets are approximate in the moment: we are deliberately trading precision at admission for the ability to admit at all.
The forecast input is a request-class taxonomy:
Request class | Typical duration | Cost forecast basis | Queue |
|---|---|---|---|
Chat turn | seconds | tokens: narrow band, low variance | interactive |
Tool-heavy task | seconds–minutes | tokens + N tool calls | standard |
Deep research / delegation | minutes–hours | session tree: depth × fan-out × tokens | batch |
Hold reservations in one context; unwind on rejection. Because one request claims session, tokens, memory and tool budgets sequentially, a rejection at step four must release steps one through three. A request-scoped admission context — one object that tracks every reservation — makes the unwind mechanical instead of something we hand-roll at each call site. It does mean every admission decision now carries a little context object around with it, which is a modest amount of plumbing for a considerable amount of sleep.
Two operational notes complete the picture. Backoff is not overload protection: clients retrying with exponential backoff still pile up, so we cap retries with a retry budget and stop failure multiplying traffic ( Timeouts, retries, and backoff with jitter ). And we tune the layer with a two-sided signal: rising queue age means we are admitting too little; rising downstream throttle errors mean we are admitting too much. Healthy is the corridor between them, and finding it takes a few iterations against real traffic — this is not a set-and-forget layer.
4. The agent delta
If you've built multi-tenant SaaS before, the previous section reads like standard fairness engineering. The delta is what agents add on top.
Budgets have new dimensions. A tenant budget counted in requests or tokens misses how multi-agent systems actually spend. A supervisor that delegates consumes sessions (one per sub-agent, its own slot held for the longest branch), depth (each delegation is a full reasoning loop, not a millisecond tool call), fan-out (sub-agent counts multiply per level), and bytes (results streamed back up through the coordinator amplify payload size multiplicatively, against a request/response payload cap of 100 MB that is documented as non-adjustable — as of July 2026, see AgentCore quotas and limits ). Admission has to budget the delegation tree, not the request.
"Delegate or call a tool" is an admission decision. The same capability packaged as a tool call costs milliseconds; packaged as a sub-agent it costs a session, a context window, and an open-ended number of model calls. That choice belongs inside the admission model, not only in the prompt.
Guards must face inward. Per-tenant limits assume the tenant is the actor. When one of your own supervisors loops, or a prompt injection turns an agent against its budget, the spender is your code wearing a tenant's identity. Per-agent and per-loop budgets — iteration caps, per-branch token ceilings — are companions to tenant fairness rather than substitutes for it: an iteration cap is a runaway guard, not capacity management.
5. Design rules
- Admit on a forecast, reconcile on actuals. We cannot know an agent request's cost before it runs — tokens, tool calls and reasoning turns are post-hoc facts. Estimate per request class (chat turn, tool-heavy, deep research), admit against the estimate, and settle the difference against the tenant's budget when the run completes.
- Give every tenant its own bucket. Guard your own code with per-agent and per-loop budgets. Never a global bucket: it lets one tenant exhaust everyone. But per-tenant limits alone don't stop a runaway loop or a prompt-injected agent spending on a tenant's behalf — pair them with budgets on each agent and each iteration loop.
- Reserve through one admission context; unwind everything on rejection. One agent request claims a session slot, then tokens, then memory writes, then N tool calls. Track all reservations in a request-scoped context object so a failure at step 4 releases what steps 1–3 reserved — never scatter per-call checks that leak budget.
- Budget the delegation tree, not just the request. Depth, fan-out and response bytes are admission dimensions: every sub-agent multiplies sessions, tokens and payload size, and a supervisor's context window is a hard cap you will hit. Treat "delegate vs tool call" as an admission decision, not a prompt-engineering one.
- Schedule by size class, and tune with a two-sided signal. A 10-second chat turn queued behind an hours-long research run is head-of-line blocking that tenant fairness can't fix — separate queues per request class. Tune admission by watching both sides: queue age (too tight) vs downstream throttle errors (too loose).
If rate limiting and noisy neighbours in a pooled multi-tenant deployment are new to you, the SaaS Operations workshop covers that ground hands-on. This post assumes it and starts where it stops.
6. Key takeaways + next in series
- Agent cost is a post-hoc fact: admit on a forecast per request class, reconcile the tenant's budget on actuals — check-then-allow cannot work.
- Fairness needs per-tenant buckets and inward-facing guards: per-agent and per-loop budgets, plus depth, fan-out and byte limits on the delegation tree.
- Tune admission in the corridor between two signals: queue age (too tight) and downstream throttles (too loose).
Next in the series : long-lived sessions demand parking, not shedding — what graceful degradation means when dropping a request destroys an hour of accumulated reasoning.
Series: Secrets of Agentic Scale (6 articles)
- …
- 3SAS02 Fairness: Admit on a forecast, reconcile on actuals This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article