SAS04 Your prompt store is a control plane
Externalising tenant behaviour into configuration is good advice that quietly creates a control plane — one that fails more often than the data plane it configures, and whose outage stays invisible until the next session tries to start. This covers the static-stability test for agent platforms, serving behaviour from per-cell snapshots, inverting the call direction so cold starts create no fan-in, and reconciling with constant work.
Series: Secrets of Agentic Scale (6 articles)
- …
- 5SAS04 Your prompt store is a control plane This article
1. Configuration scales to control plane over time
The service that went down was not on anyone's critical-path diagram. It was the tenant settings API — a small internal service, one team, modest traffic, the place where prompts, model preferences, tool entitlements and per-tenant limits were stored and served. It had never had an incident worth a postmortem.
At 14:20 it started returning 500s. Existing agent sessions kept running: they had fetched their prompts and resolved their tools at startup and held everything in memory. For nine minutes, the platform looked fine.
Then the sessions started finishing. And no new session could start, because starting a session meant fetching a prompt, resolving a tool list and reading an entitlement — three calls to a service that was down. Every new user request failed at initialization. The dashboards told a strange story: utilization falling, error rate climbing, capacity apparently abundant and completely unusable.
The recovery attempt made it worse. When the settings service came back, ten thousand sessions tried to cold-start at once, all fetching config simultaneously. The service fell over again under the stampede it had caused by being unavailable.
Nobody on that team had ever said "we are building a control plane." They had built one anyway, and put it directly in the serving path.
2. Why the published pattern stops working
The published multi-tenant pattern tells you to externalize tenant-specific behaviour into configuration: don't fork code per tenant, store their settings and read them at runtime. That advice is correct, and it quietly creates a control plane.
The distinction matters because control planes and data planes have different reliability characteristics — and AWS is explicit that control planes involve more moving parts and are consequently more likely to fail than the data planes they configure ( control plane and data plane ). The architectural rule that follows is equally explicit: don't depend on the control plane during recovery or in the serving path — Well-Architected's canonical example is Route 53, where the data plane carries a 100% SLA while the control plane does not ( REL11-BP04 ).
Three things go wrong when this is unplanned.
Nobody labels it, so nobody applies the rules. A settings table, a prompt store, a tier lookup — these get built as ordinary CRUD services with ordinary availability goals, because the team building them is thinking "admin feature," not "the thing every session depends on to exist."
The dependency is invisible in steady state. Running sessions hold their config in memory, so a control-plane outage produces no immediate errors. The failure only surfaces at the next session start — which means your blast radius is proportional to how fast your fleet turns over, and your monitoring shows health right up to the moment it doesn't.
Recovery is a stampede. Ten thousand sessions initializing at once is a thundering-herd load pattern that the config service was never sized for. The incident's aftermath becomes the incident's second act.
At 100 tenants none of this bites: session churn is low, the config service is over-provisioned relative to demand, and an outage is a few failed requests. At 10,000 tenants, the accidental control plane is the single point of failure your architecture diagram doesn't show — and we have never run this inventory with a team who found nothing.
Run the inventory yourself. Most agent platforms hold at least five of these:
Store | Read on the serving path? | Usually treated as | Should be |
|---|---|---|---|
Prompt store | yes — every session start | content / CMS | control plane; served from snapshot |
Tool registry + schemas | yes — tool resolution | service catalogue | control plane; served from snapshot |
Model routing rules | yes — per request | app config | control plane; served from snapshot |
Guardrail config | yes — per turn | safety setting | control plane; served from snapshot |
Tenant tier / limits | yes — admission | billing data | control plane; served from snapshot |
Session checkpoint store | yes — per turn write | storage | data plane; data-plane SLO |
3. How XXL ISVs build for this
The pattern is well documented, and it comes down to four moves. Each one buys availability with a little freshness or a little effort.
Split the planes explicitly, and name them. We decide which components configure the system and which serve traffic, write it down, and hold them to different availability standards. The AWS fault-isolation guidance treats this split as a primary design boundary ( control planes and data planes ). The naming matters more than it sounds: once a service is labelled a control plane, the rules about not depending on it become obvious to everyone. This costs us nothing except the meeting where we agree on it.
Make the serving path statically stable. Static stability means the data plane keeps working — including scaling up — while the control plane is unavailable, because everything it needs is already local ( static stability ). Concretely: each cell holds a local snapshot of prompts, tool schemas, entitlements and limits, refreshed in the background. A new session reads the snapshot, never the control plane. The control plane going down should limit our ability to change behaviour, never our ability to serve it. What we accept in exchange is staleness — a config change now takes a refresh interval to land, so that interval becomes a number we choose deliberately and document.
Put the smaller service in control. We invert the call direction so the high-volume side writes and the low-volume side reads. The Builders' Library pattern is that the data plane publishes its state and the control plane polls, rather than a fleet of clients calling in ( avoiding overload ). This is what prevents the recovery stampede: sessions never dial the config service, so ten thousand of them starting at once creates no fan-in. The inversion is unintuitive to build the first time, and it means the control plane learns about the fleet on a delay rather than on demand.
Do constant work. We reconcile complete snapshots on a fixed cadence rather than propagating deltas on demand, so load is identical whether nothing changed or everything did — no failure-triggered work spikes ( REL04-BP03 Do constant work ; Reliability, constant work, and a good cup of coffee ). A full tenant-config snapshot pushed every N seconds is boring in the best way: the mechanism behaves the same on our worst day as on our calmest. We pay for that with work we mostly don't need — the same snapshot, pushed again, whether anything changed or not.
Two supporting requirements complete the picture. Regional and cell isolation should apply to control-plane changes as much as to code, so a bad config can't reach every cell simultaneously (that's module 5's territory). And change visibility: every control-plane action should be observable in an event stream, including the ones no human initiated.
4. The agent delta
Every agent platform has a control plane, and it's larger than you think.
Behaviour lives in it. Beyond the usual tenant settings and quotas, an agent platform's control plane holds the prompt store, model routing rules, the tool registry, and guardrail configuration — everything that determines what agents do. That makes it both more critical and more frequently changed than a classic SaaS config store.
There is a concrete test. Ask: can a new session start while the config service is down? Prompt fetched, tools resolved, entitlements known — all from local state. If the answer is no, you cannot scale during the incident that most demands scaling, which is the failure mode from the cold open. It's a testable property, and worth testing deliberately rather than discovering.
The write path is a serving dependency. Agents checkpoint as they reason. If completing a turn requires a state or memory write, that store's quotas and availability belong in your data-plane budget, not filed under storage. Audit it with data-plane rigour.
Parts of the control plane may themselves be agentic. If an agent tunes routing, selects tools, or adjusts limits, then your control plane is making automated decisions at machine speed. Those decisions must land in the event stream like any other change — otherwise your audit trail describes a system that no longer exists, and your next incident review has no record of who (or what) changed the behaviour.
5. Design rules
- Find your accidental control plane. Inventory every store that shapes agent behaviour — prompt store, model routing, tool registry, guardrail config, tenant-tier tables. If the serving path reads it, it is a control plane, whether or not anyone decided to build one.
- Pass the static-stability test. A new session must be able to start — prompt fetched, tools resolved, entitlements known — while the config service is down. If we cannot scale during the incident that most demands it, the config service is not a control plane; it is a serving dependency.
- Let the smaller service do the writing. Sessions write their state; the control plane polls and reconciles full snapshots on a cadence — constant work. Never let ten thousand cold-starting sessions stampede the config service.
- Audit the write path as a data-plane dependency. If serving a turn requires a checkpoint or memory write, those quotas are in our availability equation — treat them with data-plane rigor, not control-plane tolerance.
- Make automated decisions visible. If an agent tunes routing or picks tools, our control plane is making decisions. Every automated decision must appear in the event stream, or the audit trail describes a system that no longer exists.
6. Key takeaways + next in series
- If the serving path reads it, it's a control plane — inventory your prompt store, tool registry, guardrail config and tier tables, and hold them to control-plane rules.
- Pass the static-stability test: a new session must start from local state while the config service is down. Serve from snapshots; let the smaller service poll; do constant work.
- Audit the checkpoint write path as a data-plane dependency, and put every automated decision in the event stream.
Next in the series : a bad prompt is a bad deploy — how behaviour changes ride the same waves as code.
Series: Secrets of Agentic Scale (6 articles)
- …
- 5SAS04 Your prompt store is a control plane This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article