SAS03 Park, don't shed: Graceful degradation for long-running agents
Load shedding assumes rejected work is cheap to redo. A session that has been reasoning for fourteen minutes inverts that assumption, and shedding at scale replaces load with more expensive load. This covers the four-rung degradation ladder, what a checkpoint must hold to survive suspension, why every resume needs a deadline, and why session churn belongs in your capacity model rather than your UX backlog.
Series: Secrets of Agentic Scale (6 articles)
- …
- 4SAS03 Park, don't shed: Graceful degradation for long-running agents This article
1. Agent state costs time and tokens, don't throw it away
Your platform is healthy. Utilization sits at 70%, no quota is exhausted, no alarm is firing. And your customers are telling you the product is broken.
Here's what they see. A user asks an agent to work through a large migration plan — the kind of task that takes twenty minutes and forty tool calls. At minute fourteen, the session's host is reclaimed during a routine scale-in. The platform does what it was built to do: the request fails, the client retries, a fresh session starts. Correct behaviour, textbook resilience.
Except the retry doesn't resume anything. It restarts — from the first prompt, with none of the fourteen minutes of accumulated reasoning, none of the tool results already fetched, none of the files written into the workspace. The user watches the agent rediscover what it already knew, burning the same tokens a second time, and gives up before it finishes.
Meanwhile a second failure mode is filling your support queue: users whose sessions time out at the front door during traffic peaks. Not throttled — your admission layer let them in. They died waiting for a session to be provisioned, because creating one is heavyweight work and your deadline was 25 seconds.
Neither failure appears in your capacity dashboards. Both are capacity failures.
2. Why the published pattern stops working
Every load-shedding pattern you've been taught rests on one economic assumption: rejected work is cheap to redo. Shed a request under overload, return 503, let the client retry with backoff — the cost is one round trip. That assumption is what makes shedding the correct answer to saturation, and it's why the classic advice is to shed early and shed decisively.
Agent sessions invert the economics. A session that has been running for fourteen minutes holds conversation history, tool outputs, intermediate reasoning, and files on disk. Shedding it doesn't cost a retry; it costs everything the session has accomplished — and then costs it again in tokens when the retry redoes the work. The longer a session has run, the more expensive it is to drop, which is exactly backwards from the assumption your shedding logic encodes. Worse, at scale, mass shedding of long sessions creates a retry storm of expensive restarts: you shed to reduce load and the replacement load is higher than what you shed.
The second broken assumption is about where time goes. Classic capacity thinking treats provisioning as instantaneous and throttling as the thing that ruins deadlines. For agent runtimes it's the reverse: each session runs in a dedicated microVM with isolated CPU, memory and filesystem ( how AgentCore Runtime works ), so creating one is real provisioning work rather than a function call — and each new session re-pays setup costs the old one had already amortized, including re-authenticating to the tools it needs. Measure it on your own platform before trusting a deadline: if provisioning consumes most of the budget, a cold start is not a latency detail, it is the deadline. You can be far below every documented quota and still fail requests purely because you had no warm capacity ready.
Put together: your saturation response destroys value, and your scale-up response arrives too late to matter. The published pattern gives you two tools — shed and provision — both mistuned for work that is long-lived and stateful.
At 100 tenants a dropped session is an apology and a credit note. At 10,000 tenants it is your product's reputation, several times an hour — and in my experience no team plans for this before the first time it happens to them.
3. How XXL ISVs build for this
Operators of long-running, stateful workloads at scale don't choose between serving and shedding. They build a ladder of degradation and put shedding on the bottom rung. We climb down it one rung at a time.
Rung one: serve. Normal path, full capacity.
Rung two: queue. Delay admission rather than refuse it — the Builders' Library treats queueing and load shedding as the same toolkit, chosen by how long the work can wait ( Using load shedding to avoid overload ). For agent work we queue new sessions before touching running ones: a user waiting ten seconds to start is a better outcome than a user losing fourteen minutes of progress. The cost is that queue depth is now something we have to watch and bound, or we have simply moved the failure later.
Rung three: park. Suspend a running session, persist its state, release its compute, and schedule a bounded resume. The parked session keeps its place in the world — its slot, its identity, its workspace — and comes back where it left off. This is the rung the published patterns don't give us, and it's the one agent workloads need most. It is also the most expensive to build: we need somewhere durable to put the state, a resume path that works, and the discipline to test both.
Rung four: shed — only work that has accumulated nothing, and only when the rungs above are exhausted.
What each rung costs, and what it buys back:
Rung | Action | Applies to | Cost to the user | Capacity freed |
|---|---|---|---|---|
1 | Serve | all work | none | none |
2 | Queue | new sessions | wait to start | admission deferred |
3 | Park | running sessions with state | resume delay | compute released, state kept |
4 | Shed | work with nothing accumulated | full restart | immediate |
Two engineering properties make the ladder real. First, checkpoints we have actually restored. Parking is only as good as the state we wrote: conversation history, tool outputs, the current plan, and workspace files, written on a cadence rather than at the moment of crisis. A checkpoint that has never been restored is a shed with extra steps. Writing them costs storage and a little latency on every turn, which is the price of the rung above. Second, bounded resumes. Every park carries an explicit deadline — resume by T, or fail loudly. Unbounded parking is a capacity leak that looks like success.
The other half of the answer is refusing to be surprised by demand. Pre-provisioned capacity plus graceful throttling past it is the shape AWS itself ships for control-plane-heavy scale: Amazon EKS Provisioned Control Plane lets you buy headroom in advance and degrades predictably beyond it rather than failing at the moment of the spike ( EKS Provisioned Control Plane ). Applied to agent platforms: we maintain a warm pool sized to our arrival rate, reuse sessions instead of recreating them, and treat session churn as a capacity problem rather than a client-side concern. Warm capacity is idle capacity, so this is a deliberate spend — worth sizing against the deadline we are actually trying to meet rather than the one we hope for.
Finally, the admission and parking layers must themselves fail open. A degradation ladder that stops serving when its own coordinator is unhealthy has added a new single point of failure to the system it was meant to protect — the Builders' Library is explicit that overload-protection machinery must not become the outage.
4. The agent delta
Parking isn't a new idea; workflow engines have suspended and resumed work for years. What's new is that the unit of work has properties that make parking mandatory rather than clever.
The state worth keeping is the expensive part. In a workflow engine, the suspended state is a few variables and a program counter. In an agent session it's a context window built over many turns, tool results that cost real money to fetch, and a plan the model reasoned its way to. The asset is the reasoning, and it exists nowhere else.
Runtimes now make parking implementable. AgentCore Runtime defaults to a
maxLifetime of 28,800 seconds — 8 hours — with a 15-minute idle timeout, both configurable ( lifecycle settings , as of July 2026), and offers managed per-session file storage ( file system configuration ). That is exactly the substrate a park needs: somewhere durable to leave the workspace while compute goes away. What used to require building your own suspension machinery is now mostly a matter of deciding what to checkpoint and when to resume.Session granularity is a design decision with four consequences. Per-user sessions or one shared session per tenant? That single choice sets your fairness key (a shared session makes one user's runaway loop every user's problem), the blast radius of a park, how well cold starts amortize across users, and — critically — the isolation boundary you own, because runtimes generally do not validate that a caller owns the session it resumes. Default to per-user sessions with per-tenant admission above them; treat pooled tenant sessions as a deliberate cost tier with explicit in-session isolation.
Also agent-shaped: task durations span seconds to hours in the same queue, so parking decisions interact with the size-class scheduling from module 2 — park the long branch, keep the short ones moving.
5. Design rules
- Degrade down a ladder: serve → queue → park → shed. Shedding a stateless request costs a retry; shedding an agent session destroys accumulated reasoning, tool results and workspace. Make shed the last rung of an explicit ladder, not the reflex.
- Park with a bounded resume. A parked session keeps its slot, its locks and its workspace. Give every park an explicit resume deadline so parked work frees capacity predictably instead of leaking it forever.
- Design the checkpoint before we need it. Decide what must survive a park — conversation state, tool outputs, the plan — and write it on a cadence. Then test it, a checkpoint we have never restored is a shed with extra steps.
- Treat session churn as a capacity problem. Session creation is heavyweight — environment provisioning plus a per-session auth tax — so cold starts inside a deadline are self-inflicted. Reuse warm sessions and provision headroom ahead of demand; parking is capacity management, not just UX.
- Pick session granularity deliberately. Per-user vs shared per-tenant sessions sets four things at once: the fairness key, a park's blast radius, cold-start amortization, and the isolation boundary you own. Default to per-user sessions with per-tenant admission above them.
6. Key takeaways + next in series
- Shedding assumes rejected work is cheap to redo; agent sessions invert that economics — the longer one has run, the more expensive it is to drop.
- Degrade down an explicit ladder — serve, queue, park, shed — and give every park a checkpoint you have restored and a bounded resume deadline.
- Session churn is a capacity problem: provision warm headroom and reuse sessions, because a cold start inside a tight deadline is the deadline.
Next in the series : your prompt store is a control plane — and the serving path has to survive without it.
Series: Secrets of Agentic Scale (6 articles)
- …
- 4SAS03 Park, don't shed: Graceful degradation for long-running agents This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article