
Your AI Agent Shouldn't Go Down When a Model Gets Throttled: Model Fallback with Amazon Bedrock AgentCore Gateway
How Amazon Bedrock AgentCore Gateway’s new model fallback keeps agents running when a model is throttled, with CLI setup, circuit breaker cost behavior, and the limits to know before production.
Every production agent has a hidden single point of failure: the model behind it. When that model is throttled, out of capacity, or unreachable, the agent stops, no matter how well the rest of your system is built.
The usual fix is retry logic in every client. It works, but it means every agent, script, and service you own needs its own copy of that logic, and each copy will drift. On October 22, 2026, AWS announced a different approach: model fallback routing in Amazon Bedrock AgentCore Gateway . You declare the fallback chain once on the gateway, and every client benefits without changing a line of code.
This post explains how it works, walks through configuring it with the AWS CLI, and covers the details that decide whether it helps or quietly costs you money.
What changed
Two things arrived together:
- InvokeModel support on the gateway's inference endpoint. The gateway now supports the Bedrock
InvokeModelandInvokeModelWithResponseStreamAPIs, so an existing Bedrock SDK integration can point at the gateway and reach any model provider through one governed endpoint.
- Fallback routing. When the primary destination is throttled, out of capacity, or unreachable, the gateway moves the request to an alternative model or provider automatically. The client sends one request and gets one response.
The second one is the interesting part, so that is what the rest of this post covers.
The mental model
Fallback is configured on a gateway rule. A rule has conditions (which requests it applies to) and actions (what to do with them). For model fallback, the pieces are:
Piece | What it does |
|---|---|
matchModels | Chooses which requested models the rule applies to |
routeToTarget with staticRoute | Pins matching requests to a target, and optionally to a specific model |
fallback | Declares what happens when that primary fails |
targetFallback | An ordered list of one or two alternative destinations |
triggerOn | The failures that make the gateway move on |
firstByteTimeoutMilliseconds | How long to wait before treating a stalled destination as failed |
circuitBreaker | Skips destinations that are currently failing |
A destination is a target, optionally with a specific model. That detail matters: the fallback can be a different provider serving the same model, or a different, cheaper model on the same target.
What you need
- An AgentCore Gateway with at least two inference targets. See the inference targets guide .
- The AWS CLI, with permission to create gateway rules
- The target names and model IDs your targets actually serve. Every name in the examples below is a placeholder.
Step 1: Route a model to a primary target
Start without fallback, so you can see the base rule:
bash
aws bedrock-agentcore-control create-gateway-rule \ --gateway-identifier my-gateway-id \ --priority 100 \ --description "Route my-model to the primary target" \ --conditions '[ {"matchModels": {"anyOf": [{"value": "my-model"}]}} ]' \ --actions '[ {"routeToTarget": {"staticRoute": {"targetName": "primary-target"}}} ]'This pins every request for
my-model to primary-target . Without a rule, the gateway uses its default model-based routing, which can send a request to any matching target.Step 2: Add a fallback chain
Now add a
fallback block inside the staticRoute :bash
aws bedrock-agentcore-control create-gateway-rule \ --gateway-identifier my-gateway-id \ --priority 100 \ --description "my-model with a two-step fallback" \ --conditions '[ {"matchModels": {"anyOf": [{"value": "my-model"}]}} ]' \ --actions '[ { "routeToTarget": { "staticRoute": { "targetName": "primary-target", "model": "my-model", "fallback": { "circuitBreaker": {"mode": "ENABLED"}, "triggerOn": { "statusCodes": ["5XX", "429"], "errors": ["CONNECTION_ERROR", "FIRST_BYTE_TIMEOUT", "UNROUTABLE"] }, "firstByteTimeoutMilliseconds": 5000, "targetFallback": [ {"targetName": "second-provider-target"}, {"targetName": "primary-target", "model": "smaller-model"} ] } } } } ]'Read the chain like this: try
my-model on the primary target. If it fails, try the second provider. If that also fails, degrade to a smaller model on the primary target. If everything fails, the caller receives the last error.Step 3: Choose your triggers on purpose
The
triggerOn block decides when the gateway gives up on a destination. It has two fields:-
statusCodes: up to ten exact codes or classes such as429or5XX. Only error responses can trigger fallback;2xxand3xxresponses never do.
-
errors: up to three conditions:CONNECTION_ERROR,FIRST_BYTE_TIMEOUT, andUNROUTABLE.
A response that matches none of your triggers goes back to the caller unchanged. That is a feature. Throttling (
429 ) and server errors ( 5XX ) are the classic reasons to move on. A 400 that says your request is malformed will fail on the next model too, so falling back on it just adds cost. Start narrow and widen only with a reason.The
FIRST_BYTE_TIMEOUT trigger deserves special attention. Providers do not always fail cleanly; sometimes they just stall. Setting firstByteTimeoutMilliseconds lets the gateway treat a silent destination as a failed one instead of leaving your agent hanging.Step 4: Understand the circuit breaker, because it controls your bill
Each outbound attempt the gateway makes is billed as one gateway invocation. A request that falls through three destinations is three invocations, and the model cost of each attempt is billed separately by Bedrock or the provider.
The circuit breaker exists to limit that waste. It tracks destinations that are failing and skips them:
State | What the gateway does | Billing effect |
|---|---|---|
Closed (healthy) | Sends to the primary, falls back if a trigger fires | Failed primary attempts are billed |
Open (failing) | Skips the primary and goes straight to the next destination | The skipped primary is not billed |
Half-open (recovering) | Lets a small share of requests test the primary | Those test attempts are billed |
It is enabled by default. You might turn it off if you prefer to keep using a cheaper primary rather than failing over to a more expensive secondary. With the breaker disabled, the chain still runs in the order you declared.
Step 5: Know the limits
- Streaming has a point of no return. Once a streaming response has started, the gateway can no longer fall back. The caller receives the partial response and an error and has to retry. Your client still needs a retry path for mid-stream failures.
- Fallback chains are short. You can list one or two alternatives after the primary.
- Every destination must be distinct. The same target and model combination cannot appear twice in a rule.
- Fallback applies to inference targets only.
- Weighted routes get fallback per leg. If you split traffic with a
weightedRoute, each leg carries its ownfallbackblock. A leg without one returns the error to the caller.
- Clients cannot see which destination answered. The gateway adds no indicator to the response, so you rely on its routing logs to see where a request went and when the breaker skipped a destination.
The part nobody should skip: fallback changes behavior
Fallback keeps your agent available, but a different model is a different model. It may follow tool-calling instructions differently, format output differently, or reason less carefully. Treat every fallback destination as part of your agent's quality surface:
- Run the same evaluation prompts against each model in the chain, not just the primary.
- Prefer a fallback that is a close substitute over one that merely exists.
- Decide deliberately whether a degraded answer is better than an error for each agent. For some workflows, failing loudly is the safer choice.
AgentCore also offers evaluations you can use to check agent quality, so quality checks for fallback models can live next to the rest of your agent testing.
How to test it safely
Do not wait for a real outage. In a non-production gateway, force a fallback on purpose:
- Create the rule with a very low
firstByteTimeoutMillisecondsso the primary appears to stall.
- Send a request and confirm you still get a valid response.
- Check the routing logs to confirm the second destination served it.
- Raise the timeout back to a realistic value before you copy the rule to production.
Then repeat with the circuit breaker enabled and disabled, so you see both behaviors and understand the billing difference before it appears on an invoice.
Cleanup
Rules and test gateways you create for experiments keep existing until you remove them. Delete the test rule and any test targets when you are done, and check the gateway invocation charges for your test period.
Takeaways
- Put fallback on the gateway and keep client code simple, instead of copying retry logic into every agent.
- Choose triggers deliberately: throttling and server errors yes, bad requests no.
- Set a first-byte timeout so stalled providers do not hang your agent.
- Leave the circuit breaker on unless you have a cost reason not to.
- Evaluate every fallback model, because availability is not the same as quality.
For the full reference, read the model routing and fallback documentation . If you are building agents on Amazon Bedrock, I would like to hear which fallback chain you would use and why.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article