AWS Builder Center
Your AI Agent Shouldn't Go Down When a Model Gets Throttled: Model Fallback with Amazon Bedrock AgentCore Gateway

Your AI Agent Shouldn't Go Down When a Model Gets Throttled: Model Fallback with Amazon Bedrock AgentCore Gateway

How Amazon Bedrock AgentCore Gateway’s new model fallback keeps agents running when a model is throttled, with CLI setup, circuit breaker cost behavior, and the limits to know before production.

Every production agent has a hidden single point of failure: the model behind it. When that model is throttled, out of capacity, or unreachable, the agent stops, no matter how well the rest of your system is built.
The usual fix is retry logic in every client. It works, but it means every agent, script, and service you own needs its own copy of that logic, and each copy will drift. On October 22, 2026, AWS announced a different approach:  model fallback routing in Amazon Bedrock AgentCore Gateway . You declare the fallback chain once on the gateway, and every client benefits without changing a line of code.
This post explains how it works, walks through configuring it with the AWS CLI, and covers the details that decide whether it helps or quietly costs you money.

What changed

Two things arrived together:
  1. InvokeModel support on the gateway's inference endpoint. The gateway now supports the Bedrock  InvokeModel  and  InvokeModelWithResponseStream  APIs, so an existing Bedrock SDK integration can point at the gateway and reach any model provider through one governed endpoint.
  1. Fallback routing. When the primary destination is throttled, out of capacity, or unreachable, the gateway moves the request to an alternative model or provider automatically. The client sends one request and gets one response.
The second one is the interesting part, so that is what the rest of this post covers.

The mental model

Fallback is configured on a gateway rule. A rule has conditions (which requests it applies to) and actions (what to do with them). For model fallback, the pieces are:
Piece
What it does
 matchModels 
Chooses which requested models the rule applies to
 routeToTarget  with  staticRoute 
Pins matching requests to a target, and optionally to a specific model
 fallback 
Declares what happens when that primary fails
 targetFallback 
An ordered list of one or two alternative destinations
 triggerOn 
The failures that make the gateway move on
 firstByteTimeoutMilliseconds 
How long to wait before treating a stalled destination as failed
 circuitBreaker 
Skips destinations that are currently failing
A destination is a target, optionally with a specific model. That detail matters: the fallback can be a different provider serving the same model, or a different, cheaper model on the same target.

What you need

  • The AWS CLI, with permission to create gateway rules
  • The target names and model IDs your targets actually serve. Every name in the examples below is a placeholder.

Step 1: Route a model to a primary target

Start without fallback, so you can see the base rule:
bash
aws bedrock-agentcore-control create-gateway-rule \
--gateway-identifier my-gateway-id \
--priority 100 \
--description "Route my-model to the primary target" \
--conditions '[
{"matchModels": {"anyOf": [{"value": "my-model"}]}}
]' \
--actions '[
{"routeToTarget": {"staticRoute": {"targetName": "primary-target"}}}
]'
This pins every request for  my-model  to  primary-target . Without a rule, the gateway uses its default model-based routing, which can send a request to any matching target.

Step 2: Add a fallback chain

Now add a  fallback  block inside the  staticRoute :
bash
aws bedrock-agentcore-control create-gateway-rule \
--gateway-identifier my-gateway-id \
--priority 100 \
--description "my-model with a two-step fallback" \
--conditions '[
{"matchModels": {"anyOf": [{"value": "my-model"}]}}
]' \
--actions '[
{
"routeToTarget": {
"staticRoute": {
"targetName": "primary-target",
"model": "my-model",
"fallback": {
"circuitBreaker": {"mode": "ENABLED"},
"triggerOn": {
"statusCodes": ["5XX", "429"],
"errors": ["CONNECTION_ERROR", "FIRST_BYTE_TIMEOUT", "UNROUTABLE"]
},
"firstByteTimeoutMilliseconds": 5000,
"targetFallback": [
{"targetName": "second-provider-target"},
{"targetName": "primary-target", "model": "smaller-model"}
]
}
}
}
}
]'
Read the chain like this: try  my-model  on the primary target. If it fails, try the second provider. If that also fails, degrade to a smaller model on the primary target. If everything fails, the caller receives the last error.

Step 3: Choose your triggers on purpose

The  triggerOn  block decides when the gateway gives up on a destination. It has two fields:
  •  statusCodes : up to ten exact codes or classes such as  429  or  5XX . Only error responses can trigger fallback;  2xx  and  3xx  responses never do.
  •  errors : up to three conditions:  CONNECTION_ERROR ,  FIRST_BYTE_TIMEOUT , and  UNROUTABLE .
A response that matches none of your triggers goes back to the caller unchanged. That is a feature. Throttling ( 429 ) and server errors ( 5XX ) are the classic reasons to move on. A  400  that says your request is malformed will fail on the next model too, so falling back on it just adds cost. Start narrow and widen only with a reason.
The  FIRST_BYTE_TIMEOUT  trigger deserves special attention. Providers do not always fail cleanly; sometimes they just stall. Setting  firstByteTimeoutMilliseconds  lets the gateway treat a silent destination as a failed one instead of leaving your agent hanging.

Step 4: Understand the circuit breaker, because it controls your bill

Each outbound attempt the gateway makes is billed as one gateway invocation. A request that falls through three destinations is three invocations, and the model cost of each attempt is billed separately by Bedrock or the provider.
The circuit breaker exists to limit that waste. It tracks destinations that are failing and skips them:
State
What the gateway does
Billing effect
Closed (healthy)
Sends to the primary, falls back if a trigger fires
Failed primary attempts are billed
Open (failing)
Skips the primary and goes straight to the next destination
The skipped primary is not billed
Half-open (recovering)
Lets a small share of requests test the primary
Those test attempts are billed
It is enabled by default. You might turn it off if you prefer to keep using a cheaper primary rather than failing over to a more expensive secondary. With the breaker disabled, the chain still runs in the order you declared.

Step 5: Know the limits

  • Streaming has a point of no return. Once a streaming response has started, the gateway can no longer fall back. The caller receives the partial response and an error and has to retry. Your client still needs a retry path for mid-stream failures.
  • Fallback chains are short. You can list one or two alternatives after the primary.
  • Every destination must be distinct. The same target and model combination cannot appear twice in a rule.
  • Fallback applies to inference targets only.
  • Weighted routes get fallback per leg. If you split traffic with a  weightedRoute , each leg carries its own  fallback  block. A leg without one returns the error to the caller.
  • Clients cannot see which destination answered. The gateway adds no indicator to the response, so you rely on its routing logs to see where a request went and when the breaker skipped a destination.

The part nobody should skip: fallback changes behavior

Fallback keeps your agent available, but a different model is a different model. It may follow tool-calling instructions differently, format output differently, or reason less carefully. Treat every fallback destination as part of your agent's quality surface:
  • Run the same evaluation prompts against each model in the chain, not just the primary.
  • Prefer a fallback that is a close substitute over one that merely exists.
  • Decide deliberately whether a degraded answer is better than an error for each agent. For some workflows, failing loudly is the safer choice.
AgentCore also offers  evaluations  you can use to check agent quality, so quality checks for fallback models can live next to the rest of your agent testing.

How to test it safely

Do not wait for a real outage. In a non-production gateway, force a fallback on purpose:
  1. Create the rule with a very low  firstByteTimeoutMilliseconds  so the primary appears to stall.
  1. Send a request and confirm you still get a valid response.
  1. Check the routing logs to confirm the second destination served it.
  1. Raise the timeout back to a realistic value before you copy the rule to production.
Then repeat with the circuit breaker enabled and disabled, so you see both behaviors and understand the billing difference before it appears on an invoice.

Cleanup

Rules and test gateways you create for experiments keep existing until you remove them. Delete the test rule and any test targets when you are done, and check the gateway invocation charges for your test period.

Takeaways

  • Put fallback on the gateway and keep client code simple, instead of copying retry logic into every agent.
  • Choose triggers deliberately: throttling and server errors yes, bad requests no.
  • Set a first-byte timeout so stalled providers do not hang your agent.
  • Leave the circuit breaker on unless you have a cost reason not to.
  • Evaluate every fallback model, because availability is not the same as quality.
For the full reference, read the  model routing and fallback documentation . If you are building agents on Amazon Bedrock, I would like to hear which fallback chain you would use and why.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article