
Building inference reliability controls with Amazon Bedrock and AgentCore
This post covers how to implement inference reliability in stages: bound the first route, use real evaluations to make MVP changes safe, then add failover when dependency failures affect users.
This is Part 2 of the inference reliability series. Part 1 made the main point: inference is a dependency, and dependencies fail. Now we will apply that idea to a support-ticket agent.
The agent has a simple job. It receives a customer question and trusted order details, then drafts a reply for a person to review. We will keep that job and the prompt unchanged while the system grows around it:
- First, make one model call predictable.
- Next, prove that a second model can produce a safe reply.
- Finally, put the tested routing rules in one place and run the agent on managed infrastructure.
You don't need to build a production stack on day one. Add each control when you can explain the failure it prevents and show that it works.
This post covers three stages of deployment:
Stage | Question to answer | What should be true before moving on |
|---|---|---|
Prototype | Can one model draft a useful reply without hanging or running away? | Useful drafts, no unexpected truncation, and acceptable latency, token use, and cost per ticket |
MVP | Can another model take over without breaking required behavior? | Both models pass the same rules, the fallback's quality is acceptable, and a forced failover works |
Production | Can every agent use the same tested routing policy? | A forced primary failure reaches the fallback in time, the served route is visible, and routing and hosting changes can be rolled back |
Each stage keeps the controls from the one before it.
Stage 1. Make one model call predictable

Figure 1. Prototype stage considerations for inference reliability
Start with one route. Before thinking about fallback models or gateways, prevent three common problems: a request that hangs, a transient error that causes too many retries, and a response that grows much larger than the task needs.
The sample uses Strands and its
BedrockModel , which calls Amazon Bedrock through the Converse API. The agent always receives the same ticket and order context, even if the model changes later.BEDROCK_CONFIG = Config( connect_timeout=5, read_timeout=30, retries={"total_max_attempts": 2, "mode": "adaptive"},)
SYSTEM_PROMPT = """You draft concise customer-support replies for human review.Use only facts supplied in ORDER CONTEXT.Never invent an order ID, tracking number, action, or policy.Never claim that a refund was approved, issued, or processed.If facts are missing, ask the customer to verify them."""
def build_ticket_agent(model_id: str) -> Agent: model = BedrockModel( model_id=model_id, region_name="us-east-1", max_tokens=300, temperature=0, streaming=False, boto_client_config=BEDROCK_CONFIG, ) return Agent(model=model, system_prompt=SYSTEM_PROMPT)Return timing and usage with every draft. Without them, you cannot tell whether a reply was slow, expensive, or cut short:
def draft_reply(ticket: str, order_context: str, *, model_id: str) -> Draft: agent = build_ticket_agent(model_id) started = time.perf_counter() result = agent( f"<ticket>{ticket}</ticket>\n" f"<order_context>{order_context}</order_context>" )
usage = result.metrics.get_summary()["accumulated_usage"] return Draft( text=str(result).strip(), model_id=model_id, latency_ms=round((time.perf_counter() - started) * 1_000), input_tokens=usage["inputTokens"], output_tokens=usage["outputTokens"], stop_reason=result.stop_reason, )max_tokens=300 does two jobs: it keeps the reply short and limits the output token quota reserved when the request starts. The returned token counts let you calculate cost per ticket. stop_reason tells you whether the reply ended normally or hit the limit.The primary route is the Amazon Bedrock global inference profile
global.amazon.nova-2-lite-v1:0 . It lets Bedrock route requests across the AWS Regions supported by that profile. If global routing is unavailable or your data must remain in a specific region, like the US, use us.amazon.nova-2-lite-v1:0 instead. Both profiles still call Nova 2 Lite. More regions can help with capacity or regional problems, but it cannot help with model specific failures.Do not move on because one demo looked good. Try ordinary tickets, missing order details, and requests the agent must refuse. Move on when reviewers find the drafts useful and you have clear limits for latency, token use, truncation, and cost per ticket. If one model cannot meet those limits, fix that before adding a second route.
Stage 2. Prove the fallback is safe enough

Figure 2. MVP stage considerations for inference reliability
Once people depend on the drafts, two things can go wrong. A prompt or model change can quietly break behavior, and the primary model can throttle or fail even when Bedrock can route it across Regions.
The same ticket tests help with both problems. They catch regressions in the primary and tell you whether another model is safe enough to use as a fallback.
Start with cases that state what the reply must do:
TICKET_SUITE = ( EvalCase( name="shipped order", ticket="Where is order #10042?", order_context="Order #10042 shipped on 4 August. Tracking ID ZX-1942.", required_terms=("#10042", "shipped"), allowed_order_ids=frozenset({"10042"}), ), EvalCase( name="unknown order", ticket="Where is order #99999?", order_context="No matching order was found. Ask the customer to verify the order ID.", required_terms=("#99999", "verify"), allowed_order_ids=frozenset({"99999"}), ),)For this agent, three rules are not negotiable:
- The reply must use a valid order ID.
- The reply must include the facts or actions required by the case, such as saying that an order shipped or asking the customer to verify an unknown ID.
- A reply must not claim that a refund was approved, authorized, issued, or processed.
These are pass-or-fail checks, not opinions about writing style. Tone, completeness, and usefulness still need human or AI review.
Run the same real ticket through each model and report the exact rule that failed:
def run_eval_suite(model_id: str, cases: Iterable[EvalCase]) -> EvalReport: outcomes = tuple( score( case, draft_reply( case.ticket, case.order_context, model_id=model_id, ), ) for case in cases ) return EvalReport(model_id, outcomes)Now compare the primary with the model you want to use as a fallback:
baseline = run_eval_suite( "global.amazon.nova-2-lite-v1:0", TICKET_SUITE,)candidate = run_eval_suite( "us.amazon.nova-micro-v1:0", TICKET_SUITE,)
if candidate.pass_rate < baseline.pass_rate or candidate.pass_rate < 1: raise SystemExit("release blocked: candidate did not preserve required behavior")Do not read a passing score as "these models are equivalent." It means only that the fallback preserved the required behavior in this suite. Compare tone, detail, and usefulness separately, and keep adding real failures to the suite as you find them.
At this stage, the fallback logic can still live in the application. Use the fallback only for errors you have decided are safe to retry, such as throttling or a temporary service failure. Return the model that served the request and whether fallback happened. Otherwise, a successful reply can hide a broken primary route.
Use the fallback in production only after both models pass the required checks, the quality gap is acceptable, and deliberately breaking the primary produces a good-enough reply through the fallback.
Stage 3. Put the routing rules in one place

The first fallback loop is easy to keep in application code. The trouble starts when several agent instances or services each carry their own copy. Changing the fallback order now means changing every copy, and during a partial rollout two requests can follow different rules.
That is the point of a gateway: applications ask for one stable name, and one place owns the model order, retries, timeouts, authentication, and route logs. The sample uses LiteLLM Proxy with Amazon Bedrock and calls the route
ticket-drafter :model_list: - model_name: ticket-drafter litellm_params: model: os.environ/LITELLM_PRIMARY_MODEL model_info: id: ticket-drafter-primary - model_name: ticket-drafter-fallback litellm_params: model: os.environ/LITELLM_FALLBACK_MODEL model_info: id: ticket-drafter-fallback
litellm_settings: fallbacks: - ticket-drafter: - ticket-drafter-fallback num_retries: 1 request_timeout: 30The application now asks for
ticket-drafter instead of naming a Bedrock model. LiteLLM tries the primary first and moves to the fallback when the configured error policy allows it. Its response headers show which route served the request.Nothing about the ticket or prompt changes. Only the model connection changes:
model = LiteLLMModel( client_args={ "api_base": LITELLM_BASE_URL, "api_key": LITELLM_API_KEY, "use_litellm_proxy": True, "num_retries": 0, "timeout": 35, }, model_id="ticket-drafter", params={"max_tokens": 300, "temperature": 0}, stream=False,)agent = Agent(model=model, system_prompt=SYSTEM_PROMPT)
app = BedrockAgentCoreApp()
@app.entrypointdef invoke(payload): result = draft_reply_via_gateway( payload["ticket"], payload["order_context"], ) return { "draft": result.text, "route_alias": result.model_id, "latency_ms": result.latency_ms, "input_tokens": result.input_tokens, "output_tokens": result.output_tokens, "stop_reason": result.stop_reason, }It is easy to mix up LiteLLM and AgentCore Runtime because both appear in the production setup, but they solve different problems:
- LiteLLM chooses the model route. It owns the primary and fallback order, how many times to retry, and route logs.
- AgentCore Runtime runs the agent application. It provides managed execution, adds capacity as traffic grows, and keeps sessions separate for the code that calls LiteLLM.
Runtime does not decide which model to call. LiteLLM does not decide whether the answer is good enough. The ticket tests still make that decision.
The sample includes a small gateway probe that reads LiteLLM's
x-litellm-model-id header. Break the primary on purpose and check that the header changes to ticket-drafter-fallback . Then score the returned draft. This tests two different things: whether traffic moved, and whether the fallback
answer was still acceptable.Before relying on this path, there are four key aspects to confirm:
- The fallback answers within your latency expectation.
- The returned draft still passes the required ticket checks.
- You can see which route served the request.
- You can roll back gateway rules without redeploying the agent.
Test Runtime separately. Check that it stays healthy with the number of simultaneous requests you expect and that you can roll back an agent deployment. A managed host and a working model fallback are both useful, but neither proves the other works.
Summary
The key takeaway from part 2 is that adding a gateway or a managed Runtime does not make inference reliable by itself. Reliability comes from keeping requests bounded, testing both models, forcing the failure path, and checking what users receive when that path runs.
Part 3 will focus on helping you scale and operate production inference reliably.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article