AWS Builder Center
A practical mental model for inference reliability

A practical mental model for inference reliability

A practical guide to keeping inference working reliably as your product and customer base grow.

I help EMEA Startups build with Agentic AI



by Diana Alvarado and Lorenzo Boccaccia
Many startups we work with start building their genai apps using a single call to a model endpoint. That is often enough for a proof of concept. Once customers depend on the feature, it usually is not.
As you move toward production, treat inference like any other external dependency by giving requests a timeout, limiting retries, planning for capacity, and adding observability, fallback routes, and a useful degraded experience.
Inference can fail in unfamiliar ways, but the reliability principles are the same ones you already use for databases, backends, and other critical parts of your application. This series puts the approach we use with customers into writing: Part 1 helps you decide what you need and when, while Parts 2 and 3 show you how to build and operate it as your workload grows.

"Everything fails, all the time"

Amazon CTO Werner Vogels' observation that "everything fails, all the time" is a useful starting point because you already expect servers, Availability Zones, network calls, and storage to fail. Inference belongs on that list too, even though its failures can look different.
A request might be throttled because you reached a quota or high volume in a Region, while a large or complex prompt might take longer than expected, especially under load. Even when the request completes successfully, it can still fail from the user's point of view if the answer does not meet your application's requirements.
Models and model versions can also behave differently over time, become unavailable, or be withdrawn, which makes model selection an ongoing dependency decision rather than a one-time choice.
Endpoint health alone will not catch these problems. In September 2025,  Anthropic described three infrastructure bugs  that sometimes reduced response quality even though requests still completed, with the impact varying by model, hardware platform, and route. Research has also found that the  "same" hosted LLM service can change substantially over time , while independent projects such as  MarginLab's historical tracker  show how repeated task evaluations can reveal changes that normal endpoint metrics miss.
This is one reason to consider a second, independently evaluated model: it gives you another route when a particular model or serving stack has a bad day, not only when an entire provider is unavailable. That fallback helps only if you can detect the problem and confirm that the second model still produces acceptable results.
You do not need a new reliability discipline for inference; start with the practices you already use for other dependencies, then add controls as failures become more likely or more costly. You will still balance cost, response quality, and latency when choosing a model, but availability sits underneath all three because a model that cannot serve the request cannot meet your quality or latency target. Cost matters here when it limits the capacity you can reserve, the fallback you can afford, or the degraded response you can offer, while prompt compression, context management, and choosing models to reduce cost are related topics that deserve a separate discussion.

Expand what your system can survive

You can make these decisions by asking two questions:
  • Architecture: Which failures can your system handle without disrupting users?
  • Product maturity: When does the customer impact justify another layer of complexity?
You will usually begin with one model in one primary Region, then add layers that let the system handle a wider set of failures. This widens the failure boundary, meaning more can fail before the failure reaches your users.
Figure 1 shows four ways to do that, each representing an architecture option rather than a product stage.

Figure 1. Inference reliability progression
Each layer addresses a different gap.
  • Single path: Your application calls one model in its primary Region, with timeouts, a bounded retry, and graceful degradation protecting individual requests. A regional or model-specific problem can still affect users.
  • Broader capacity: You keep the same model but make more capacity available through a secondary Region, a regional routing endpoint, or a global endpoint, depending on the provider and workload. These are alternatives within the same layer, not a checklist you must complete in order.
  • Model fallback: You add a different model for problems that more same-model capacity cannot solve, such as model-specific throttling, a drop in answer quality, or model withdrawal. Evaluate the fallback before relying on it because an available answer that is not good enough is still a failure.
  • Gateway control plane: When routing policy spans endpoints, Regions, models, consumers, and service tiers, put that policy behind a stable alias instead of spreading it through application code. The gateway manages routes you already have; it is not another fallback destination.
Building every layer at the start adds cost and operational work before you know what you need, so add the smallest improvement that addresses a real risk or a problem your measurements have exposed.

You don't need perfect resilience on day one

Your reliability work should match where the product is today because, although each new layer helps you handle more failures, it also gives you more policy to test and operate. Add it too early and you carry unnecessary complexity; add it too late and customers feel the failure first.
Consider the ticket processing system in Figure 2, which reads an incoming support ticket, retrieves order information, and drafts a reply for a person to approve. Its reliability needs change as it moves from an experiment to a service customers rely on.

Figure 2. Ticket processing system.
Figure 3 shows product maturity separately from routing architecture: prototype, MVP, and production describe what you need to prove as users become more dependent on the feature, while scale or sharing sits alongside those stages because competing workloads can create capacity problems at any point.

Figure 3. Product reliability progression
The goal changes as the product matures: first, prove that the task is useful; then prove that it keeps working as the product changes; and finally, make sure a dependency failure does not automatically become a customer incident. Choose routing changes separately, based on the failures you actually see.
Here is what that looks like for the ticket processing system.
Prototype: prove that it works. Check that the model can draft a useful reply and that the cost per completed ticket makes sense, then add a timeout, one bounded retry, an output limit, and a useful failure response while keeping the path simple enough to learn from.
MVP: prove that it keeps working. Real tickets will expose cases the prototype missed just as changes to prompts, models, and configuration begin to affect paying customers, so turn those cases into evaluations and make sure you can promote or revert application code and inference configuration together.
Production: handle dependency failures. As more customers rely on the drafts, capacity constraints, regional failures, and model-specific problems can become incidents, so add the smallest routing change that addresses the problem you observe. Use broader same-model capacity for regional or capacity issues, an evaluated fallback for model-specific issues, and a gateway when the routing policy has become too complex to keep in application code.
Scale or sharing: protect workloads from one another. When the ticket system serves several tenants or shares inference capacity with other workloads, add isolation, capacity policies, fan-out limits, and per-consumer attribution. This might happen during MVP or production because the trigger is competition for shared capacity, not a stage name.
These controls build on one another: the timeout you added for the prototype still matters behind a gateway, the evaluations you built for the MVP become the test for a fallback model, and routing telemetry tells you whether extra capacity is helping.
Move forward when your evidence shows that the current design is affecting customers, rather than adding a layer simply because the product has reached a particular stage. If tenants or workloads are already competing for the same quota, however, add isolation sooner.

Summary

Treat inference as a dependency and add reliability controls as customer impact grows. Start by bounding each request, then add more same-model capacity when regional or capacity failures reach users. Add a different model when your evaluations show that model-specific failures matter, and introduce a gateway when the routing policy no longer belongs in application code, keeping the earlier controls in place as you add each layer.
In part 2, we'll show you how to build these layers.
For broader AWS guidance, see  Reliability in the AWS Well-Architected Generative AI Lens , which covers throughput management, error handling and failover, prompt and model versioning, and distributed availability.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article