The AI Agent That Knows When NOT to Think: Building a Cost-Aware AI Router on AWS
What if an AI application could decide when to answer directly, retrieve evidence, call a tool, or ask for clarification? Explore a practical architecture for building cost-aware, evidence-driven AI agents with Amazon Bedrock, Python, and AWS services, including routing logic, safety controls, and reproducible benchmarks.
The AI Agent That Knows When NOT to Think
Building a Cost-Aware, Evidence-Driven AI Router on AWS
1. Introduction
Not every user request needs a powerful AI model, a knowledge retrieval pipeline, or an autonomous agent.
Consider these four requests:
- “Explain what an API is.” — A direct answer may be sufficient.
- “Summarize our company’s internal deployment guide.” — The system needs relevant source material.
- “Check the current status of my order.” — The system needs an authorized tool call.
- “Book the meeting for the best time.” — The system may need clarification before taking action.
Treating all four requests identically can increase latency, cost, and the risk of incorrect responses.
This leads to an important design question:
What if an AI application could decide when to answer directly, when to retrieve evidence, when to use a tool, and when to ask for clarification?
In this article, we explore a practical architecture for an adaptive AI router using AWS services. We will examine its routing logic, security boundaries, observability, and a reproducible evaluation plan.
The goal is not to build an agent that performs the most reasoning. It is to build one that performs the right amount of reasoning for each request.
2. The Problem: One Workflow Does Not Fit Every Request
A common implementation sends every request through the same model and tool pipeline.
That approach is straightforward, but it can introduce unnecessary work.
| Challenge | Why it matters | Possible consequence |
|---|---|---|
| Overusing powerful models | Simple requests receive more processing than they need | Higher inference cost |
| Retrieving documents unnecessarily | Retrieval adds extra processing steps | Increased latency |
| Calling tools without need | Every tool introduces another dependency | More failure points |
| Answering without evidence | The model may lack the information required | Unsupported claims |
| Guessing when information is missing | The system hides uncertainty | Poor user trust |
| Logging too little | Failures become difficult to diagnose | Slower debugging |
A better design evaluates the requirements of each request before selecting an execution path.
The router should not merely ask, “How confident is the model?”
It should ask:
- Does this request require current or external information?
- Does it require private or source-specific evidence?
- Does it require an action?
- Is the request sufficiently clear?
- Is the requested action authorized?
- Can the system answer safely with the available information?
These questions turn routing into an explicit engineering decision rather than an uncontrolled model behavior.
3. Proposed Architecture
The proposed system uses a lightweight routing layer to select an appropriate workflow.
The architecture separates request handling, policy enforcement, execution, and observability.
| Component | AWS service | Responsibility |
|---|---|---|
| Application entry point | Amazon API Gateway | Accept requests and apply API-level controls |
| Routing and policy logic | AWS Lambda | Classify requests, enforce policies, and select a route |
| Foundation model | Amazon Bedrock | Generate responses using a suitable model |
| Knowledge retrieval | Amazon Bedrock Knowledge Bases and Amazon S3 | Retrieve relevant source material |
| Controlled actions | Allow-listed Lambda functions or approved APIs | Execute permitted operations |
| Monitoring | Amazon CloudWatch | Capture logs, metrics, and operational failures |
| Permissions | AWS Identity and Access Management (IAM) | Apply least-privilege access controls |
This is a proposed reference architecture, not a claim that a deployed production system has already been built or benchmarked.
The exact services can be adjusted to match the application. For example, a project that does not need document retrieval can omit the knowledge base component.
Why separate routing from execution?
Keeping routing separate makes the system easier to test and maintain.
The routing layer determines what should happen. The execution layer performs the selected operation. The policy layer checks whether that operation is allowed.
This separation helps prevent the language model from becoming the only authority responsible for choosing and executing actions.
4. How the Router Makes a Decision
The router can select one of four primary paths.
| Route | When to use it | Expected behavior |
|---|---|---|
| Direct answer | A clear, low-risk request that does not require external evidence | Answer with an appropriate model |
| Retrieval | The answer depends on documents or specific knowledge sources | Retrieve evidence and generate a grounded response |
| Tool execution | The request requires a supported external action or live data | Check authorization, validate inputs, and invoke an approved tool |
| Clarification or abstention | The request is ambiguous, unsafe, unsupported, or lacks necessary evidence | Ask a question or explain the limitation |
Routing should be based on requirements
A simplistic router might classify a request based only on keywords. That can be useful for a first prototype, but keywords alone are not enough for a robust application.
For example, the word “weather” might indicate a need for current information, while “explain how weather forecasting works” may only require a direct explanation.
A more capable router can combine:
- Deterministic rules for known requirements and security constraints.
- Structured classification for identifying intent and required capabilities.
- Policy checks before any action or access to protected data.
- Fallback behavior when the request cannot be classified safely.
The classification result should be treated as a routing proposal—not as authorization to perform an action.
5. Implementation: Start With Explicit Routing Rules
For an initial prototype, begin with deterministic routing rules. This makes the behavior easier to inspect and test before introducing more complex classification.
The screenshot illustrates an example route policy with four outcomes: direct, retrieval, tool, and clarification.
In a real implementation, the router would receive structured signals derived from the request and application context. These signals would then be evaluated against the routing policy.
A practical implementation should also define:
- Which routes are available for each request type.
- What evidence is required before answering.
- Which tool actions are permitted.
- How missing or conflicting information is handled.
- What happens if a dependency fails or times out.
Important design principle
Routing and authorization are different decisions.
A classifier may identify that a request needs an order-status tool. That does not mean the user is allowed to access every order.
The application must independently verify identity, permissions, input validity, and the scope of the requested action.
6. Secure Tool Execution
Tool calling is one of the most valuable capabilities of an AI agent, but it also introduces additional risks.
A model-generated tool name or argument should never be trusted automatically.
The code snapshot demonstrates an allow-list and authorization check before execution. It is illustrative; a production implementation must also validate arguments, enforce resource-level access, handle errors, and apply timeouts.
A secure tool execution sequence should follow this order:
- Identify the required tool.
- Verify that the tool is on the approved allow-list.
- Authenticate the user and authorize the specific operation.
- Validate the arguments against a defined schema.
- Execute the operation with least-privilege permissions.
- Record the outcome and return only permitted information.
| Risk | Recommended control |
|---|---|
| Unauthorized tool call | Allow-list tools and enforce authorization |
| Invalid arguments | Schema validation and input constraints |
| Excessive permissions | Least-privilege IAM roles |
| Repeated or expensive calls | Budgets, rate limits, and call limits |
| Destructive action | Explicit confirmation and additional policy checks |
| Sensitive information leakage | Data filtering and access-aware responses |
For high-impact actions, such as deleting resources or making financial changes, the application should require appropriate confirmation and authorization rather than relying on the model’s judgment alone.
7. Retrieval and Evidence-Grounded Answers
Some questions cannot be answered reliably from a model’s general knowledge alone.
Examples include questions about:
- Internal technical documentation.
- A company’s deployment procedures.
- A product-specific knowledge base.
- Documents that may have changed since the model was trained.
For these requests, the router can select a retrieval workflow using Amazon Bedrock Knowledge Bases and a supported data source, such as Amazon S3.
The workflow is:
- Retrieve relevant passages from the permitted knowledge source.
- Check whether the retrieved material is relevant to the question.
- Generate an answer using the available evidence.
- Provide citations or source references when supported by the application.
- Abstain or ask for clarification if the evidence is insufficient.
Retrieval does not automatically guarantee correctness. A system can retrieve irrelevant documents, miss critical context, or generate claims that the retrieved material does not support.
Evaluation should therefore measure both retrieval quality and answer groundedness.
8. When the Correct Answer Is “I Don’t Know”
A reliable AI system must distinguish between a missing answer and an answer that can be safely inferred.
Consider these situations:
| Situation | Recommended response |
|---|---|
| The request is ambiguous | Ask a targeted clarification question |
| Required information is unavailable | Explain what information is missing |
| Retrieved sources conflict | Describe the conflict or request verification |
| The user lacks permission | Decline the restricted action |
| A tool fails | Report the failure without inventing a result |
| Evidence does not support a conclusion | State the limitation instead of guessing |
Abstention is not simply refusing to answer. It is a deliberate outcome when the available evidence, permissions, or request details are insufficient.
The system should also distinguish between uncertainty about factual content and uncertainty about whether an action is authorized. The latter must be handled through policy checks, not model confidence.
9. Benchmarking the Router
An adaptive router should be evaluated against simpler alternatives to determine whether the additional complexity is worthwhile.
Compare three configurations:
| Configuration | Execution strategy | What it helps evaluate |
|---|---|---|
| Single-model baseline | Send each request to one selected model | Baseline quality, latency, and cost |
| Retrieval-first | Use retrieval for most or all requests | Effect of retrieval overhead and evidence availability |
| Adaptive router | Select a route based on request requirements | Whether routing improves the overall trade-off |
Build a reproducible evaluation dataset
A useful starting point is a test set of approximately 100 requests, distributed across four categories:
| Category | Suggested test cases | Example |
|---|---|---|
| Direct answers | 25 | Explain the difference between HTTP and HTTPS |
| Retrieval questions | 25 | Summarize a specific section of a supplied document |
| Tool requests | 25 | Retrieve an authorized order status using a mock tool |
| Clarification or abstention | 25 | Respond to an ambiguous request with missing information |
These counts are a proposed evaluation plan, not measured results. The dataset should include expected outcomes, relevant evidence where applicable, and a consistent scoring method.
Use the same test cases and comparable model settings across configurations. Record failures as well as successful requests.
Metrics to measure
| Metric | What it measures |
|---|---|
| Task success rate | Whether the system fulfilled the request correctly |
| Groundedness or citation pass rate | Whether evidence-based claims are supported by the retrieved sources |
| Average latency | Typical end-to-end response time |
| p95 latency | Response time for the slowest 5% of requests |
| Cost per 100 requests | Estimated model and supporting-service cost |
| Unnecessary tool-call rate | How often a tool is called when the task does not require it |
| Unsafe-action rate | Whether the system attempts or performs prohibited actions |
| Appropriate abstention rate | Whether it declines or clarifies when required |
Replace the
TBD values in the image with your measured results. Document the test date, model configuration, dataset, and scoring rules so readers can understand how the results were produced.Estimate the cost
For model inference, an approximate calculation is:
Model cost per request
= (input tokens × input price per token)
- (output tokens × output price per token)
For the full application, also account for applicable retrieval, storage, Lambda, API Gateway, logging, and other service charges.
Use the pricing applicable to the selected AWS Region, model, and service configuration. Actual costs depend on usage and pricing changes.
Do not assume that an adaptive router will always be cheaper. Classification itself has a cost, and retrieval or tool execution may be necessary for correctness. The goal is to measure whether the complete design improves the quality-cost-latency trade-off.
10. Observability: Make Routing Decisions Explainable
When a response is incorrect or unexpectedly expensive, developers need to understand which route was selected and why.
The example demonstrates structured logging for a request ID, selected route, routing reason, latency, token usage, tool calls, and outcome.
A production implementation can send appropriate logs and metrics to Amazon CloudWatch.
Recommended fields include:
| Field | Purpose |
|---|---|
| Request ID | Correlate events across components |
| Route | Identify the execution path |
| Route reason | Explain the policy or requirement that selected the path |
| Latency | Identify performance bottlenecks |
| Token usage | Support model cost analysis |
| Tool-call count | Detect unnecessary or repeated tool execution |
| Outcome | Record success, failure, clarification, or abstention |
Avoid logging secrets, credentials, raw sensitive documents, or unnecessary personal data. Use access controls, retention policies, and redaction appropriate to the application.
Observability should make system behavior understandable without creating a new privacy or security risk.
11. Common Failure Modes and Mitigations
| Failure mode | Why it happens | Mitigation |
|---|---|---|
| Direct route chosen for a live-data request | The router misses a requirement | Add explicit requirement detection and targeted tests |
| Retrieval returns irrelevant content | Poor matching or incomplete sources | Evaluate retrieval quality and relevance thresholds |
| Tool called without permission | Classification is mistaken for authorization | Enforce independent authorization checks |
| Router adds more latency than it saves | Classification is unnecessarily expensive | Keep simple rules deterministic and measure overhead |
| Model invents unsupported details | Evidence is missing or ignored | Require evidence for grounded claims and test abstention |
| Cost increases unexpectedly | Retries, long outputs, or excessive tool calls | Set budgets, limits, timeouts, and alerts |
No routing strategy eliminates every failure. The objective is to make failures observable, bounded, and recoverable.
12. A Practical Implementation Roadmap
Build the project incrementally instead of starting with a fully autonomous agent.
| Phase | Work | Deliverable |
|---|---|---|
| 1. Define the task | Choose the supported request types and expected outcomes | Test specification |
| 2. Build a baseline | Implement a single-model workflow | Baseline application |
| 3. Add routing | Introduce explicit route selection | Adaptive router |
| 4. Add retrieval | Connect a supported knowledge source | Evidence-based responses |
| 5. Add a mock tool | Implement an authorized, non-destructive tool | Controlled tool workflow |
| 6. Add observability | Record latency, route, tokens, and outcomes | Operational metrics |
| 7. Evaluate | Run the same dataset across all configurations | Completed comparison table |
| 8. Harden security | Test permissions, injection attempts, failures, and limits | Security test results |
Start locally with mocked model and tool interfaces if you want to validate the routing logic before provisioning AWS resources. Add real services only when each stage is testable.
13. Security and Responsible Design
The router should not be treated as a security boundary by itself.
A production implementation should address:
- Prompt injection: Treat retrieved documents and external content as untrusted input.
- Least privilege: Give each AWS component only the permissions it requires.
- Tool authorization: Verify permissions independently of model-generated decisions.
- Input validation: Validate tool arguments and constrain accepted values.
- Cost controls: Apply quotas, rate limits, timeouts, and monitoring.
- Data protection: Redact sensitive values and restrict log access.
- Human confirmation: Require appropriate approval for consequential actions.
- Failure handling: Ensure that timeouts and service errors do not produce fabricated success messages.
These controls are essential if the prototype evolves into a real application.
14. Conclusion
Building an AI agent is not just about choosing a model or connecting tools. It is about deciding which capabilities a request actually needs—and enforcing that decision safely.
An adaptive AI router provides a structured way to make that choice. Direct answers can remain lightweight, retrieval can provide evidence, tools can perform authorized operations, and clarification can prevent the system from guessing when important information is missing.
The key engineering principle is simple:
Use the minimum workflow that can satisfy the request reliably, and measure whether it actually improves quality, latency, and cost.
The next step is to implement the baseline, add the routing layer, and publish the measured comparison. That turns the idea from an architecture proposal into a reproducible engineering project.
References and Further Reading
- Amazon Bedrock: https://aws.amazon.com/bedrock/
- Amazon Bedrock Converse API: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html
- Amazon Bedrock Knowledge Bases: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html
- AWS Lambda: https://aws.amazon.com/lambda/
- Amazon CloudWatch: https://aws.amazon.com/cloudwatch/
- AWS Identity and Access Management: https://aws.amazon.com/iam/
- AWS Well-Architected Framework: https://aws.amazon.com/architecture/well-architected/
Suggested tags: Artificial Intelligence, Amazon Bedrock, Generative AI, AI Agents, AWS Architecture
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article