One User Question Can Trigger 20 Tool Calls. That's the Hidden Cost of AI Agents.

I used to think one question to an AI agent meant one request to a model. Then I learned how tool calling, repeated reasoning, and large context windows can turn a simple question into multiple model calls and unnecessary latency. In this article, I explore why AI agents can become expensive, how programmatic tool calling works, and how I want to build AI systems that are faster, cheaper, and easier to control.

Pranav Gawai

Published Oct 16, 2026

Last modified Oct 16, 2026

I used to think building an AI agent was mostly about connecting a language model to a few tools.

Give it access to an API.

Let it retrieve information.

Allow it to call a database.

Give it enough context to make a decision.

Then let it answer the user.

Simple enough.

But the more I learned about agentic AI, the more I started thinking about a problem that isn't always obvious in a demo:

How much work does an AI agent actually perform before it gives you one answer?

Imagine asking an AI assistant:

Which members of my engineering team exceeded their travel budget last quarter?

The answer sounds simple.

But think about what the application might need to do.

It might retrieve a list of employees, fetch expense records, look up budget limits, calculate totals, compare the results, and finally generate a summary.

One question.

Multiple operations.

Potentially multiple model calls.

And that is where things start getting interesting.

One Question Doesn't Always Mean One Model Call

My original mental model looked like this:

1
2
3
4
5
6
7
User
  |
  v
LLM
  |
  v
Answer

But an agent that uses tools might work more like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
User Question
      |
      v
   Model Call 1
      |
      v
   Select a Tool
      |
      v
   Execute Tool
      |
      v
 Return Tool Result
      |
      v
   Model Call 2
      |
      v
   Select Another Tool
      |
      v
   Execute Tool
      |
      v
 Return Tool Result
      |
      v
   Model Call 3
      |
      v
   Final Answer

This is a simplified example, not a fixed execution pattern. Some agents can invoke several tools in parallel, while others need sequential calls because one result determines the next action.

The important point is that one user request can involve multiple model invocations and tool executions.

And every additional step creates more opportunities for latency, errors, and unnecessary work.

Let's Take a Real Example

In an AWS article published in May 2026, AWS engineers explained the bottlenecks that can occur when an AI model uses tools to analyze employee travel expenses.

Consider a task involving 20 employees.

A traditional tool-calling workflow could retrieve the employee list, fetch expense records through 20 separate tool calls, retrieve budget thresholds, and then reason over the results.

The workflow can end up passing more than 2,000 expense records through the model's context, even though most of those records may not be needed in the final answer.

Source: Implementing Programmatic Tool Calling on Amazon Bedrock

That example changed how I started thinking about AI agents.

The problem isn't necessarily that the model is bad.

The problem might be that we are making the model do work that ordinary code could handle more efficiently.

The Hidden Cost of Repeated Tool Calling

Let's break the problem down.

1. Every model round trip takes time

In a traditional tool-calling loop, the model chooses a tool, the application executes it, and the result goes back to the model.

The model then determines what happens next.

For a multi-step task, this sequence can repeat multiple times.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Model
  |
  v
Tool A
  |
  v
Model
  |
  v
Tool B
  |
  v
Model
  |
  v
Tool C
  |
  v
Model

If the calls must happen sequentially, the latency from each round trip accumulates.

Some tool calls can run in parallel, which helps. But parallel execution is not always possible, especially when one operation depends on the result of another.

This is why I started looking beyond just the speed of the model itself.

Sometimes the architecture around the model is the bottleneck.

2. Tool results can consume a lot of tokens

Imagine a tool returns 1,000 records.

The model might only need five of them.

But if the application sends all 1,000 records back to the model, that information can occupy a large amount of context.

1
2
3
4
5
6
7
8
9
10
Tool returns 1,000 records
           |
           v
All records enter model context
           |
           v
Model filters the data
           |
           v
Answer uses only five records

A lot of information was processed even though most of it wasn't necessary for the final response.

That can increase token consumption and make the reasoning task harder.

A better architecture might filter the records in ordinary code before sending a summary to the model.

3. More steps mean more failure opportunities

Suppose an agent performs ten operations.

One API times out.

Another returns incomplete data.

A third returns a response in an unexpected format.

Even if each individual component is reasonably reliable, a long workflow introduces more places where something can go wrong.

The agent may need to retry a tool, choose an alternative, or reconsider its plan.

Those retries can create additional model calls and execution time.

So I don't just want to ask:

How accurate is my model?

I also want to ask:

How many steps does my architecture require to solve this problem?

What Actually Costs Money in an AI Agent?

This was another thing I needed to understand.

A typical AI agent's expenses are not limited to the model.

Depending on its architecture, the application may incur costs from:

Input and output tokens.
Multiple model invocations.
Lambda executions.
Step Functions state transitions.
Database reads and writes.
Vector retrieval and knowledge-base operations.
API calls and data transfer.
Agent runtime and sandbox execution.

The exact bill depends on which services are used, how much work they perform, the selected model, the pricing plan, and the AWS Region.

A simplified way to think about it is:

1
2
3
4
5
6
7
8
9
10
11
Total Application Cost
          |
          +-- Model inference
          |
          +-- Tool execution
          |
          +-- Data retrieval
          |
          +-- Workflow orchestration
          |
          +-- Storage and transfer

This is why optimizing only the model's token price may not be enough.

If an agent repeatedly retrieves the same data, calls tools unnecessarily, or executes a long workflow for a simple task, the overall architecture can still be inefficient.

AWS covers these cost drivers in its serverless agentic AI cost optimization guidance .

The Interesting Idea: Programmatic Tool Calling

This was the concept that caught my attention.

Programmatic tool calling, or PTC, changes how an agent can coordinate multiple tools.

Instead of requiring the model to select a tool, receive its result, and reason again after every step, the model can generate code that orchestrates multiple tool calls in a sandboxed execution environment.

The code can handle loops, conditionals, parallel calls, filtering, and aggregation.

The result is that a sequence of operations can often happen without sending every intermediate result back through the model.

Conceptually, the difference looks like this.

Traditional tool calling

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
User Question
      |
      v
    Model
      |
      v
   Call Tool A
      |
      v
   Return Data
      |
      v
    Model
      |
      v
   Call Tool B
      |
      v
   Return Data
      |
      v
    Model
      |
      v
   Call Tool C
      |
      v
   Return Data
      |
      v
    Model
      |
      v
  Final Answer

Programmatic tool calling

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
User Question
      |
      v
    Model
      |
      v
Generate Execution Code
      |
      v
Sandboxed Environment
      |
      +----> Call Tool A
      |
      +----> Call Tool B
      |
      +----> Call Tool C
      |
      v
Filter and Process Results
      |
      v
Return Compact Result
      |
      v
    Model
      |
      v
  Final Answer

Instead of repeatedly asking the model to orchestrate each operation, the generated code coordinates the work and returns only the information needed for the final answer.

AWS describes this pattern in its Amazon Bedrock programmatic tool calling walkthrough .

The key insight for me is simple:

Use the model for reasoning. Use ordinary code for predictable computation and orchestration.

Why Not Use Programmatic Tool Calling for Everything?

This is where I think it's important not to blindly adopt a new technique just because it sounds more efficient.

Traditional tool calling is perfectly reasonable when an agent only needs one or two tools, or when each decision depends on the previous result.

Programmatic tool calling becomes especially interesting when the task involves many tools, repeated data retrieval, loops, calculations, filtering, or aggregation.

Consider two different requests.

Request A

What's the weather in Pune?

One appropriately selected weather tool call may be enough.

Introducing a code execution environment would probably add unnecessary complexity.

Request B

Compare the monthly expenses of 50 employees, retrieve their budgets, identify the people exceeding their limits, and calculate the differences.

This request involves multiple data retrieval operations and deterministic calculations.

It may benefit from coordinating the operations in code, running independent calls in parallel, and returning only a concise summary to the model.

The right choice depends on the workflow.

I don't want my architecture to become more complicated than the problem requires.

Ordinary Code Is Better at Some Things

This was one of the most useful lessons from this topic.

Language models are useful for interpreting natural language, understanding ambiguity, summarizing information, and making decisions that benefit from reasoning.

But many tasks do not require a language model.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Sum these numbers
        |
        v
Python

Filter records where status = APPROVED
        |
        v
Database query or ordinary code

Sort these records by timestamp
        |
        v
Code

Explain what these results mean
        |
        v
Language model

Why ask a model to reason through a calculation that a short, deterministic function can perform exactly?

Why send every database record to a model if SQL can return the relevant rows?

Why use an expensive reasoning model to format a predictable response?

I'm learning that building better AI applications often means knowing when not to use AI.

A Better AI Architecture

Suppose I am building an AI analytics assistant.

A user asks:

Which products had the biggest increase in sales this month?

A naive approach might look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
User
  |
  v
Model
  |
  v
Get Products
  |
  v
Model
  |
  v
Get Sales for Product A
  |
  v
Model
  |
  v
Get Sales for Product B
  |
  v
Model
  |
  v
Repeat
  |
  v
Model Summarizes Everything

This creates many opportunities for repeated work.

A more efficient architecture could be:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
User Question
      |
      v
Understand the Request
      |
      v
Query the Database
      |
      v
Calculate Sales Changes in Code
      |
      v
Rank Products
      |
      v
Return Top Results
      |
      v
Model Explains the Findings

Now the database performs data retrieval.

Code performs arithmetic and sorting.

The model provides the interpretation that the user actually wants.

Depending on the implementation, programmatic tool calling could coordinate several underlying tools inside a controlled execution environment.

That is a more deliberate division of responsibilities.

But Generated Code Creates a Security Problem

There is an important catch.

If a model generates code that executes tools, I must not treat that code as automatically safe.

Code execution introduces a different set of risks.

I need to consider:

Running generated code inside an isolated sandbox.
Restricting which tools the code can access.
Applying least-privilege permissions.
Limiting execution time and resource consumption.
Validating tool arguments and returned data.
Preventing unauthorized network access.
Requiring approval for sensitive actions when appropriate.

The sandbox should expose only the capabilities needed for the task.

For example, a reporting assistant might have permission to read sales data but should not automatically have permission to delete customer records.

1
2
3
4
5
6
7
8
9
10
Generated Code
      |
      v
Sandbox
      |
      v
Restricted Tool Access
      |
      v
Validated Operations

Programmatic tool calling is an architectural technique, not permission to execute arbitrary model-generated code with unrestricted access.

AWS's implementation guide demonstrates sandboxed approaches, including isolated containers and Amazon Bedrock AgentCore Code Interpreter.

How Amazon Bedrock Fits Into This

Since I've been learning AWS, this is where the topic becomes especially interesting to me.

Amazon Bedrock provides model inference capabilities and supports tool use in applications.

AWS's May 2026 technical walkthrough describes several approaches to implementing programmatic tool calling with Bedrock, including a self-hosted Docker sandbox and a managed approach using Amazon Bedrock AgentCore Code Interpreter.

The broader engineering lesson applies beyond one particular model or framework.

When an agent needs several tools, there is more than one way to orchestrate them.

The choice affects:

1
2
3
4
5
Latency
Token consumption
Execution complexity
Security boundaries
Cost

I want to understand the trade-offs rather than assuming that every agent needs the same architecture.

Measure the Agent, Don't Guess

This connects directly to my earlier article about AI observability.

If I want to optimize an AI application, I need to know what's happening inside a real request.

Useful things to measure include:

1
2
3
4
5
6
7
8
9
Model invocations per user request
Input tokens per request
Output tokens per request
Number of tool calls
Time spent in each tool
Total end-to-end latency
Failed tool calls
Retries
Estimated cost per completed task

Amazon Bedrock publishes runtime metrics through Amazon CloudWatch, including invocation counts, latency, input tokens, output tokens, and error-related measurements.

Invocation logging can also record supported model request and response details when configured.

Official documentation:

I would also track those measurements against the actual task being completed.

For example, knowing that a model used 20,000 tokens is less useful than knowing it used 20,000 tokens to answer a question that could have been solved with 2,000.

The goal isn't to minimize every metric at any cost.

The goal is to understand how much time and money the application spends producing a useful result.

The Optimization Checklist I Want to Follow

Before calling an AI agent production-ready, I want to ask a few questions.

1. Does this step really need a model?

If ordinary code or a database query can reliably perform the task, I should consider using that instead.

2. Am I retrieving more information than I need?

Returning the right 10 records is often more useful than returning 10,000 and asking a model to filter them.

3. Can independent tool calls run in parallel?

If two operations don't depend on each other, parallel execution may reduce end-to-end latency.

4. Is the agent calling the same tool repeatedly?

Repeated calls can indicate unnecessary reasoning steps, poor tool descriptions, or missing cached results.

5. Have I placed limits on execution?

I want limits on model iterations, tool calls, execution time, and other resources wherever appropriate.

6. Can I observe and attribute costs?

I want enough telemetry to understand which application, workflow, or request is consuming the resources.

7. Does optimization preserve correctness?

A faster agent that produces incorrect answers is not an improvement.

I want to evaluate quality alongside cost and latency.

What I Want to Build

I want to create a small experiment comparing two ways to solve the same multi-tool task.

For example:

1
2
Task:
Analyze sales data from multiple sources.

Version 1: Traditional Tool Calling

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Model
  |
  v
Call Tool
  |
  v
Read Result
  |
  v
Model
  |
  v
Call Another Tool
  |
  v
Read Result
  |
  v
Model
  |
  v
Final Answer

Version 2: Programmatic Tool Calling

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Model
  |
  v
Generate Execution Plan
  |
  v
Sandboxed Code
  |
  +----> Retrieve Data
  |
  +----> Run Calculations
  |
  +----> Filter Results
  |
  v
Compact Summary
  |
  v
Model
  |
  v
Final Answer

Then I would compare:

Total model invocations.
Input and output tokens.
End-to-end latency.
Tool failures.
Accuracy of the final answer.
Overall cost of completing the task.

I wouldn't assume the second version wins every time.

I would measure both approaches and decide based on the results.

That feels like a better way to learn AI engineering than simply adding more tools to an agent and hoping the system becomes smarter.

My Current Mental Model

I'm starting to think about an AI agent as three different parts.

1
2
3
4
5
6
7
8
9
10
11
12
                 AI Agent
                    |
          +---------+---------+
          |         |         |
          v         v         v
       Reasoning   Tools    Execution
          |         |         |
          v         v         v
       Model      APIs      Code
                    |
                    v
              External Data

The model helps understand the request and make decisions.

Tools expose capabilities and information.

Ordinary code handles predictable operations.

The execution environment coordinates those capabilities while enforcing security and resource limits.

Then observability tells me how the complete system behaved.

This is a much more useful mental model than treating the LLM as the entire application.

What I Understand Now

When I first started learning AI agents, I thought the main goal was to give the model more tools.

More APIs.

More data.

More capabilities.

But now I'm beginning to understand that more capability doesn't automatically mean a better system.

A useful agent needs to know which tools to use, what information to retrieve, when to stop, and how to return a result without performing unnecessary work.

Sometimes that means adding a tool.

Sometimes it means removing a tool.

Sometimes it means replacing repeated model reasoning with a straightforward database query or a few lines of code.

And sometimes, as AWS's programmatic tool calling work demonstrates, it means changing how the tools are orchestrated in the first place.

My biggest takeaway is simple:

The best AI architecture isn't the one that makes the model do everything. It's the one that gives each component the right job.

I'm still learning how to measure and optimize these systems, but that's the direction I want to explore next: building AI agents that are not only capable, but also efficient, observable, and practical to operate.

Builder Center