AWS Builder Center
An agent that knows when to stop: building CostPilot for Agents for Humans

An agent that knows when to stop: building CostPilot for Agents for Humans

CostPilot scans a Python codebase for LLM API calls, prices each one against public rate cards, and opens a pull request to swap an expensive model for a cheaper one. When the evidence does not clear the bar, it does nothing and says so.

Watch the demo run:

The hardest part of letting an LLM touch production code is not getting it to act. It is getting it to stop.
CostPilot, my entry for the Agents for Humans hackathon, scans a Python codebase for LLM API calls, prices each one against public rate cards, and opens a pull request to swap an expensive model for a cheaper one. When the evidence does not clear the bar, it does nothing and says so.
CostPilot is for engineering teams that already have working LLM-powered applications and want to automate the repeatable part of model-cost review without handing the repository to an autonomous coding agent. This is a build log about the decisions that shaped it, and about one observation that changed how I think about agent boundaries.

Finding the calls

The scanner walks the Python AST and matches  client.chat.completions.create(...)  chains, the Anthropic SDK equivalent, and a couple of LangChain constructors. For every call site it reports a structured fact: file, line, model, how the model was specified (literal, constant, environment, parameter default), and a confidence score. It prices each call against public rate cards. Unknown models come back unpriced, never estimated.
That last rule turned out to matter more than the scanner itself. On the vanna repository, the scanner found 20 call sites and could not price 19 of them, about 95 percent. Another repository in the same benchmark, pandas-ai, came back with every model priced. The unknown rate belongs to the codebase, not the tool. I could have made the rate table bigger or guessed a price for look-alike names. Both would have produced nicer numbers and a worse tool. The full run is reproducible from the benchmark report checked into the repo, priced against public rate tables as of August 2026.

Deciding over facts, not source

The decision step is a Strands Agents SDK agent with four tools:  list_candidates ,  get_candidate ,  decide ,  finish . It never reads a source file. It reads this per call site:
{
"file": "sample_app.py",
"line": 14,
"model": "gpt-4o",
"model_source": "literal", # literal | constant | env | parameter
"confidence": 0.98,
"cost_per_1k_calls": 1.01,
}
The choice to feed typed facts instead of source text did two things I did not fully plan for.
The first is about attack surface. The agent never sees raw source text. It sees a normalized model-name field that the scanner extracted from the AST. The repository is untrusted input, so an attacker can write anything into that field, and I will not claim the system is immune to that. What the design buys is data minimization: the model-facing surface is one short field instead of a whole file. That does not eliminate prompt injection. It makes the surface small enough to reason about, which raw source text never is.
The second consequence was bigger: the decision layer became testable without a model. Because the input is a typed schema, the entire pipeline runs under pytest. Sixty-six tests, no network, no API bill. They are not spread evenly: 19 cover the scanner and the structured facts, 16 cover the git policy gate and rollback, 13 cover pricing honesty and the downgrade policy, 10 cover the agent loop and the decide tool, and the last 8 cover the demo endpoint and its contract tests.When I changed the verdict logic, the affected decision tests failed and showed me which expectations had changed.Anyone who has tried to write a test for a free-text LLM call knows what that buys.
Figure 1. The evidence schema is a security boundary and a testing boundary at the same time.

The agent ends every review by calling  decide  with one of three verdicts:  approve ,  keep , or  ask_human . The third one exists for the cases the agent refuses to settle on its own. In my recorded run, the agent screened five call sites and approved exactly one downgrade: a  gpt-4o  call with a literal model string moved to  gpt-4o-mini , 94 percent lower in modeled cost per 1K calls. The other four stayed untouched. Keep, in this system, has a precise meaning: the change does not happen, and the reason ships with the verdict. Among them were a model loaded from a shared constant and a call already running the cheapest model.

The observation that changed the design

The pull request and the recorded run are two real runs of the same agent. In the pull request run, it approved three swaps, including a Claude call, for 80.3 percent lower modeled cost. In the recorded run, it approved one swap and kept its Claude call. I did not get the same answer twice.
Run
Result
Pull request #1
approved 3 swaps, 80.3 percent lower modeled cost
Recorded run
approved 1 of 5 call sites, kept the other 4
LLMs are not deterministic. Once I stopped pretending otherwise, the architecture followed: anything the LLM approves still has to clear a deterministic gate.
That gate is an allowlist, boring on purpose. Branches must start with  costpilot/ , commits with  costpilot: , and only a short list of git subcommands is allowed. If the repo's own tests fail after the patch, the working tree is rolled back. No commit, no pull request.
The gate earned its place on live data. The LLM once approved a downgrade on a constant-derived model. From the facts alone, the agent had no way to know that changing one constant changes every caller. The rule layer blocked it. When the judge itself can vary between runs, that gate is the floor under the whole system.


What can be proven, and what cannot

CostPilot's central claim is narrow on purpose, and the word prove means exactly one thing here: a modeled cost reduction under four explicit conditions. Three of them gate the proposal: both models are in the public price table, the model is a string literal at the call site, and the prompt is short with bounded output. The fourth, a green test run, is different in kind. Tests run after the patch is applied, and a green run is what turns a proposal into a commit. That ordering is the whole point, and the artifact records it.
Four criteria is a small list, and that is the point. They are the properties a machine can verify exhaustively, every time, without judgment.
The properties a machine cannot verify are listed on the pull request itself as the human's checklist: output quality, refusal rate, JSON schema stability, tool-calling behavior, latency, and token-usage variance of the cheaper model. The artifact never claims the two models are semantically equivalent. Cheap proofs belong to the machine; expensive ones stay with the human. Drawing that line explicitly, instead of pretending the machine covers both, is what makes the approval worth anything.


The pull request, and what happened next

The recorded pull request is real: three swaps, about 80 percent lower modeled cost, tests passed, opened by the agent. Modeled cost is arithmetic on public token prices for the same call shape, compared before and after the swap. It is not a measurement of real traffic, and the artifact says so. The validation section reports what actually ran. An artifact that hides a weak validation record is worse than one that prints it.

The human half of the loop is now on the record too. The review on the pull request goes through the four criteria, confirms the math and the tests, and then does the thing the system was built to make possible: it declines to merge, because output quality has not been measured. Across the pull request and the recorded run, the human was asked to make one kind of decision: whether the cheaper model behaves the same. The agent took the repetitive screening. The human took the judgment.
The math is approved. The behavior check is mine, and it is still open. That is the deal the agent signed up for.

Why an agent at all

A fair question: if deterministic rules check every verdict, why involve an agent? A plain script would pass the same four tests. The honest answer starts with the inputs. On vanna, 19 of 20 call sites had a model the scanner could not price. A static pipeline can reject those cases, and it should. What it cannot produce is the judgment record I want from the decision layer: what was looked at, what was decided, and why, written down where a human can review it and push back. A linter prints a warning. The agent writes a decision record, and the artifact is that record. The rules decide what the agent is allowed to do. The agent decides which candidates deserve attention. That division is the design. The agent layer can be swapped or upgraded without touching the enforcement layer, as long as the evidence contract and the policy boundary stay where they are.

Notes on Strands

The  decide  tool is the only exit. That constraint is what makes the three-verdict scheme work: the agent cannot ramble, summarize, or hedge. It must choose  approve ,  keep , or  ask_human , and it must attach the typed facts that led to the choice. A tool that only returns decisions is the reason the whole pipeline can run under pytest with no model and no network.
Why Strands rather than a hand-rolled Python loop? The tool loop with a forced terminal decision is the exact shape I wanted, and I did not have to rebuild it. Strands also keeps the model provider behind one interface, which is why the provider switch is a non-event. CostPilot runs on a local Ollama model by default and moves to Amazon Bedrock with one environment variable. Building against a local model kept development offline, free, and reproducible. The point of Bedrock is not extra features: the production provider changes, and the evidence schema and the policy layer do not. The safety properties live below the model, so swapping the model above them does not touch them.

Try it

The repo is at  github.com/owain323/costpilot . Clone it, install from the lock file, and run the checks: four gates, sixty-six tests, green in under three minutes on a modest laptop. The recorded run's decision artifact is checked in at benchmark/artifacts/costpilot__optimize.md, and the benchmark report with the per-site numbers is at benchmark/benchmark-report.json. The read-only scanner is live at  costpilot.owain32380.cn . The pull request is at  github.com/owain323/costpilot-demo/pull/1 , with the human review attached.
The system is deliberately incomplete, and that is the design. The agent owns the ambiguous part. Deterministic code owns the irreversible part. Tests own the regressions. Humans own the semantic part.
An agent that knows when to stop is not the safest system possible. It is the safest system we can build while still being useful.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article