Agents for Humans: my README made a claim my code did not support
"The LLM orchestrates; it never touches a number." It was a good line and it was false. Here is how I found out, what I replaced it with, and the one-command test that lets anyone prove the replacement wrong.
The first version of my README said this:
The LLM orchestrates; it never touches a number.
It was a good line. It was also false, and it stayed on the front page of my repo for two days before someone made me check.
This post is about what "false" meant, what I replaced it with, and the flag I added so that the replacement can be proven wrong by anyone in about thirty seconds.
The claim and the check
Settlement Sentinel audits delivery-platform settlement statements for a small restaurant. The whole pitch rests on a separation: deterministic Python computes and decides, the language model only sequences the steps. If that separation is real, the project is trustworthy in a way a "let the LLM read your invoices" tool is not. If it is decorative, the project is the thing it claims not to be.
So the claim is doing a lot of work, which is exactly why I should have tested it earlier than I did.
I ran the sentence past a second model with an adversarial framing — not "is this good" but "find what is wrong with this." It came back with something I could verify in my own code in five minutes: tool results are returned to the model. Every figure the tools compute enters the model's context. The model does not calculate them, but "never touches a number" is not what "does not calculate" means.
Then I went looking and found something worse. This was in my system prompt:
For a 'what is pending' request, call list_pending_escalations("all") andshow each pending order as: platform, order id, amount at stake, then itsquestion_for_owner VERBATIM. Do not rewrite the question.
I was instructing the model to transcribe amounts into its output. On one code path, the number the owner read was a number the model had retyped. I had written "never touches a number" on the front page while shipping a path that asked it to.
Nobody caught this by reading the architecture diagram. It was caught by asking what would make the claim false.
Two ways to respond
When a claim turns out to be too strong there are two moves.
Weaken the claim until it is true. Fast, honest, and it costs you the property you wanted.
Change the system until the claim is true. Slower, and sometimes impossible.
I did some of each, which I think is the right split when you look at what each part costs.
The pending path was fixable, so I fixed it. Rendering moved into
report.py as a plain function — deliberately not a tool, so the model cannot call it and cannot see its output:def render_pending(pending: list) -> str: """Render the owner queue. Plain function (not a tool): the program prints this itself so the model never re-types an id, amount or question."""
Now the model's entire contribution to that command is one sentence: "There is 1 order currently awaiting your decision." Python prints the order id, the amount and the question. That path went from "the model retypes figures" to "the model counts."
Tool results entering context was not fixable, or not without breaking the agent. An agent that cannot see what its tools returned cannot decide what to do next. So the claim had to move:
Every number is computed, gated, and rendered by deterministic Python. The LLM only orchestrates.
Longer. Less quotable. True.
Prompts are not enforcement
Then I watched a real run and learned the same lesson a second time.
I had told the model, in the system prompt, not to re-type figures. It mostly complied. But its closing summary said:
One discrepancy was found and a dispute has been automatically filed fororder BM-1005, where Baemin overcharged commission by 1,000 KRW.
There it is. 1,000 KRW, retyped by the model, in the output the owner reads.
The number happened to be right. That is not the point. The point is that I had asked it not to, and it did, and nothing in the system stopped it — because nothing in the system could. A system prompt is a request. It is not a constraint.
I could have quietly removed that observation and left the claim looking clean. Instead it went into the README as a known limitation:
The model may quote a figure in its closing sentences. It cannot compute or gate one, and the rendered report — not the model's prose — is the record.
That is the honest boundary. Computation and gating are enforced by control flow; the model cannot reach them if it tries. Prose is not enforced, so I say so rather than pretending my prompt is a fence.
The general version: if a property must hold, it lives in code, and anything you can only ask for politely should be described as something you asked for politely.
Making the claim testable
A rewritten claim is still just a sentence. I wanted the reader to be able to check it without trusting me, so I added a flag:
python agent.py baemin --no-llm
This runs the same three files in the same order with no model in the loop at all. Not a mock, not a fixture — the same
load_settlement , recompute_deductions , flag_discrepancies , decide_actions , render_report , called directly.The test is a diff:
rm -f data/*.jsonl && python agent.py baemin > with_llm.txtrm -f data/*.jsonl && python agent.py baemin --no-llm > without_llm.txtdiff with_llm.txt without_llm.txt
Every row of every table is identical. The differences are the tool-call trace and the model's two closing sentences. If the model were computing anything, or influencing any verdict, removing it would change the table — and it does not.
The README says what happens if it ever does:
If a table row ever differs between these two runs, the claim on this page is false and should be treated as such.
Writing your own falsification condition into your documentation is uncomfortable, and I think that discomfort is the signal that it is worth doing. A claim with no stated way to fail is marketing. A claim with a one-command test is an engineering property.
The side effect I did not plan for
The flag turned out to solve a problem I had not thought about.
The hackathon rules require judges to be able to run the project. My project needs an Amazon Bedrock API key with access to a specific model — and I have since seen other participants report their accounts blocked from model access entirely. A judge in that position would have opened my repo and seen nothing work.
With
--no-llm they get the full pipeline, all three platforms, the gate refusing to act on the largest discrepancy, the owner queue — with no credentials and no network.I built it as an epistemics tool and it doubled as an accessibility one. Those turn out to be the same property from two directions: a system whose core behaviour can be demonstrated without its most expensive dependency is both easier to verify and easier to hand to a stranger.
What I would do earlier next time
Write the claim first, then attack it. I wrote the sentence because it sounded right, then spent two days building around it before testing whether it was true.
Ask "what would make this false" rather than "is this correct." The second question invites confirmation. The first one found the bug in five minutes.
Ship the falsification test with the claim. Not a paragraph explaining why you are trustworthy — one command that would embarrass you if you were wrong.
Distinguish enforced from requested, in writing. Control flow is a fence. A system prompt is a note on the fridge. Say which one you have.
Previous posts in this series: the one-line bug that made the auditor pass a 10.9% overcharge as clean , and why the agent refuses to file the biggest discrepancy it finds .
Built for the AWS Agents for Humans hackathon, Professional Agents track.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article