AWS Builder Center

Agents for Humans, the tool corrected its own author on day one

Three things that went wrong while building a page reader for the Agents for Humans hackathon, and what each one changed in the shape of the agent. The last one is the tool telling its author that eight of the twenty two contests he had cleared by hand were closed to him.

I built Asterisk for the Agents for Humans hackathon. It reads an offer page and reports where the page takes back its own headline. Free that becomes charged three sections down. A prize figure that a footnote says is not a prize. Open to everyone, except for four words beside the register button.
What follows is not the pitch. It is the three things that went wrong while I built it, because those are the parts worth reading.
## The starting audit was not the agent's decision
My first shape was the obvious one. Give the agent the tools, give it the page, let it decide what to do. The agent had a one page audit as one of its tools, and I asked it to check a page I had already read by hand. The page had three contradictions in it.
The agent called that page clean.
It had not run the audit. It read the page, formed an opinion, and answered from the opinion. Nothing in the run looked like a failure. There was no error, no empty result, no warning. Just a confident paragraph that happened to be wrong.
So the audit stopped being a tool the agent may call. It is now the first step of the pipeline, before any model is involved, and the agent receives its output rather than deciding whether to produce it. The model still does the part that needs judgement, reading one claim against candidate passages. It no longer decides whether the work happens.
If you take one thing from this post, take that. A step that must always run is not a tool. It is a step.
## A tool that lets a model attest to its own output has verified nothing
My second shape had a `verify_quote` tool. The model would produce a quote, call the tool, and the tool would say whether the quote appeared on the page. Clean idea. It failed in a way I did not expect.
The model called `verify_quote`, got its answer, then rephrased the quote in the sentence it wrote for the reader, and labelled that sentence verified. The label travelled. The text under it did not.
The fix was to stop trusting anything that happens before the last token. There is now an output guard that runs after generation, on the text the reader will actually see, and a quote absent from the source page is dropped rather than shown. The guard does not ask the model anything. It does not need to.
This is the same mistake as the first one wearing a different hat. Both times I had let the thing being checked participate in the checking.
## The longest document was not where the answer lived
While building the eligibility side of this, I read the full legal rules of twenty two contests by hand and concluded twelve were open to me. Then I built the tool and pointed it at the same pages. It came back with `Students only` on pages I had cleared myself three hours earlier.
The real number was eight.
I had read the longest document on every page and missed the shortest field, four words in a card next to the register button. The rules text did not contradict it. It simply did not mention it, and I had treated silence as permission.
The tool's first useful act was to correct its author, on work the author had done carefully, with a source the author had never thought to open. That is the moment I stopped calling it a demo.
## What it is built on
Python, and Strands Agents for the agent layer. The deterministic side uses no model at all, because legal formulas are one of the few things in this domain that were standardised, so they can be matched exactly, for free, and they invent nothing. The model runs only where the page is written by a human who never read a standard.
Sixty nine tests, and the ones I care about are the six built to be refused. A validator that rejects nothing has proved nothing about what it accepts.
The repository is public under MIT, the report gallery is published, and the numbers in the README are re-derived by the test suite rather than typed in, because I have now been burned twice by a number that was true when it was written.
## The part I would still change
The model runs on an OpenAI compatible endpoint rather than on Bedrock. That is honest and it is also a weakness in a hackathon whose sponsor is AWS. The agent layer is genuinely theirs, hooks and tool cancellation included, but the inference is not. I would rather write that sentence than quietly leave it out, since the whole value of this project is that it says what pages leave unsaid.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article