AWS Builder Center

Agents for Humans, why half of this agent refuses to use a model

Half of my Agents for Humans entry runs with no model at all, on purpose. What exact matching buys, where the model still earns its place, and the six tests written to be refused.

This is the second thing I wrote about Asterisk, my entry for the Agents for Humans hackathon. The first was about three design mistakes. This one is about a decision I keep having to defend, so I may as well write it down once.
Half of the tool does not call a model at all. That half is the half I trust.
## The two kinds of sentence on an offer page
Read enough terms pages and you notice they contain two different species of text.
The first species is standardised. A price with a currency. A notice period. A clause saying the fee applies after a named date. A jurisdiction. These were written by someone copying a formula that thousands of pages share, because the law or the industry settled the wording years ago. They vary in surface and not in shape.
The second species is not standardised at all. Someone in marketing wrote it on a Tuesday. It says the thing is free, or that support is included, or that you can cancel any time. There is no formula. There will never be a formula.
A model is the wrong instrument for the first species and the only instrument for the second.
## What the deterministic side buys
For the standardised sentences, exact matching wins on four counts at once, and I want to be specific rather than hand wave.
It is free. It runs on a page in milliseconds. It gives the same answer twice on the same input, which means a regression test on it is a real test rather than a mood reading. And it cannot invent, because a rule either matched a span of text on the page or it did not.
That last one is the one that matters for this project. The whole promise is that every quotation shown to the reader exists on the page. A matcher cannot break that promise. A model can break it while sounding certain, and I have watched it happen.
So the rules run first, with no model in the loop, and they produce the spans that anything downstream is allowed to talk about.
## Where the model earns its place
The model reads one claim against a small set of candidate passages and says whether the passage takes the claim back. That is a judgement about meaning between two pieces of text a human wrote freely. No rule I can write will do it, and I tried before I admitted it.
The important part is the shape of the request. The model is not asked to find contradictions on a page. It is asked, for one specific claim and one specific passage, whether the second undoes the first. Narrow question, small input, and the answer is checked afterwards against the page rather than trusted.
An agent layer sits above this, from Strands Agents, for the case where the reader points at a whole site rather than one page. It ranks which condition links are worth spending a request on and follows them. That is genuinely a job for an agent, because the set of pages is not known in advance and the cost of reading all of them is real.
## The tests I would show first
Sixty nine tests, no network, no key. Six of them exist to be refused, and those are the ones I would put in front of a reviewer.
A validator that only receives well formed input has proved nothing about what it rejects. So there are schemas built to be wrong in specific ways, and the test asserts the refusal, with the reason. When I later changed how the guard worked, those six caught two mistakes the happy path tests waved through.
There is also a check that reads the numbers printed in the README and fails the suite if they no longer match what the suite produces. That exists because I have twice published a number that was true on the day I typed it. A number in a document is a claim like any other, and the point of this whole project is that claims should be checkable.
## What this costs
Rules are work. Every family of standardised clause is a small piece of research and a set of witnesses. It is slower to start than handing everything to a model and hoping.
It pays back in the only currency this project has, which is a reader who can check me. The deterministic findings come with the exact span. The model findings come with the exact span too, because the guard drops any that do not.
Written for the Agents for Humans hackathon. The submission form asks contributors to tag their posts #AgentsofFootball, so here it is.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article