Agents for Humans: a Strands Model provider that makes your whole agent loop unit-testable
A real Strands Model provider that replays a script instead of calling a backend, so the whole agent loop is assertable offline with no credentials.
I am building Second Budget for the Agents for Humans hackathon, Good Neighbor Agents track. A benefits navigator sits with a household holding a SNAP notice of action, and the household asks the only question that matters: is this number right? Today nobody checks. Second Budget re-derives the allotment from the underlying facts and, when it disagrees, says what would have to be true about the household for the notice to be correct.
The claim the whole thing rests on is that its decisions are provable. Which makes the first thing I built not the agent, and not the budget engine, but a
Model provider that returns a script.Right now the suite is 163 tests, green, in under two seconds, with no AWS credentials in the environment — exercising the real graph, real edge conditions, real interventions, real interrupts. This post is that provider, in full (
src/second_budget/models/scripted.py ), and what it buys.The problem with testing an agent
An agent whose behaviour you can only observe by paying a provider is not proven, it is sampled. Two runs of the same graph with the same prompt can take different paths, and the interesting questions are not about the prose at all:
- Did the cycle converge in three iterations, or did it converge because the execution cap stopped it?
- Did my
end_turnactually halt the loop, or did the model just happen to stop?
- Did the intervention
Denyreach the model, and did the model retry?
- Does the interrupt resume path put the facts in the ledger?
Each of those is a property of the loop, and none of them needs a real model to answer. What they need is a model that does exactly what you tell it to.
The provider
Model is a small interface: its four abstract methods are stream , structured_output , update_config , get_config . Implementing it honestly — emitting the same StreamEvent sequence Bedrock emits — means the agent loop cannot tell the difference. That distinction is the entire point. A mock that bypasses the loop proves nothing about the loop.from dataclasses import dataclass, fieldfrom typing import Any
class ScriptExhausted(RuntimeError): """The agent asked for one more turn than the script provides."""
@dataclassclass Turn: """One model response: free text, or one or more tool calls."""
text: str | None = None tool_calls: list[tuple[str, dict[str, Any]]] = field(default_factory=list) stop_reason: str | None = None
@classmethod def say(cls, text: str) -> "Turn": return cls(text=text, stop_reason="end_turn")
@classmethod def tool(cls, name: str, payload: dict[str, Any]) -> "Turn": return cls(tool_calls=[(name, payload)], stop_reason="tool_use")
@classmethod def tools(cls, *calls: tuple[str, dict[str, Any]]) -> "Turn": return cls(tool_calls=list(calls), stop_reason="tool_use")And the provider itself:
import jsonfrom collections.abc import AsyncGenerator, AsyncIterablefrom typing import Any, TypeVar
from pydantic import BaseModelfrom strands.models.model import Modelfrom strands.types.content import Messagesfrom strands.types.streaming import StreamEventfrom strands.types.tools import ToolChoice, ToolSpec
T = TypeVar("T", bound=BaseModel)
class ScriptedModel(Model): """Deterministic ``Model``: replays ``turns`` in order, counting calls."""
def __init__(self, turns: list[Turn], *, model_id: str = "scripted") -> None: self._turns = list(turns) self._config: dict[str, Any] = {"model_id": model_id} self.calls = 0 self.seen_messages: list[Messages] = []
def update_config(self, **model_config: Any) -> None: self._config.update(model_config)
def get_config(self) -> dict[str, Any]: return dict(self._config)
async def structured_output( self, output_model: type[T], prompt: Messages, system_prompt: str | None = None, **kwargs: Any, ) -> AsyncGenerator[dict[str, T | Any], None]: turn = self._next() if len(turn.tool_calls) != 1: raise ScriptExhausted( "structured_output needs a turn with exactly one tool call " "carrying the payload" ) yield {"output": output_model.model_validate(turn.tool_calls[0][1])}
async def stream( self, messages: Messages, tool_specs: list[ToolSpec] | None = None, system_prompt: str | None = None, *, tool_choice: ToolChoice | None = None, **kwargs: Any, ) -> AsyncIterable[StreamEvent]: self.seen_messages.append(messages) turn = self._next()
yield {"messageStart": {"role": "assistant"}}
index = 0 if turn.text: yield {"contentBlockStart": {"contentBlockIndex": index, "start": {}}} yield {"contentBlockDelta": { "contentBlockIndex": index, "delta": {"text": turn.text}}} yield {"contentBlockStop": {"contentBlockIndex": index}} index += 1
for n, (name, payload) in enumerate(turn.tool_calls): yield {"contentBlockStart": { "contentBlockIndex": index, "start": {"toolUse": { "name": name, "toolUseId": f"scripted-{self.calls}-{n}"}}}} yield {"contentBlockDelta": { "contentBlockIndex": index, "delta": {"toolUse": {"input": json.dumps(payload)}}}} yield {"contentBlockStop": {"contentBlockIndex": index}} index += 1
stop = turn.stop_reason or ("tool_use" if turn.tool_calls else "end_turn") yield {"messageStop": {"stopReason": stop}} yield {"metadata": { "usage": {"inputTokens": 0, "outputTokens": 0, "totalTokens": 0}, "metrics": {"latencyMs": 0}}}
def _next(self) -> Turn: if self.calls >= len(self._turns): raise ScriptExhausted( f"the agent asked for turn {self.calls + 1} but the script has " f"{len(self._turns)}. Either the loop is not terminating, or " f"the script is short." ) turn = self._turns[self.calls] self.calls += 1 return turnThat is the whole thing. Usage is one line:
agent = Agent(model=ScriptedModel([ Turn.tool("record_fact", {"fact_id": "shelter.rent", "value": 900}), Turn.say("I have what I need."),]), tools=[record_fact], callback_handler=None)Two design choices are load-bearing.
self.calls counts model invocations, so a test can assert on how many times the loop went round. And running off the end of the script raises rather than producing a benign default turn — a silent fallback would let a runaway loop look like a passing test, which is precisely the failure a scripted provider exists to catch.Four things it lets me assert
That
end_turn really halted the loop. My HaltWhenFrontierCloses hook sets event.end_turn on AfterToolsEvent once the recorded facts satisfy the budget, so the agent does not make one more paid round-trip whose only possible output is "I have everything". "The loop halted" is otherwise indistinguishable from "the model happened to stop". The counter distinguishes them:def test_the_halt_saves_a_model_call_and_the_counter_proves_it() -> None: ledger = FactLedger() halt = HaltWhenFrontierCloses(ledger) # The script offers a second turn. If the halt works, it is never consumed. agent, model = _agent( [Turn.tools(*[_call(f, v) for f, v in ALL_FACTS]), Turn.say("this turn must never be reached")], ledger, [halt], )
agent("Record what the household told me.")
assert halt.halted is True assert model.calls == 1, "the loop asked the model again after it had everything" assert len(ledger) == 12That the cyclic graph converges in exactly three iterations. Second Budget's elicitation loop is a bounded cyclic
GraphBuilder graph, because the required-fact set is conditional: learning that a member is elderly or disabled adds a medical-expense requirement that did not exist a moment earlier. So I script an elicitor that dribbles facts out over three rounds, with the elderly fact arriving in round two, and assert on the topology the real graph actually walked:order = [node.node_id for node in result.execution_order]assert order == ["elicitor", "engine", "elicitor", "engine", "elicitor", "engine", "drafter"], orderassert engine.runs == 3assert result.status is Status.COMPLETEDThat last line is not decoration. A graph limit breach does not raise:
multiagent/graph.py:787-790 sets status = FAILED and returns. I ran the runaway case and read the result off it — failed_nodes is an int and is 0 , every one of the 14 node results still says completed , and only status carries the breach. if result.failed_nodes: misses a runaway entirely. Scripting a model that records the same fact forever is how I test the non-convergent path at all.A full interrupt raise-and-resume cycle, offline.
BeforeToolsEvent raises one interrupt carrying a whole twelve-fact batch, with the facts the model inferred marked for scrutiny. Two tests drive the two halves; trimmed together they read:paused = agent("Record what the household told me.")
assert paused.stop_reason == "interrupt"assert len(paused.interrupts) == 1, "twelve facts must cost one gate, not twelve"assert len(ledger) == 0 # nothing was written before a human saw it
agent([ {"interruptResponse": { "interruptId": paused.interrupts[0].id, "response": {"rejected": [FactId.EARNED_INCOME.value]}, }}])known = {f: ledger.value(f) for f in ledger.established}assert not is_closed(known), "a rejected fact must leave the frontier open"An intervention
Deny , and the model's retry after it. NumbersGate denies any tool payload carrying a currency token absent from a frozen engine certificate — set membership, decidable. The script gives the model a fabricated figure first and a real one second:agent("Draft the request.")
assert written == [f"The allotment is ${allotment}."]assert gate.denials == [("write_finding", (1800.0,))]# The model was told, retried, and succeeded -- three calls, not a crash.assert model.calls == 3A
Deny at before_tool_call cancels the call and hands the reason back as the tool result; a sibling test asserts that result comes through with status == "error" and the reason text intact. Nothing in the framework bounds how often the model may then try again — the per-invocation Limits in types/agent.py are opt-in and unset unless a caller passes them. So the denial reason is the model's entire steering signal and has to be an instruction rather than a complaint. Whether it works as an instruction is a question for a real model; whether the gate fires, blocks, and leaves the agent alive is a question this test settles for good.One sharp edge
The event loop catches provider exceptions and re-raises them as
EventLoopException ( event_loop/event_loop.py:404 ). A test waiting for the bare exception never sees it. The original is preserved as __cause__ :with pytest.raises(EventLoopException) as caught: agent("go")assert isinstance(caught.value.__cause__, ScriptExhausted)This is defensible framework behaviour — one exception type at the loop boundary is easier to handle in production than every provider's own — but it costs ten confusing minutes if you have not met it.
The cost angle, said plainly
A hackathon budget is free credits. The failure mode is not one expensive test, it is a cyclic graph with a condition that never flips, discovered in the morning. Two things protect against it here, and both are free: a script that raises the moment the agent asks for a turn it does not have, and
result.status is Status.COMPLETED as the only correct convergence check. I develop the graph, the gates and the interrupts against zero-cost turns, and spend real tokens only on the question real tokens can answer.When not to use it
A scripted model proves the harness, not the prompt. Nothing in those 163 tests tells me whether the elicitor asks a good question, or whether the model can turn a household's narrative into typed facts. That needs real runs, and I ran them on
us.amazon.nova-pro-v1:0 in us-east-1 .They found a bug the scripted suite structurally could not. "She spent sixty dollars on her inhaler" went into the budget as a sixty-dollar medical deduction. Only the portion above $35 is deductible — and the USDA microdata I validate the engine against already carries that column net of the threshold, so the validation path was always right and only the narrative path was wrong. The rule now lives in the engine, the fact is named
expenses.medical_gross , and the elicitor prompt is explicitly forbidden from subtracting thresholds itself.The live runs also confirmed the parts a script can only assume: thirteen correctly typed and assigned facts out of one free-form narrative over two elicitation rounds; the same narrative refused outright for an Illinois household, with the reason and the source in plain text; and for Ohio, $473 derived against $210 on the notice, shelter stage correctly uncapped because of the elderly member, medical deduction correctly $25 out of $60 of expenses.
Scripted turns tell you the machine is wired correctly. Real turns tell you it understood the household. You need both, and only one of them has to cost money.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article