AWS Builder Center

Agents for Humans: a Strands Model provider that makes your whole agent loop unit-testable

A real Strands Model provider that replays a script instead of calling a backend, so the whole agent loop is assertable offline with no credentials.

I am building  Second Budget  for the Agents for Humans hackathon, Good Neighbor Agents track. A benefits navigator sits with a household holding a SNAP notice of action, and the household asks the only question that matters: is this number right? Today nobody checks. Second Budget re-derives the allotment from the underlying facts and, when it disagrees, says what would have to be true about the household for the notice to be correct.
The claim the whole thing rests on is that its decisions are provable. Which makes the first thing I built not the agent, and not the budget engine, but a  Model  provider that returns a script.
Right now the suite is 163 tests, green, in under two seconds, with no AWS credentials in the environment — exercising the real graph, real edge conditions, real interventions, real interrupts. This post is that provider, in full ( src/second_budget/models/scripted.py ), and what it buys.

The problem with testing an agent

An agent whose behaviour you can only observe by paying a provider is not proven, it is sampled. Two runs of the same graph with the same prompt can take different paths, and the interesting questions are not about the prose at all:
  • Did the cycle converge in three iterations, or did it converge because the execution cap stopped it?
  • Did my  end_turn  actually halt the loop, or did the model just happen to stop?
  • Did the intervention  Deny  reach the model, and did the model retry?
  • Does the interrupt resume path put the facts in the ledger?
Each of those is a property of the loop, and none of them needs a real model to answer. What they need is a model that does exactly what you tell it to.

The provider

 Model  is a small interface: its four abstract methods are  stream ,  structured_output ,  update_config ,  get_config . Implementing it honestly — emitting the same  StreamEvent  sequence Bedrock emits — means the agent loop cannot tell the difference. That distinction is the entire point. A mock that bypasses the loop proves nothing about the loop.
from dataclasses import dataclass, field
from typing import Any

class ScriptExhausted(RuntimeError):
"""The agent asked for one more turn than the script provides."""

@dataclass
class Turn:
"""One model response: free text, or one or more tool calls."""

text: str | None = None
tool_calls: list[tuple[str, dict[str, Any]]] = field(default_factory=list)
stop_reason: str | None = None

@classmethod
def say(cls, text: str) -> "Turn":
return cls(text=text, stop_reason="end_turn")

@classmethod
def tool(cls, name: str, payload: dict[str, Any]) -> "Turn":
return cls(tool_calls=[(name, payload)], stop_reason="tool_use")

@classmethod
def tools(cls, *calls: tuple[str, dict[str, Any]]) -> "Turn":
return cls(tool_calls=list(calls), stop_reason="tool_use")
And the provider itself:
import json
from collections.abc import AsyncGenerator, AsyncIterable
from typing import Any, TypeVar

from pydantic import BaseModel
from strands.models.model import Model
from strands.types.content import Messages
from strands.types.streaming import StreamEvent
from strands.types.tools import ToolChoice, ToolSpec

T = TypeVar("T", bound=BaseModel)

class ScriptedModel(Model):
"""Deterministic ``Model``: replays ``turns`` in order, counting calls."""

def __init__(self, turns: list[Turn], *, model_id: str = "scripted") -> None:
self._turns = list(turns)
self._config: dict[str, Any] = {"model_id": model_id}
self.calls = 0
self.seen_messages: list[Messages] = []

def update_config(self, **model_config: Any) -> None:
self._config.update(model_config)

def get_config(self) -> dict[str, Any]:
return dict(self._config)

async def structured_output(
self,
output_model: type[T],
prompt: Messages,
system_prompt: str | None = None,
**kwargs: Any,
) -> AsyncGenerator[dict[str, T | Any], None]:
turn = self._next()
if len(turn.tool_calls) != 1:
raise ScriptExhausted(
"structured_output needs a turn with exactly one tool call "
"carrying the payload"
)
yield {"output": output_model.model_validate(turn.tool_calls[0][1])}

async def stream(
self,
messages: Messages,
tool_specs: list[ToolSpec] | None = None,
system_prompt: str | None = None,
*,
tool_choice: ToolChoice | None = None,
**kwargs: Any,
) -> AsyncIterable[StreamEvent]:
self.seen_messages.append(messages)
turn = self._next()

yield {"messageStart": {"role": "assistant"}}

index = 0
if turn.text:
yield {"contentBlockStart": {"contentBlockIndex": index, "start": {}}}
yield {"contentBlockDelta": {
"contentBlockIndex": index, "delta": {"text": turn.text}}}
yield {"contentBlockStop": {"contentBlockIndex": index}}
index += 1

for n, (name, payload) in enumerate(turn.tool_calls):
yield {"contentBlockStart": {
"contentBlockIndex": index,
"start": {"toolUse": {
"name": name, "toolUseId": f"scripted-{self.calls}-{n}"}}}}
yield {"contentBlockDelta": {
"contentBlockIndex": index,
"delta": {"toolUse": {"input": json.dumps(payload)}}}}
yield {"contentBlockStop": {"contentBlockIndex": index}}
index += 1

stop = turn.stop_reason or ("tool_use" if turn.tool_calls else "end_turn")
yield {"messageStop": {"stopReason": stop}}
yield {"metadata": {
"usage": {"inputTokens": 0, "outputTokens": 0, "totalTokens": 0},
"metrics": {"latencyMs": 0}}}

def _next(self) -> Turn:
if self.calls >= len(self._turns):
raise ScriptExhausted(
f"the agent asked for turn {self.calls + 1} but the script has "
f"{len(self._turns)}. Either the loop is not terminating, or "
f"the script is short."
)
turn = self._turns[self.calls]
self.calls += 1
return turn
That is the whole thing. Usage is one line:
agent = Agent(model=ScriptedModel([
Turn.tool("record_fact", {"fact_id": "shelter.rent", "value": 900}),
Turn.say("I have what I need."),
]), tools=[record_fact], callback_handler=None)
Two design choices are load-bearing.  self.calls  counts model invocations, so a test can assert on how many times the loop went round. And running off the end of the script raises rather than producing a benign default turn — a silent fallback would let a runaway loop look like a passing test, which is precisely the failure a scripted provider exists to catch.

Four things it lets me assert

That  end_turn  really halted the loop. My  HaltWhenFrontierCloses  hook sets  event.end_turn  on  AfterToolsEvent  once the recorded facts satisfy the budget, so the agent does not make one more paid round-trip whose only possible output is "I have everything". "The loop halted" is otherwise indistinguishable from "the model happened to stop". The counter distinguishes them:
def test_the_halt_saves_a_model_call_and_the_counter_proves_it() -> None:
ledger = FactLedger()
halt = HaltWhenFrontierCloses(ledger)
# The script offers a second turn. If the halt works, it is never consumed.
agent, model = _agent(
[Turn.tools(*[_call(f, v) for f, v in ALL_FACTS]),
Turn.say("this turn must never be reached")],
ledger, [halt],
)

agent("Record what the household told me.")

assert halt.halted is True
assert model.calls == 1, "the loop asked the model again after it had everything"
assert len(ledger) == 12
That the cyclic graph converges in exactly three iterations. Second Budget's elicitation loop is a bounded cyclic  GraphBuilder  graph, because the required-fact set is conditional: learning that a member is elderly or disabled adds a medical-expense requirement that did not exist a moment earlier. So I script an elicitor that dribbles facts out over three rounds, with the elderly fact arriving in round two, and assert on the topology the real graph actually walked:
order = [node.node_id for node in result.execution_order]
assert order == ["elicitor", "engine",
"elicitor", "engine",
"elicitor", "engine",
"drafter"], order
assert engine.runs == 3
assert result.status is Status.COMPLETED
That last line is not decoration. A graph limit breach does not raise:  multiagent/graph.py:787-790  sets  status = FAILED  and returns. I ran the runaway case and read the result off it —  failed_nodes  is an  int  and is  0 , every one of the 14 node results still says  completed , and only  status  carries the breach.  if result.failed_nodes:  misses a runaway entirely. Scripting a model that records the same fact forever is how I test the non-convergent path at all.
A full interrupt raise-and-resume cycle, offline.  BeforeToolsEvent  raises one interrupt carrying a whole twelve-fact batch, with the facts the model inferred marked for scrutiny. Two tests drive the two halves; trimmed together they read:
paused = agent("Record what the household told me.")

assert paused.stop_reason == "interrupt"
assert len(paused.interrupts) == 1, "twelve facts must cost one gate, not twelve"
assert len(ledger) == 0 # nothing was written before a human saw it

agent([
{"interruptResponse": {
"interruptId": paused.interrupts[0].id,
"response": {"rejected": [FactId.EARNED_INCOME.value]},
}}
])
known = {f: ledger.value(f) for f in ledger.established}
assert not is_closed(known), "a rejected fact must leave the frontier open"
An intervention  Deny , and the model's retry after it.  NumbersGate  denies any tool payload carrying a currency token absent from a frozen engine certificate — set membership, decidable. The script gives the model a fabricated figure first and a real one second:
agent("Draft the request.")

assert written == [f"The allotment is ${allotment}."]
assert gate.denials == [("write_finding", (1800.0,))]
# The model was told, retried, and succeeded -- three calls, not a crash.
assert model.calls == 3
A  Deny  at  before_tool_call  cancels the call and hands the reason back as the tool result; a sibling test asserts that result comes through with  status == "error"  and the reason text intact. Nothing in the framework bounds how often the model may then try again — the per-invocation  Limits  in  types/agent.py  are opt-in and unset unless a caller passes them. So the denial reason is the model's entire steering signal and has to be an instruction rather than a complaint. Whether it works as an instruction is a question for a real model; whether the gate fires, blocks, and leaves the agent alive is a question this test settles for good.

One sharp edge

The event loop catches provider exceptions and re-raises them as  EventLoopException  ( event_loop/event_loop.py:404 ). A test waiting for the bare exception never sees it. The original is preserved as  __cause__ :
with pytest.raises(EventLoopException) as caught:
agent("go")
assert isinstance(caught.value.__cause__, ScriptExhausted)
This is defensible framework behaviour — one exception type at the loop boundary is easier to handle in production than every provider's own — but it costs ten confusing minutes if you have not met it.

The cost angle, said plainly

A hackathon budget is free credits. The failure mode is not one expensive test, it is a cyclic graph with a condition that never flips, discovered in the morning. Two things protect against it here, and both are free: a script that raises the moment the agent asks for a turn it does not have, and  result.status is Status.COMPLETED  as the only correct convergence check. I develop the graph, the gates and the interrupts against zero-cost turns, and spend real tokens only on the question real tokens can answer.

When not to use it

A scripted model proves the harness, not the prompt. Nothing in those 163 tests tells me whether the elicitor asks a good question, or whether the model can turn a household's narrative into typed facts. That needs real runs, and I ran them on  us.amazon.nova-pro-v1:0  in  us-east-1 .
They found a bug the scripted suite structurally could not. "She spent sixty dollars on her inhaler" went into the budget as a sixty-dollar medical deduction. Only the portion above $35 is deductible — and the USDA microdata I validate the engine against already carries that column net of the threshold, so the validation path was always right and only the narrative path was wrong. The rule now lives in the engine, the fact is named  expenses.medical_gross , and the elicitor prompt is explicitly forbidden from subtracting thresholds itself.
The live runs also confirmed the parts a script can only assume: thirteen correctly typed and assigned facts out of one free-form narrative over two elicitation rounds; the same narrative refused outright for an Illinois household, with the reason and the source in plain text; and for Ohio, $473 derived against $210 on the notice, shelter stage correctly uncapped because of the elderly member, medical deduction correctly $25 out of $60 of expenses.
Scripted turns tell you the machine is wired correctly. Real turns tell you it understood the household. You need both, and only one of them has to cost money.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article