Agents for Humans: your agent should be measured by how rarely it needs you
Field notes from building Standing Order for the AWS Agents for Humans Hackathon,
part 1 of 3.
Most agent demos I've seen this year end with the same slide: "our agent
automated N% of the work." I built my hackathon entry around the opposite
number, and this post is about why.
The problem is not the work — it's the re-deciding
My target user is a solo business owner: a freelancer, an indie hacker, someone
who is the entire accounting department on top of being the entire engineering
department. Watch how they actually lose a day. It's not the invoicing. It's
this loop, thirty times a day:
- "Is this ₩42,000 cafe charge a client-entertainment expense or personal?"
- "Does this task fall inside the retainer, or is it billable extra?"
- "Do I chase this overdue invoice today or let it ride?"
Each one takes twenty seconds. Each one was already decided the last time
the same situation came up. The cost isn't the twenty seconds — it's the
context switch out of real work and back. I started calling this
re-decision cost, and once it had a name, the product designed itself:
don't automate the work, stop re-asking the decisions.
Standing orders, literally
The mechanism is borrowed from banking. A standing order is an instruction you
give once, and the bank executes it every cycle without asking again. My agent,
Standing Order, does the same for admin judgments:
- An event arrives (transaction, invoice, subscription charge).
- Compiled rules — plain Python predicates, zero LLM calls — check whether an
approved standing order covers it. Exactly one match: act, and write a
receipt with the rule id. Zero matches or a conflict: escalate to a human.
- When the human answers, an LLM (Claude Sonnet 4.6 on Bedrock, via Strands'
structured_output ) proposes a rule candidate from that single answer.
The candidate does nothing until the human approves it, and its
effective_from is the approval date — never retroactive.
The interesting design decision is what the agent is not allowed to do: it
cannot widen its own authority. Rules are born from human answers and activate
only through human approval. The LLM sits outside the decision path entirely.
Why "automation rate" is the wrong axis
If I had graphed automation rate, the graph would flatter the model: a better
model classifies more things, number goes up. But my user's actual pain is
interruptions, so I measured those instead — and gave the agent a hard
budget: three interruptions a day. Everything beyond the budget waits for
tomorrow's digest, ranked by a deterministic priority score
(log-amount + overdue marker + irreversibility by event kind).
Then I replayed 60 days of seeded events (420 events, deterministic with
--seed 42 : 300 transactions, 40 invoices, 20 subscription charges, 60
calendar entries, with the same cafes and clients genuinely recurring). The
seeded persona answers a queue item the first time a merchant appears, and the
resulting rule takes effect from the merchant's second occurrence. No
retroactivity, so the curve you get is earned, not staged:
- escalations per day: 4 on day 1 → 1 on day 60 (first-10-day average 2.7,
last-10-day average 0.9)
- coverage after 60 days: 75.3% of the 360 decision-bearing events handled
without asking
- standing orders accumulated: 27
- model calls during the entire replay: 0
That last number matters more than it looks. Because matching is compiled
predicates, the whole 60-day replay runs with the network gone. There's a
pytest fixture that monkeypatches socket.socket to raise, plus a canary test
that proves the fixture itself works, and then the full replay runs to
completion under it. If a model call ever sneaks into the decision path, CI
fails the same hour.
One labeling decision worth stealing
Halfway through, I noticed my dashboard was about to lie. The hero copy says
"the agent called you less," but with a hard budget of 3, the actual
interruption count is flat at the cap — the thing that falls over time is
demand: how often the agent would have called you. Labeling that falling
line "interrupts" would have been the kind of mismatch a careful reviewer
catches in ten seconds.
So the chart draws both lines and names them honestly: "escalations per day
(interrupt demand)" falling from 4 toward 1, and "actual interrupts
(budget-capped)" sitting at the limit until demand drops below it. The budget
mechanism becomes visible instead of hidden, and every number on the screen
links to the execution log that produced it. There's even a test that fails if
the README's numbers drift from the latest docs/metrics.json .
What this buys you
Two honest properties fall out of this architecture:
The value doesn't ride on the model. Interrupt demand falls because rules
accumulate, not because the model got smarter. Swap the model, the curve
doesn't move. Model prices drop, the curve doesn't move. The product's worth
is in the rule book the user built, which they can read, edit, and version.
The failure mode is bounded. The worst thing a bad rule can do is
miscategorize events matching its own narrow conditions — merchant pinned,
amount capped — until the human revises it (revisions create a new version;
old versions are preserved with their lineage back to the original queue
answer). The agent physically cannot send email, move money, or touch anything
outside its five DB-only tools. Part 2 covers how Strands hooks enforce that
at runtime rather than by prompt.
Not in scope, stated plainly because it's the honest part: no bank API — events
come from seeds, CSV, and webhooks; no tax judgment; single user. What's left
is small, but it's the part that was actually missing: an agent measured by how
rarely it needs you.
Standing Order is an entry in the AWS Agents for Humans Hackathon
(Professional Agents track), built on the Strands Agents SDK. The repo, the
seed generator, and the replay path are public — every number above reproduces
with one command.
Loading article