AWS Builder Center

Agents for Humans: your agent should be measured by how rarely it needs you

Solo owners lose their day to re-deciding, not to the work. Standing Order compiles each answer into a rule with a hard interrupt budget — and the metric that matters falls from 4 to 1.

Series: Standing Order — Agents for Humans Hackathon (3 articles)

  1. 1
    Agents for Humans: your agent should be measured by how rarely it needs you This article

Agents for Humans: your agent should be measured by how rarely it needs you

Field notes from building Standing Order for the AWS Agents for Humans Hackathon, part 1 of 3.
Most agent demos I've seen this year end with the same slide: "our agent automated N% of the work." I built my hackathon entry around the opposite number, and this post is about why.

The problem is not the work — it's the re-deciding

My target user is a solo business owner: a freelancer, an indie hacker, someone who is the entire accounting department on top of being the entire engineering department. Watch how they actually lose a day. It's not the invoicing. It's this loop, thirty times a day:
  • "Is this ₩42,000 cafe charge a client-entertainment expense or personal?"
  • "Does this task fall inside the retainer, or is it billable extra?"
  • "Do I chase this overdue invoice today or let it ride?"
Each one takes twenty seconds. Each one was already decided the last time the same situation came up. The cost isn't the twenty seconds — it's the context switch out of real work and back. I started calling this re-decision cost, and once it had a name, the product designed itself: don't automate the work, stop re-asking the decisions.

Standing orders, literally

The mechanism is borrowed from banking. A standing order is an instruction you give once, and the bank executes it every cycle without asking again. My agent, Standing Order, does the same for admin judgments:
  1. An event arrives (transaction, invoice, subscription charge).
  1. Compiled rules — plain Python predicates, zero LLM calls — check whether an approved standing order covers it. Exactly one match: act, and write a receipt with the rule id. Zero matches or a conflict: escalate to a human.
  1. When the human answers, an LLM (Claude Sonnet 4.6 on Bedrock, via Strands'  structured_output ) proposes a rule candidate from that single answer. The candidate does nothing until the human approves it, and its  effective_from  is the approval date — never retroactive.
The interesting design decision is what the agent is not allowed to do: it cannot widen its own authority. Rules are born from human answers and activate only through human approval. The LLM sits outside the decision path entirely.

Why "automation rate" is the wrong axis

If I had graphed automation rate, the graph would flatter the model: a better model classifies more things, number goes up. But my user's actual pain is interruptions, so I measured those instead — and gave the agent a hard budget: three interruptions a day. Everything beyond the budget waits for tomorrow's digest, ranked by a deterministic priority score (log-amount + overdue marker + irreversibility by event kind).
Then I replayed 60 days of seeded events (420 events, deterministic with  --seed 42 : 300 transactions, 40 invoices, 20 subscription charges, 60 calendar entries, with the same cafes and clients genuinely recurring). The seeded persona answers a queue item the first time a merchant appears, and the resulting rule takes effect from the merchant's second occurrence. No retroactivity, so the curve you get is earned, not staged:
  • escalations per day: 4 on day 1 → 1 on day 60 (first-10-day average 2.7, last-10-day average 0.9)
  • coverage after 60 days: 75.3% of the 360 decision-bearing events handled without asking
  • standing orders accumulated: 27
  • model calls during the entire replay: 0
That last number matters more than it looks. Because matching is compiled predicates, the whole 60-day replay runs with the network gone. There's a pytest fixture that monkeypatches  socket.socket  to raise, plus a canary test that proves the fixture itself works, and then the full replay runs to completion under it. If a model call ever sneaks into the decision path, CI fails the same hour.

One labeling decision worth stealing

Halfway through, I noticed my dashboard was about to lie. The hero copy says "the agent called you less," but with a hard budget of 3, the actual interruption count is flat at the cap — the thing that falls over time is demand: how often the agent would have called you. Labeling that falling line "interrupts" would have been the kind of mismatch a careful reviewer catches in ten seconds.
So the chart draws both lines and names them honestly: "escalations per day (interrupt demand)" falling from 4 toward 1, and "actual interrupts (budget-capped)" sitting at the limit until demand drops below it. The budget mechanism becomes visible instead of hidden, and every number on the screen links to the execution log that produced it. There's even a test that fails if the README's numbers drift from the latest  docs/metrics.json .

What this buys you

Two honest properties fall out of this architecture:
The value doesn't ride on the model. Interrupt demand falls because rules accumulate, not because the model got smarter. Swap the model, the curve doesn't move. Model prices drop, the curve doesn't move. The product's worth is in the rule book the user built, which they can read, edit, and version.
The failure mode is bounded. The worst thing a bad rule can do is miscategorize events matching its own narrow conditions — merchant pinned, amount capped — until the human revises it (revisions create a new version; old versions are preserved with their lineage back to the original queue answer). The agent physically cannot send email, move money, or touch anything outside its five DB-only tools. Part 2 covers how Strands hooks enforce that at runtime rather than by prompt.
Not in scope, stated plainly because it's the honest part: no bank API — events come from seeds, CSV, and webhooks; no tax judgment; single user. What's left is small, but it's the part that was actually missing: an agent measured by how rarely it needs you.
Standing Order is an entry in the AWS Agents for Humans Hackathon (Professional Agents track), built on the Strands Agents SDK. The repo, the seed generator, and the replay path are public — every number above reproduces with one command.



Series: Standing Order — Agents for Humans Hackathon (3 articles)

  1. 1
    Agents for Humans: your agent should be measured by how rarely it needs you This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article