Agents for Humans: A Household Fraud Agent That Publishes Its Own Misses
An engineering deep dive into Gatehouse, a fraud-defense agent for families on AWS: the architectural line a model is never allowed to cross, a silence law that decides who gets woken, an evaluation protocol built to make dishonesty inconvenient, and the live run that dropped our accuracy from 1.00 to 0.75 the week we shipped.
Two messages reached my family's phone eleven minutes apart. Both named the same major Indian bank. Both carried the correct last four digits of a real account. Both used the same clipped, urgent register that real bank SMS uses.
One was genuine. The other pointed at a lookalike host of the form
<bankname>.bank.in , where the attacker owns bank.in and serves the brand as a subdomain. On a phone the URL truncates and you read the leftmost label, which is the bank's own name.I could not tell them apart on a phone screen. Neither could my father, and he is the person it was aimed at. The difference was not in the language, the formatting, or the urgency. It was in a registry lookup that takes a machine four milliseconds and a human a phone call to the bank.
That is the gap Gatehouse closes. This post is the full engineering account: what it is, what it measured, what it got wrong, and the number that arrived after I thought I was finished.
Everything below is real, measured, and reproducible. The repository is public and one command regenerates every metric in this post.
Table of contents
- The wrong question, and the narrower one
- The architectural line
- Fencing: untrusted text is evidence, not instructions
- Registries beat vibes
- The graduated silence law
- Degradation has to be loud
- Spend is a metric, not a receipt
- The evaluation protocol
- Results: publishing 30.6 percent
- Where the false gates actually were
- Then I ran it on traffic the benchmark never contained
- What broke
- Limitations, honestly
- Reproduce it
1. The wrong question, and the narrower one
Every consumer fraud tool I examined asks the same thing: is this a scam? Then it alerts whenever the answer is uncertain.
Uncertainty is the common case. So the tool alerts constantly. So the family mutes it. So it protects nobody. The tool is technically correct and practically worthless, and the failure is structural rather than a tuning problem.
The question worth answering is narrower:
Does a human need to be interrupted?
Those are not the same question. "Probably a scam" and "worth waking someone at 3am" are different claims, and collapsing them is precisely why fraud tooling gets muted. Once you take the second question seriously, silence stops being the absence of a feature and becomes the deliverable. Every design decision below follows from that.
2. The architectural line
A family member forwards anything suspicious to a Telegram bot. Five stages run, and the ordering is load-bearing.
Stage | What it does | Model involved |
|---|---|---|
Fence | Normalise, annotate instruction-shaped spans, wrap, attach canary | No |
Triage | Score urgency | Yes, one float |
Verify | Adjudicate hosts and claims against registries | No |
Graph | Hash identifiers, record, correlate | No |
Guardian | Compose verdict, confidence, reason codes, silence band | No |
The model participates in exactly one stage and contributes exactly one number.
Plain text
scam_likelihood: float, 0.0 to 1.0reason_code: string, 3 to 60 chars
That is the entire model contract. Deterministic Python maps the float to a band, then takes the stronger of the model's band and a rule engine's independent band. The rule engine is a floor the model cannot lower.
The consequence is worth stating precisely, because it is the whole security argument: the worst a hallucinating or prompt-injected model can do is raise urgency. It cannot clear a message that deterministic evidence flagged. That is not a prompt instruction, which a model may ignore. It is control flow, which it cannot.
Running on AWS in
ap-south-1 :Service | Role |
|---|---|
API Gateway (HTTP API) | Webhook intake, secret header verified before parsing |
AWS Lambda × 3 | Intake and investigation (Python 3.12), scheduled digest, console API (Node.js 22) |
Amazon Bedrock | Nova Micro via the APAC inference profile, triage only |
DynamoDB × 2 | Cases and evidence bundles; threat graph |
EventBridge | Case event bus and the digest schedule |
CloudFront + S3 | Static guardian console |
CloudWatch | Scrubbed structured traces, spend alarms |
Deployed with AWS SAM. No containers, no idle cost.
3. Fencing: untrusted text is evidence, not instructions
Every forwarded message is hostile input by definition. Before any model sees it:
- Normalise. Unicode NFKC folding, zero-width characters removed, control characters stripped. That closes the invisible-character smuggling class.
- Annotate, do not delete. Instruction-shaped spans (
ignore all previous,system prompt,you are now,new instructions,disregard, and any attempt to forge a closing tag) are wrapped in visible[QUOTED_EVIDENCE]markers.
- Wrap. The payload is enclosed in
<untrusted_signal id="...">tags, and the system prompt states that text inside those tags is data containing no instructions.
- Canary. A random per-case token rides along. If it ever appears in anything a member or guardian sees, the reply is replaced with a refusal,
CANARY_TRIPlands in the reason codes and the degraded flags, and an error is logged.
Step 2 is the one people argue with. Deleting an injection feels safer. It is not: a message that says "ignore all previous instructions" is almost certainly hostile, and that is the strongest signal in it. Strip it and the model scores a laundered message. Keep it visible and quoted, and the model sees the tell.
4. Registries beat vibes
The verify stage runs no model at all. It extracts every host in the message, including bare domains without a scheme, because scam SMS almost never carries
https:// , and adjudicates each one:Outcome | Condition | Weight |
|---|---|---|
PASS | Host inside one of 40 official issuer domains | 0.90 |
PASS | Host inside one of 21 curated trusted services (logistics, e-commerce, government) | 0.85 |
INCONCLUSIVE | Anything else | 0.20 |
FAIL | Message names a known brand, and links resolve outside that brand's domains | 0.50 |
Unknown domains return INCONCLUSIVE, never FAIL. Not knowing is not evidence. That choice sets the direction the system errs in: a missing registry entry causes a false gate, never a missed scam.
The last row is the spoof detector, and it is what caught the message that started this post. The message claimed a bank; the host it linked to was not inside that bank's registered domains; the claim hard-failed. The registry covers 40 issuers, so this is a property of the check, not of one brand.
All scoring is data, not code. Weights live in a YAML country pack, checksummed by SHA-256 into every evidence bundle so any verdict replays against the exact registry that produced it:
Pack field (India, v0.2.0) | Value |
|---|---|
strong_phrase | 0.45 |
moderate_phrase | 0.30 |
weak_phrase | 0.15 |
payment_intent_bonus | 0.20 |
url_present_bonus | 0.15 |
Languages | English, Hindi |
One strong phrase alone reaches the screening band and nothing else does. A moderate phrase plus both structural bonuses reaches 0.65, still below the escalation band, because structure alone should not escalate. A new country is a new file.
5. The graduated silence law
Every verdict carries a band, and the band decides who gets interrupted.
Band | Meaning | Guardian paged? |
|---|---|---|
SILENT_KILL | Settled scam, confidence ≥ 0.95 | No. Member warned, case logged to the weekly report |
AGENT_SCREEN | Confidence ≥ 0.75 | A human sees the result, never the panic |
BADGED_RING | Confidence ≥ 0.40 | Surfaced with a risk badge; the human decides |
PASS | No threat found | Invisible processing, not a suppressed alarm |
Four rules make the ladder conservative, and each one exists because the alternative was worse:
-
NEEDS_HUMANcan never be silenced. That is the pipeline explicitly asking for a person.
- Only SCAM reaches
SILENT_KILL. SUSPICIOUS means the evidence did not settle, and an unsettled case is not something to handle quietly however high the number beside it reads.
- A degraded case is never silenced. Some dependency did not answer, so the confidence was computed on partial evidence and has not earned quiet.
- A panic press overrides the band every time.
The arithmetic behind 0.95 is worth showing, because a threshold without its derivation is a magic number. A SCAM confidence is composed as:
Plain text
0.70 + 0.10 × (hard failures) + 0.20 × (largest failure weight)+ 0.05 if the identifiers have prior graph events
One issuer-rule failure at weight 0.50 gives 0.90. Add prior graph events and it reaches exactly 0.95. So full silence requires either two independent hard failures, or one hard failure plus an identifier this system has seen before. A single detector, however confident, never buys silence on its own.
6. Degradation has to be loud
Every stage runs inside its own fault boundary. A stage may fail; the pipeline may not; nothing fails quietly.
Failure | Behaviour | Flag |
|---|---|---|
Triage model errors | Fall back to the rule engine | TRIAGE_MODEL_FALLBACK |
Breaker refuses the call | Rule engine only | TRIAGE_BUDGET_REFUSED |
Verify cannot run | Verdict forced to NEEDS_HUMAN | VERIFY_UNAVAILABLE |
Graph store unreachable | Empty finding, disclosed | GRAPH_UNAVAILABLE |
Composition itself fails | NEEDS_HUMAN | PIPELINE_FAULT |
Row three is the one I would defend hardest. A SAFE verdict produced without verification is a lie of absence, so the code refuses to produce one.
There is a matching subtlety that cost us a real defect. An empty graph result and a graph outage are different functions returning different flags. Most messages carry no correlatable identifier at all, which is a normal empty result. Conflating the two marked healthy traffic as degraded and misreported every soak window until it was separated.
7. Spend is a metric, not a receipt
Two ceilings, both checked before every model call:
Control | Value | Scope |
|---|---|---|
Per-case call cap | 12 | One investigation |
Per-case budget | $0.02 | One investigation |
Hourly breaker | 600 calls | Process-wide, rolling window |
The hourly breaker exists because a fresh meter is constructed per case, so a flood of forwards would otherwise be a flood of independent budgets with no ceiling above them.
Pricing is a table, and an unknown model ID resolves to the most expensive known rate rather than zero, so an unrecognised model fails conservative:
Model | USD per Mtok (in / out) |
|---|---|
amazon.nova-micro-v1:0 | 0.35 / 1.40 |
amazon.nova-lite-v1:0 | 0.60 / 2.40 |
amazon.nova-pro-v1:0 | 0.80 / 3.20 |
Unknown | 5.00 / 25.00 |
Measured across the full 480-case evaluation: $0.12485 total, $0.00026 per investigation. Latency in that run was p50 475 ms, p95 549 ms.
8. The evaluation protocol
Four rules, each one paid for:
Wilson 95 percent intervals or silence. Precision 1.00 over 336 cases prints as CI [0.9887, 1.0]. Without the interval, "perfect" is a vanity claim that collapses on the first live miss.Pre and post publish together. A threshold change without its before-number is a confession you did not keep.The failure taxonomy is generated from the run's own miss ledger. The classification is code, not hand-written humility.Every number names its denominator. A percentage over rows nobody can audit is a number nobody should trust.
The benchmark is 600 synthetic cases across 15 strata, split 480 dev and 120 sealed hold-out. Synthetic by necessity and by ethics: real fraud data from real families is unavailable and would be unconscionable to collect.
Cross-split leakage is structurally impossible, not asserted. Dev reference tokens are drawn from 10000-49999 and hold-out from 50000-99999, so a dev string cannot appear in the hold-out. An overlap check runs anyway and fails the build.
The hold-out opens exactly twice in the product's life. The runner refuses
--split holdout without an explicit confirmation flag and a written reason, and that reason is embedded in the artifact permanently. Every look is an auditable event. It has been opened zero times so far.Calibration runs on the dev split only. An eight-point threshold sweep found floors 0.25 through 0.45 statistically indistinguishable and published a recommendation of no change, with the grid showing where recall falls off a cliff above 0.50. That is what a calibration artifact should look like: a recommendation to do nothing, with the evidence for why.
9. Results: publishing 30.6 percent
The most useful number this system ever produced is a bad one.
Run (480 cases, real model) | Precision | Recall | False-gate rate |
|---|---|---|---|
First real-model run | 0.884 | 1.00 | 30.6% |
After the fix | 1.00 CI [0.9887, 1.0] | 1.00 CI [0.9887, 1.0] | 0.0% |
44 benign messages escalated out of 144. Recall was perfect, so nothing dangerous slipped, but the product was unusable by its own definition. A shield that interrupts a family a dozen times a week gets muted, and a muted shield protects nobody.
Both artifacts are committed side by side, pre and post, in the same commit. Not the good one.
The fix caps channel-free messages to SAFE, but the two guards on it matter more than the cap:
- The band must be model-driven. If the deterministic rules independently escalated the message, the cap does not apply.
- The rule engine's own conclusion must be weak. If the rules said SCREEN or worse, two detectors agreeing outranks one absence heuristic.
So it caps model opinion and nothing else. A text-only scam script never loses its escalation. That constraint was the design; the false-positive reduction was its consequence.
10. Where the false gates actually were
Aggregate accuracy hides exactly the failures that matter. Here is the same 44 misses, per stratum:
Benign stratum | n | False gates, before | Rate | After |
|---|---|---|---|---|
otp_forward | 16 | 16 | 100% | 0 |
legit_bank_offer | 24 | 15 | 62.5% | 0 |
delivery_update | 20 | 10 | 50.0% | 0 |
newsletter_promo | 28 | 3 | 10.7% | 0 |
family_chatter | 28 | 0 | 0% | 0 |
govt_notice_legit | 20 | 0 | 0% | 0 |
govt_legit_trap | 8 | 0 | 0% | 0 |
Every scam stratum stayed at zero misses through both runs: KYC (72), digital arrest (48), investment (56), UPI collect (48), lottery (32), courier (32), job scams (32), relative impersonation (16).
Read the top row again. The system gated every single OTP forward. A family member forwarding their own one-time password, the single most common benign forward there is, was escalated 16 times out of 16. An aggregate false-gate rate of 30.6 percent describes that badly. "One hundred percent of OTP forwards" describes it correctly, and it is the number that told me what the bug actually was.
That is the entire argument for per-stratum confusion tables.
11. Then I ran it on traffic the benchmark never contained
Here is the part that arrived after I thought I was finished, and it is the most important section in this post.
I deployed, then pushed fifteen fresh real-world messages through the live pipeline: eight scams spanning digital arrest, UPI collect abuse, courier customs, credit-score loan bait, lottery advance-fee, electricity disconnection and task-job scams, and seven genuine messages including a real bank debit alert, an OTP forward, a delivery notification and a government payment credit.
| Benchmark, 480 cases | Live, 15 unseen messages |
|---|---|---|
Precision | 1.00 | 0.75 |
Recall | 1.00 | 0.75 |
False-gate rate | 0.0% | 28.6% |
Spend per case | $0.00026 | $0.00025 |
Six caught, two missed, two false gates, five correctly ignored.
The most important miss indicts my own fix. A digital arrest script, the highest-harm scam category in India, came back SAFE with the reason code
NO_ACTION_CHANNEL . That is precisely the cap I built to eliminate the 44 false positives. A digital arrest opener carries no link, no payment handle and no explicit ask; the coercion is the payload. That is exactly the shape the cap clears.My fix for false positives bought a false negative in the highest-harm category. That is not an implementation bug. It is a real tension in the design, and the benchmark could not have revealed it, because the benchmark's digital-arrest cases all carried a callback number.
The second miss is quieter and more interesting. The model returned a schema validation error, the system correctly fell back to the rule engine and correctly flagged
TRIAGE_MODEL_FALLBACK . Then it produced SAFE at confidence 1.00. Reading the code afterwards: SAFE returns the PASS band before the degraded check ever executes. So the rule that a degraded case is never silenced holds on the SCAM path and not on the SAFE path. And because a SAFE confidence is computed as 1.0 − triage_confidence , knowing nothing at all yields maximum confidence.Both false gates were genuine bank credits, flagged on payment intent. My own family's real messages would have been gated.
The benchmark shares an author with the detection lexicons. I wrote both. That limitation is stated in the repository's README, and this run is what it looks like when the limitation stops being theoretical.
12. What broke
The repository carries a ledger with symptom, root cause, fix and prevention for every incident. Four worth naming:
Incident | Root cause | Prevention now enforced |
|---|---|---|
A doctrine implemented nowhere | The silence law was documented in three files, guarded by a config validator, tested at the validator, and consumed by no code anywhere | A constant introduced for a documented behaviour ships with its consumer in the same commit, or it does not ship |
The hourly breaker was a setting with no mechanism | breaker_hourly_call_cap was referenced exactly once, at its own definition | Unused-configuration audit is a review step, not a lucky grep |
A bug behind a type cast | DynamoDB string sets unmarshal to a JS Set ; the route cast it as string[] . Four other read paths "defensively" used Object.values , which returns empty for a Set , so the console silently reported zero reason codes and zero degraded cases everywhere | One normaliser, seven input shapes, checked in CI |
A test double that modelled one use case | The mock model emitted one tool call and stopped, so any agent with two tools looped forever | A double models the protocol, not the first caller that needed it |
Three of those four are the same defect: configuration that looks load-bearing and is wired to nothing. The crash announced itself. The silence was the real bug.
13. Limitations, honestly
- No live accuracy. The guardian override ledger is live and empty. Until it fills, I can describe behaviour but not accuracy on real traffic.
- The benchmark shares an author with the lexicons. Section 11 is what that costs.
- The channel-free cap needs narrowing so coercion scripts escape it. Direct action item from the digital arrest miss.
- The hourly breaker is process state. Under Lambda that bounds one warm container, not a genuine cross-invocation ceiling. A shared hour-bucketed counter is the real fix and is not built.
- No URL intelligence. Domain age and reputation are not checked, so shorteners stay INCONCLUSIVE. The system gates rather than guesses.
- The tool-driven investigator ships flag-gated off, because the published numbers were measured without it and a default those metrics never covered would describe a system nobody is running.
- Single region, single deployment. Deliberate at this scale.
- The sealed hold-out has never been opened.
14. Reproduce it
Bash
git clone https://github.com/Prakhar2025/gatehousemake setup # venv and pinned dependenciesmake check # 469 tests, ruff, mypy strictmake eval-mini # offline evaluation, no AWS accountmake eval-full-json # reproduces the 480-case dev split byte-identically
469 tests, 92.68 percent coverage, every gate enforced in CI, including a check that fails the build if the README's claimed test count drifts from the real one.
Artifact | Path |
|---|---|
Before the fix, 30.6% false gate | docs/eval-results/staging-dev-metrics-pre.json |
After the fix | docs/eval-results/staging-dev-metrics.json |
Threshold sweep recommending no change | docs/eval-results/calibration-dev.json |
Failure ledger | docs/what-broke.md |
Repository: https://github.com/Prakhar2025/gatehouse
Live console: https://d2p659vmmc9l0e.cloudfront.net/console
The worst number in there is the one I would show you first.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article