Agents for Humans: 42,388 government records found six rules I had misread
How a public dataset that contains both the inputs and the answer turned six misread SNAP rules into a residual histogram I could read. Built with the Strands Agents SDK on Amazon Bedrock.
I am building Second Budget for the AWS "Agents for Humans" hackathon, Good Neighbor Agents track. It does one thing. A benefits navigator sits with a household holding a SNAP notice of action, and the household asks the only question that matters: is this number right? Today nobody checks. Second Budget re-derives the allotment from the underlying facts and, when it disagrees, says what would have to be true about the household for the notice to be correct.
That last part is why the arithmetic has to be exact. "Your benefit looks wrong" is worthless to a caseworker. "For your notice to be correct, your unearned income would have to be $584.41 higher than what you told me" is checkable in thirty seconds, by a person who is allowed to be skeptical of me.
So before I wrote a line of agent code, I went looking for a public dataset that contained both the inputs and the answer.
The dataset that grades you
USDA publishes SNAP Quality Control public-use microdata. FY2024 is 44,891 real de-identified households, each carrying the components that go into the budget calculation and the benefit that came out of it. Free, no registration, no data-use agreement, at snapqcdata.net. The host 403s a bare curl and serves the same public file to a browser User-Agent. I commit the SHA256 of the archive so a reader can prove they hold the same bytes I do.
There is one codebook distinction I nearly walked past.
FSBEN is constructed — "Final calculated benefit", the reviewed computation. RAWBEN is reported — "Reported SNAP benefit received". Measuring against one and claiming the other would have overstated everything downstream. Read the codebook before you read the columns.My deterministic engine replays the whole budget from raw components: income, each deduction, shelter costs, household size, caps. It never reads the file's net-income column; net income is derived. Against 42,388 households — the ones on the contiguous benefit schedule that carry every column the budget needs — exact equality, no tolerance band:
stage | agreement |
|---|---|
excess shelter deduction | 42,386 / 42,388 = 99.995% |
total deductions | 42,386 / 42,388 = 99.995% |
net income | 42,383 / 42,388 = 99.988% |
allotment | 42,385 / 42,388 = 99.993% |
Three households remain, at residuals of -29, -61 and -19. All singletons, no pattern.
It did not start there. A naive reading of the allotment formula — subtract 30% of net income from the maximum allotment, round the result, no minimum benefit — scores 65.703%. The gap between 65.703% and 99.993% is the entire point of this post.
A flat mismatch rate tells you nothing. A shaped one names the bug.
If your errors are scattered — one dollar here, forty there, no structure — you have noise, or a hundred small problems, and no lead. But if you histogram the residual (computed minus recorded) and one bar is enormous, that bar is a rule. It has a size, and the size is a formula you have not written yet.
Six rules came out of that histogram.
1. A spike at -23 on 3,257 records. That is the FY2024 minimum benefit. Easy to guess, easy to get wrong. I measured who it applies to instead of assuming: the floor fires on 3,830 households, and every one of them is size 1 (3,197) or size 2 (633). Letting it fire at every household size instead costs 24 households — 99.993% down to 99.936%. And, critically, it applies even when the computed allotment is zero, not merely when it is small. All 3,257 households in that -23 bar have a computed allotment of exactly zero, which is why the bar sits on the constant itself instead of smearing across -22 to -1 the way the other 573 floored households do. "Small benefits get floored" and "zero gets floored to 23" are different programs, and only the data told me which one I was in.
2. The rounding sits before the subtraction. 7 CFR 273.10(e)(2)(ii)(A) gives a state two permitted ways to handle the cents: round the household's 30% share up to the nearest higher dollar and then subtract it, or do not round the share at all and round the resulting allotment down. What it does not permit is the obvious third thing — subtract, then round the result to nearest. Doing that scores 74.710%, and the misses are one dollar high on 10,717 households. The file cannot tell me whether the share is rounded up or half-up; both score 99.993%. It tells me very loudly which side of the subtraction the rounding happens on.
3. The earned income deduction truncates. 20% of earned income, rounded down. Floor matches 42,513 / 42,532 = 99.955%. Rounding to nearest matches 90.059% — and 4,209 of those 4,228 misses are high by exactly one dollar. A residual that is always the same sign and always the same magnitude is never a modelling problem. It is one operator. Worth noting that the codebook describes this column as "rounded to nearest integer" and the file floors it; I went with the file.
4. Adjusted income is rounded before it is halved, not after. Halving first drops the excess shelter deduction from 99.995% to 74.502%, with 10,806 households off by exactly a dollar. The codebook states the order plainly:
HALFNET = MAX (0,ROUND(FSGRINC-FSSTDDED-FSERNDED-FSDEPDED-FSMEDDED-FSCSDED)/2) . Round, then halve — so the half can legitimately end in fifty cents.5. Python's
round() is not the round() in the regulation. This is the one I would put on a poster. Because rule 4 leaves half of adjusted income sitting on a half-dollar, the excess shelter deduction lands on an exact half-dollar for 10,807 of the 42,388 households, and the codebook then says that value is "rounded to nearest integer". The built-in does banker's rounding — round-half-to-even — so round(0.5) == 0 and round(2.5) == 2 . Using it at that step scores 87.482% against the file's own FSSLTDED column where half-up scores 99.995%. That is 5,304 households wrong by a dollar, with no exception, no warning, and nothing in the output to suggest anything happened.import math
def half_up(value: float) -> int: """Round half away from zero, the way the regulation means "round".""" if value < 0: return -math.floor(-value + 0.5) return math.floor(value + 0.5)
def earned_income_deduction(earned_income: float) -> int: """20 percent of earned income, truncated -- never rounded up.""" return math.floor(0.20 * earned_income)
def half_of_adjusted_income(adjusted_income: float) -> float: """Half of adjusted income, with the whole-dollar rounding applied first.""" return max(0.0, half_up(adjusted_income) / 2)
def household_share(net_income: float) -> int: """The household's own contribution: 30 percent of net income, rounded up.""" return math.ceil(0.30 * net_income)Notice that the statute does not round consistently, and I am not going to pretend it does. The household's share rounds up, which lowers the benefit, and that one is written into 7 CFR 273.10(e)(2)(ii)(A). The earned income deduction truncates down, which lowers the deduction, and that one is not written anywhere I could find — it is only in the data. Both defaults fall the same way, and it is not toward the household. My job is to reproduce that, not to smooth it.
6. A spike at exactly -54 on 38 households. This one found itself. A homeless household may take a flat standard shelter deduction instead of the excess-shelter calculation — $179.66 in FY2024, carried in the file as 180 — and 79 households in the replayed set take it. I had implemented half the rule. I zeroed the excess-shelter stage for them, which is right, and then never deducted the flat amount in its place. That leaves total deductions 180 too low, and
ceil(0.30 × 180) = 54 , exactly. When your residual bar sits on a number you can reconstruct from a constant you already have, you are not guessing any more; you are reading.Thirty-eight households is 0.09% of the file. A tolerance band of a dollar or two would have swallowed it and I would have shipped a wrong rule. This is why the target is an empty residual histogram, not a small one.
Proving the constants table, and refusing to fit it
The engine needs a table: minimum benefit, shelter cap, standard deduction, maximum allotment, per region and household size. I transcribed it by hand from the published schedule — Tables F.3, F.5 and F.6 of Appendix F of the FY2024 QC technical documentation, sourced there to USDA FNS — and then checked every value per household against the microdata's own columns, across 42,438 households.
- minimum benefit
MINIMUM_BEN: 100.000%
- shelter cap
SHELCAP: 100.000%
- standard deduction
FSSTDDED: 99.990%
- maximum allotment
BENMAX: 99.984%
All 11 residual households explain themselves: the recorded constant is this table's own value for a different household size. The disagreement is between two columns of the microdata, not between the microdata and my table.
Then Illinois. All 861 Illinois households carry a standard deduction exactly $7 below the published contiguous figure at every size band — 191 vs 198, 201 vs 208, 237 vs 244, 272 vs 279. 861 of 861 is a rule, not noise. I could not find a published source for the variation.
So Illinois is dropped from coverage. I did not copy 191 out of the microdata.
This is the discipline I want to argue for, because the temptation runs the other way and it feels like progress. If I take the constant from the corpus, my table now agrees with the corpus by construction. Coverage looks better. The measurement means nothing — a table fitted to the data it is checked against has stopped being evidence. The verification direction has to run from an independent source into the data, and when it fails you either find the source or you narrow your claim. Alaska is refused for a plainer reason: it has three separate benefit schedules and the file records only the state.
Layer C also caught two of my own bugs, both invisible to the allotment total. I had modelled the minimum benefit as size-conditional in the constant itself, returning 0 above two people, which disagreed with the file on 10,412 households — the constant and the rule about who it applies to are two different things, and collapsing them hides both. And I had not normalised state names, so
"IL" resolved to the covered contiguous schedule while "Illinois" was refused. The same household getting two different answers depending on typing is the exact failure mode you cannot let near a benefits conversation.Why this belongs in an agent post
The LLM part of my system cannot be unit-tested. I can evaluate it, I can gate it, I can constrain it, but I cannot assert equality on it. Everything else can be. So the design rule I ended up with is: make the part that can be proven as large as possible, and prove it against reality rather than against my own fixtures.
That shows up structurally. In my Strands graph the engine is a
MultiAgentBase peer node, not a @tool . A tool is something a model chooses to call, chooses arguments for, and can decline. The budget is a statute compiled to arithmetic. The graph runs it, the model does not get a vote, and the result is not negotiable. The model's job is upstream (eliciting facts) and downstream (writing them up) — never inside the arithmetic.There is a second dividend I did not expect. Because the engine reproduces the file exactly, I can run the counterfactual honestly. When the notice says $210 and the engine derives $473, I can drive each candidate explanation through the same engine and drop the ones that cannot close the gap:
The notice says $210. The engine derives $473.A difference of $263 a month in the household's favour.
For the notice to be correct, one of these would have to be true: * the notice is correct only if unearned income is $584.41 higher than stated ($584.41 rather than $0.00) -- 7 CFR 273.9(b) * the notice is correct only if earned income is $728.71 higher than stated ($1,928.71 rather than $1,200.00) -- 7 CFR 273.9(b)Shelter and medical costs are missing from that list because they cannot explain the gap. Driven all the way to zero, shelter costs move the allotment from $473 only as far as $313, and the medical deduction only as far as $462. Both land well above $210. Printing them with big scary numbers would have looked more thorough and been actively misleading. Each line that survives is verified by feeding the required value back through the engine.
If you are starting an agent project this month, spend the first week on this instead of on prompts. Find a public corpus that contains the inputs and the answer. Reproduce it with no tolerance. Histogram what is left, and treat every bar as a bug report filed by 44,891 households who already know the right answer.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article