Textract read 88 as 8, and gave it 99.5% confidence
Three bugs we hit building a bill-reading ledger on Amazon Textract, S3 and DynamoDB over a hackathon weekend: a row that contradicted its own arithmetic, a timeout that only covered the response headers, and a price comparison anchored to write order instead of the bill date.
We built KiranaLedger over a weekend for First Commit. The idea is small: a shop owner photographs a supplier bill, and before it gets filed, the app checks it against every bill that supplier has sent before. Duplicate bill numbers, prices that moved, due dates coming up.
Most of the weekend went into the reading part working properly. This is about three things that broke in ways we didn't expect, all of which we only found because we ran actual photographs of actual paper through it instead of the clean test data we'd generated ourselves.
What it does
Photo goes to S3 with a presigned PUT, so the image never passes through a function. Textract reads it. The extracted bill gets compared against the supplier's history in DynamoDB, and anything odd is attached to the bill before it's written.
photo ──► S3 ──► Textract ──► checks against history ──► DynamoDB AnalyzeExpense │ └─► flags attached to the billStage | Service |
|---|---|
Photo upload | Amazon S3, presigned PUT |
Extraction | Amazon Textract, AnalyzeExpense |
Ledger and price history | Amazon DynamoDB, single table, one transaction per bill |
Hosting | AWS Amplify Hosting, Next.js SSR on Web Compute |
Permissions | IAM compute role attached to the Amplify app |
We'd planned to use Bedrock for extraction. Model access wasn't granted on our account and the request went into a support queue, so on Saturday we switched to Textract.
AnalyzeExpense turned out to suit this better anyway. It's built for invoices, so it hands back labelled fields (vendor, bill number, dates, totals) and grouped line items without any prompting, and the frozen ExtractedBill type we'd agreed on Friday didn't need a single field renamed.The 8 that should have been 88
To see how it handled a bad photo, I took one of our test bills and wrecked it: blurred, rotated six degrees, underexposed, compressed hard. Then ran it through.
Toor Dal 10 kg @ 145 = 1450Sunflower OF 5 str @ 152 = 760Sugar 25 to @ 44 = 1100Basnati Hope 20 kg @ 8 = 1760 <- the bill says 88confidence 0.995warnings: noneThe garbled names were expected. The 8 was not, and neither was the 99.5% next to it.
Our arithmetic check didn't catch it either, because it compared the sum of the line totals against the bill total, and the line total (1,760) had been read correctly. The bill added up fine. It was the row that didn't.
That matters more than a wrong number on a screen. The recorded price for Toor Dal from that supplier becomes 8. The next bill arrives at 145 a kilo, which is an ordinary price, and the app reports a 1,712% increase. The feature meant to catch a supplier raising prices starts inventing them instead.
So we added a per-row check: 20 units at 8 rupees is 160, not 1,760. When a row disagrees with itself we take the line total as correct, since the bill total agrees with it and nothing agrees with the rate, and divide back out.
"Basnati Hope" did not add up, the rate was read as 8but 1760 over 20 works out to 88The name is still garbage. We left that alone and said so in the demo, because the money is what the comparisons run on, and the money is now right.
What we took from this is narrower than "don't trust confidence scores". Textract's confidence is about the characters, and it was right about those: it saw a clear, well-formed 8. It has no opinion on whether the row makes sense. If you need that, you have to check it yourself, and usually the document gives you something to check against.
A 9 second timeout that ran for 22
Flag wording comes from a language model. It gets facts the code has already worked out and writes a sentence around them. It can't change a severity or touch a number, and if it fails we fall back to templates, so the bill still saves.
Because it sits on the upload path, I gave it a 9 second budget. Then watched one call take 22 seconds and return successfully.
const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);const response = await fetch(url, { signal: controller.signal, ... }) .finally(() => clearTimeout(timer));fetch resolves when the response headers arrive, not when the body has finished arriving. Clearing the timer in .finally on the fetch cancelled the abort as soon as the server responded at all, which left the slow part, streaming the body back, with no limit on it.One
try/finally around the read fixed it. I'm posting the snippet because it looks correct, it passes review, and the failure is invisible unless you happen to time it.While measuring, the model choice turned out to be about latency rather than quality. Same prompt, three free models, all writing acceptable copy:
Model | Time to answer |
|---|---|
nex-n2.5-mini | 1.3s |
deepseek-v4-flash | 5.8s |
nemotron-3.5-lightning | 98s |
An August bill that got cheaper
Recording the demo, we uploaded an August bill, then a September one, then re-photographed the August bill to show the duplicate check firing.
It caught the duplicate. It also reported that Toor Dal had dropped from 145 to 118.
The price lookup was asking for the most recently written price, which by then was September's. What it needed to ask for was the most recent price as of this bill's date. Those come to the same thing only when bills arrive in the order they were issued, and they don't: people photograph a week's worth at once, or find an old one in a drawer, or re-shoot one they already filed.
The sort key already started with the bill date, so it was a query change rather than a schema change:
KeyConditionExpression: "pk = :pk AND sk BETWEEN :from AND :to",ExpressionAttributeValues: { ":from": "PRICE#", ":to": `PRICE#${bill.billDate}~`,}We'd have caught this eventually in normal use, but it's the kind of thing that costs you the user rather than throwing an error. Someone shown one alert that's obviously wrong stops reading the next one.
What we'd do differently
Test on ugly inputs first. All three of these came out of real photographs and a deliberately ruined one. None of them came out of the bills we generated, which were clean, and which we spent most of Saturday being reassured by.
The handwritten bills are still beyond it. We photographed a real handwritten cash memo from a stationery shop: Textract read the printed letterhead perfectly and then returned 1,061 rupees for a bill that says 430. Printed bills work, handwritten ones don't, and the app says so rather than filing a confident wrong number. Fixing that properly is the next thing, and it probably isn't a Textract setting.
Code: https://github.com/PranavV205/kiranaledger
Live: https://main.d2djabvtfe8b0r.amplifyapp.com
Demo: https://www.youtube.com/watch?v=8-5c6h1pAoA
AI tools used: Claude Code.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article