I Worked in Customs and Tried to Teach an AI to Classify Goods. The Hard Part Wasn't the AI.
A tiered HS tariff classification pipeline (keyword rules → full-text search → vector search → LLM) and what the customs industry teaches about hallucinations, cost, and keeping humans in the loop.
I worked in the customs and freight industry, where one of the core tasks is taking an invoice line like "Front bumper bracket, steel, OEM" and turning it into a 10-digit HS tariff code. Get it wrong and the importer pays the wrong duty, a shipment sits at the border, or worse, there's a fine.
There are thousands of codes. The difference between two of them can come down to whether a part is "specifically designed for" a vehicle or "general purpose." Experienced agents carry this in their heads. New ones spend years learning it.
So the obvious question was: can an LLM do this?
Attempt 1: Send everything to the model
The first version was simple: extract line items from the invoice, send each one to a large language model, ask for the HS code.
It worked… sometimes. The problems showed up fast:
- Confident hallucinations. The model would return a code that looked valid but didn't exist in the national tariff.
- Cost and latency. A 60-line invoice meant 60 model calls. For one declaration.
- No memory. The same products from the same suppliers get classified over and over. The model was re-reasoning from scratch every time.
The lesson: an LLM is a great expert, but a terrible first line of defense.
Attempt 2: A tiered pipeline, cheapest layer first
The rebuild was a funnel where each layer only handles what the previous one couldn't:
- Keyword Tariff Map (zero AI). Past classification decisions are mined into a keyword-to-code map. If a term appears at least twice with 70%+ consistency, it becomes a rule. A huge share of repeat items stop right here, instantly and for free.
- Full-text search. For near-matches, a simple SQLite FTS5 index over tariff descriptions finds candidates.
- Vector search. For items described in unusual ways, semantic search narrows things down to the right chapter.
- The LLM, last. Only the genuinely ambiguous items reach the model, and it gets the full candidate codes from the relevant chapter as context, not a blank page.
Two changes made the biggest difference in quality:
- Never let the model invent codes. It must choose from codes that actually exist in the tariff. Anything else is rejected.
- Show uncertainty instead of hiding it. When confidence is low, the system presents the top options side by side instead of silently picking one. A human makes the final call.
What I learned
Domain data is the moat. The most valuable part of the system isn't the model. It's years of real classification decisions turned into rules.
Human-in-the-loop isn't a weakness. In regulated work, the goal isn't to replace the expert. It's to make the expert 5x faster and catch the cases that deserve their attention.
Cheap layers first. Most real-world inputs are repetitive. Paying for AI reasoning on a problem solved yesterday is waste.
Where AWS fits next
The next step is mapping this architecture onto AWS: Amazon Bedrock for the model layer, Amazon Textract for invoice extraction, and Amazon OpenSearch Service for combined keyword and vector search. Moving from a single-machine prototype to something that scales is the next learning project, and I'll write about it here.
Your turn
I'd really like to hear from people working in other regulated fields (legal, medical coding, finance, compliance):
Where do you draw the line between what the AI decides and what a human must approve? And has anyone here built a similar "cheap layers first" pipeline on Bedrock? I'd love to know what worked.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article