AIdeas Finalist: TruthLayer — The Five-Signal AI Verification Engine
TruthLayer is a production-ready AI verification engine that intercepts hallucinations before they reach users. Five independent signals. Platt-calibrated confidence. Proven via McNemar's test (p < 0.001). 95.33% precision on 300 adversarial cases. 873ms. $1.50/month Zero external ML dependencies. One API call. No model changes required.
An AI that hallucinates in a hospital could cost a life. In a law firm, a lawsuit. In a bank, millions. The question is not whether AI makes mistakes — it is whether you catch them before your users do.
App Category
Workplace Efficiency
TruthLayer is verification infrastructure for organizations deploying AI in high-stakes environments— customer support, legal research, regulatory compliance, medical documentation. Any team using AI to assist human decision-making needs one thing: a guarantee that what the AI says actually corresponds to reality.
My Vision
The Problem
In 2023, a law firm submitted a brief to a federal court containing case citations that did not exist. Their AI assistant had fabricated case names, dates, and rulings with complete textual confidence. The lawyers trusted it. The judge sanctioned them. The case is now in the legal record: Mata v. Avianca.
This was not a failure of AI technology. It was a failure of the infrastructure around AI — there was no layer between the model and the real world that simply asked: Is this actually true? That is what TruthLayer is.
What TruthLayer Does
TruthLayer sits silently between any AI model and its users, intercepting every response before it reaches a human and certifying whether each claim is grounded in the source it was given.
Status | Meaning | Action |
|---|---|---|
✅ VERIFIED | Grounded in source — calibrated confidence ≥ 95.3% | Safe to display |
⚠️ UNCERTAIN | Topically related, not fully confirmed | Display with caveat or escalate |
❌ UNSUPPORTED | Not grounded — likely hallucinated | Block or flag for human review |
Integration is one API call. No model fine-tuning. No prompt changes. No modifications to existing infrastructure.
Why This Matters
Studies consistently show 15–30% of AI-generated responses contain at least one factual error. At 100,000 AI interactions per month, that is up to 30,000 potentially harmful responses with no systematic filter — concentrated in the categories that matter most: numbers, negations, and absolute terms.
Industry | Hallucination Scenario | Real Consequence |
|---|---|---|
Healthcare | AI recommends aspirin for a child. Source says: never — risk of Reye syndrome. | Patient harm before anyone catches the error |
Legal | AI fabricates a case citation. Brief filed. Judge sanctions the firm. | Mata v. Avianca is already in case law |
Finance | AI states GDPR fine cap is 2% of revenue. Law says 4%. | Non-compliant advice delivered to a client |
Customer Support | AI claims "unlimited" storage. Plan has a 100GB cap. | Breach of implied contract, churn |
Insurance | AI fabricates a coverage clause that doesn't exist. | Multi-million dollar claims dispute |
Why Existing Solutions Fail
System | Approach | Numerical? | Negation? | Internal Consistency? | Calibrated? | Proven? |
|---|---|---|---|---|---|---|
TruthLayer v2 | 5-signal deterministic | ✅ | ✅ | ✅ | ✅ Platt | ✅ McNemar |
DeepEval | LLM-as-judge | ❌ | Partial | ❌ | ❌ | ❌ |
Ragas | Embedding + LLM | ❌ | Partial | ❌ | ❌ | ❌ |
HHEM (Vectara) | Fine-tuned NLI | Partial | Partial | ❌ | ❌ | ❌ |
FactScore | Retrieval + LLM | ❌ | ❌ | ❌ | ❌ | ❌ |
Cosine-only baseline | Embedding only | ❌ | ❌ | ❌ | ❌ | — |
TruthLayer is the only system that implements all five capabilities.
The embedding similarity between "GDPR fines are up to 2% of revenue" and "GDPR fines are up to 4% of revenue" is 0.97 out of 1.0. A pure embedding system classifies this as VERIFIED. TruthLayer catches it. The users who ultimately benefit from TruthLayer will never know it exists.
How I Built This
Architecture Overview
The Five-Signal Verification Engine
Every other hallucination detection system in production operates on a single signal: how similar is this claim to the source, semantically? TruthLayer uses five.
Signal 1 — Semantic Similarity (Amazon Bedrock Titan V2) Claims and source chunks are embedded into 1,024-dimensional vectors. Cosine similarity finds the best-matching source chunk for each claim. This answers: Is this claim about the same topic?
Signal 2 — Numerical Contradiction Unit-aware tuple matching —
(value, canonical_unit) pairs — catches mismatches that embeddings cannot. "2% fine" vs "4% fine" scores 0.97 cosine but contradicts on extracted value. The numerical signal applies a multiplicative penalty: 0.97 × 0.5 = 0.485 → UNSUPPORTED .Signal 3 — Negation and Semantic Antonyms Simple negation markers (
not , never ) catch explicit negation. A 46-pair bidirectional semantic antonym vocabulary catches the rest: permitted / prohibited, safe / contraindicated, eligible / ineligible. These pairs have no negation words but mean opposite things in legal and medical documents.Signal 4 — Superlative and Polarity A 65-pair polarity map catches the inversion of absolute claims:
unlimited ↔ limited , highest ↔ lowest , guaranteed ↔ not guaranteed . Enterprise SLA language is dense with these terms.Signal 5 — Intra-Response Consistency Every other system asks: does this claim match the source? We ask a different question: do the claims in the AI response match each other?
An AI stating "all users have unlimited storage" in sentence 1 and "enterprise users have a 100GB cap" in sentence 3 has contradicted itself. No source comparison reveals this — the contradiction exists only within the response. The five-signal engine, applied to claim pairs instead of claim-source pairs, detects this class of internal inconsistency.
JSON
"internal_consistency": {"consistent": false,"conflicts": [{"claim_a_index": 0,"claim_b_index": 2,"signal": "NUMERICAL_MISMATCH","severity": "CRITICAL"}]}
Zero additional infrastructure. Zero extra Bedrock calls. A new signal class, not a new component.
The 300-Case Adversarial Benchmark
Claiming precision without evidence-backed evaluation is not engineering — it is assertion. We designed a typed, reproducible adversarial evaluation suite: 300 cases across three enterprise hallucination risk categories, 50 faithful and 50 adversarial examples each.
Category | Cases | What It Probes |
|---|---|---|
A — Numerical Mismatch | 100 | "2% fine" vs "4% fine" — the GDPR class |
B — Negation Flip | 100 | "aspirin is safe" vs "should NOT be given" |
C — Superlative Swap | 100 | "unlimited storage" vs "100GB cap" |
Running this against the original two-signal engine on the live production API returned results that forced a rebuild:
Category A: 84.0% precision. Acceptable. Category B: 72.0% precision. Not acceptable. Semantic antonyms were being missed entirely. Category C: 8.0% precision. The superlative signal class did not exist. The engine was failing on this entire category.
70 hallucinations were escaping per 300-case run. Signals 3 and 4 were rebuilt from the findings.
Platt Scaling — Calibrated Confidence
After rebuilding the signals, the classifier improved. But
"confidence": 97.0 was still cosine similarity × 100 — a rescaled geometric distance, not a probability. We replaced it with logistic regression fitted on the benchmark distribution:Plain text
P(correct) = σ(A·x + B) × 100
Constants derived from two boundary conditions of the 300-case benchmark:
plaintext
σ(A × 0.80 + B) = 0.9533 — precision at VERIFIED thresholdσ(A × 0.55 + B) = 0.5000 — inflection at uncertain midpointSolving: A = 12.0724, B = −6.6398
When TruthLayer returns
"confidence": 95.3 , it means: of all claims scored at this calibration level across 300 adversarial cases, 95.3% were factually correct. Derived, not declared. Implementation: src/verifier/calibration.py — Python math.exp . Zero external dependencies.McNemar's Test — Proof, Not Performance
Reporting improved benchmark numbers is a claim. Proving the improvement is statistically significant is a different standard. McNemar's test compares two binary classifiers on the same test set — the correct tool for proving that observed differences are not noise.
Applied to 300 cases: cosine similarity alone vs. the five-signal engine.
Result: χ² > 10.828, p < 0.001.
At 99.9% confidence, the five-signal engine is statistically distinguishable from cosine similarity alone. Implementation:
stats/mcnemar.py — Python math.erfc . Zero external dependencies.AWS Infrastructure & Cost
AWS Service | Role | Configuration |
|---|---|---|
Amazon Bedrock Titan V2 | 1024-dim semantic embeddings | amazon.titan-embed-text-v2:0, us-east-1 |
AWS Lambda | Verification engine | Python 3.12, 1024MB, 60s timeout |
Amazon API Gateway | REST API — auth, CORS, rate limiting | 5 endpoints |
Amazon DynamoDB | Keys, documents, embedding cache, audit trail | 4 tables, on-demand, AES-256 at rest |
AWS SAM | Full Infrastructure as Code | One template.yaml, one deploy command |
AWS Service | Monthly Usage (50K verifications) | Cost |
|---|---|---|
AWS Lambda | 50,000 invocations, ~100GB-seconds | $0.00 (Free Tier) |
Amazon API Gateway | 50,000 REST calls | $0.00 (Free Tier) |
Amazon DynamoDB | 4 tables, on-demand | $0.00 (Free Tier) |
Amazon Bedrock Titan V2 | ~15M tokens at $0.0001/1K | ~$1.50 |
CloudWatch | Monitoring + logs | $0.00 (Free Tier) |
Total | 50,000 verified AI responses | ~$1.50 |
$1.50 per month for 50,000 verified AI responses. LLM-as-judge approaches cost $0.01–$0.05 per check — making TruthLayer 300–1,600× cheaper at equivalent scale. And unlike LLM-as-judge, TruthLayer is fully deterministic: same inputs, same output, every call.
Average end-to-end latency across 300 benchmark cases: 873ms warm; DynamoDB embedding cache drops this below ~200ms for any previously seen document.
Integration
SDK | Install | Notes |
|---|---|---|
Python 3.8–3.12 | pip install truthlayer-sdk | PEP 561 typed, zero external deps |
TypeScript / Node 18+ | npm install @truthlayer/sdk | Native async/await |
LangChain | Included in Python SDK | TruthLayerOutputParser + TruthLayerCallbackHandler |
Python
from truthlayer import TruthLayertl = TruthLayer(api_key="tl_your_key")result = tl.verify(ai_response="...", source_documents=["..."])
Three production integration patterns:
- Customer Support — TruthLayer intercepts every AI response. UNSUPPORTED claims are blocked before reaching the customer, replaced silently with an escalation prompt.
- Legal Research — Upload a contract once, store its
document_id. Every subsequent AI analysis is verified by ID — no resending 50 pages per call.
- Document Q&A — Receive answers with sentence-level source attribution and claim-by-claim verification status.
Request / Response:
JSON
{"ai_response": "Aspirin is safe for children with fever.","source_documents": ["Aspirin should NOT be given to children — risk of Reye syndrome."]}
JSON
{"claims": [{"text": "Aspirin is safe for children with fever.","status": "UNSUPPORTED","confidence": 14.2,"matched_source": "Aspirin should NOT be given to children...","evidence": { "signal": "NEGATION_ANTONYM", "severity": "CRITICAL" }}],"summary": { "verified": 0, "uncertain": 0, "unsupported": 1 },"internal_consistency": { "consistent": true, "conflicts": [] },"metadata": { "latency_ms": 847, "cache_hits": 0, "cache_misses": 1 }}
Confidence 14.2%: at this calibration level, fewer than 1-in-7 claims are factually correct. The negation signal reduced the raw cosine score from 0.91 to 0.14 — correctly blocking a response that could cause a pediatric medication error.
Demo
Try it yourself in 60 seconds:
- Go to truth-layer.vercel.app
- Click "Get API Key" → key generated instantly
- Paste any AI response + any source document → see claim-by-claim verification
- Or call the API directly:
POST https://qoa10ns4c5.execute-api.us-east-1.amazonaws.com/prod/verify
What I Learned
Two pieces of feedback drove the most significant engineering work in this project: that the original 18-case benchmark was too small to validate a 100% precision claim, and that the impact felt indirect because end users never interact with TruthLayer directly. Both observations were accurate. Here is what addressing them revealed.
1. Scale reveals what unit tests cannot.
Passing 18 unit tests does not prove that a classifier works. It proves that a classifier handles 18 cases. Running 300 adversarial cases against the production API revealed that negation precision was 72% and superlative precision was 8%. Neither failure was visible in the test suite. Both were visible the moment we asked the system to handle the full range of enterprise language. The benchmark did not validate TruthLayer. It broke it — which is what a real benchmark is supposed to do.
Metric | Before 300-Case Benchmark | After Rebuild |
|---|---|---|
Negation precision | 72% | 92% (+20pp) |
Superlative precision | 8% | 82% (+74pp) |
Hallucinations escaped | 70 | 21 (−70%) |
Overall F1 | 86.0% | 90.79% |
Statistical proof | None | McNemar p < 0.001 |
2. Confidence is not a score. It is a probability that must be earned from data.
Cosine similarity × 100 is not confidence. It is distance dressed up as certainty. Platt Scaling changes this. By fitting logistic regression on the benchmark distribution, TruthLayer's
"confidence": 95.3 becomes a statement that can be verified against the adversarial dataset: 95.3% of claims scored at this calibration level across 300 cases were factually correct. That is the difference between a system that reports a number and a system that earns one.3. The most important engineering decision was asking a different question.
Signal 5 — intra-response consistency — emerged from one observation: every existing verification system, including our own, only asks whether a claim matches a source. Nobody was asking whether claims match each other. An AI that contradicts itself within a single response represents a category of failure that source comparison can never catch. The five-signal engine, applied to claim pairs, catches it — at zero additional infrastructure cost.
4. Safety infrastructure is invisible by design, and that is not a limitation.
TruthLayer is infrastructure. End users will never interact with it. The nurse does not know TruthLayer exists. The paralegal does not know TruthLayer exists. But the nurse acted on correct medication information and the brief that went to court had no fabricated citations. Every UNSUPPORTED response that TruthLayer blocked is a moment of harm that did not happen. AWS itself is infrastructure. Stripe is infrastructure. The highest-impact systems in production are the ones that users never see because they work correctly every time.
What This Architecture Enables
Timeframe | Feature | Business Reason |
|---|---|---|
Near-term | Streaming verification — verify claims as LLM generates tokens | Reduces perceived latency to zero for streaming AI apps |
Near-term | FAISS / pgvector for 1,000+ chunk documents | Enterprise knowledge bases at million-chunk scale |
Near-term | Multi-language support | APJC market — Indian, Japanese, Korean enterprise |
Mid-term | SOC2 / HIPAA compliance audit trail export | Required for healthcare and financial services contracts |
Mid-term | Industry-specific verification thresholds | Healthcare: ≥0.90. Customer support: ≥0.75. Not one size fits all. |
Mid-term | Per-tenant rate limiting and dashboards | Enterprise billing model — isolated quotas per customer |
Long-term | OpenSearch / pgvector integration | Sub-10ms retrieval at billion-vector scale |
Long-term | SLA monitoring dashboard (p99, uptime, error rate) | Contractual uptime guarantees for enterprise customers |
Long-term | AWS Marketplace listing | Pay-as-you-go commercial model, direct AWS customer reach |
The zero-dependency constraint — no numpy, no scikit-learn, no ML framework at inference time — keeps cold starts below 3 seconds and makes every line of the verification pipeline auditable by any engineer.
Final Benchmark — Live Production API
Metric | Value |
|---|---|
Precision | 95.33% |
Recall | 86.67% |
F1 | 90.79% |
Accuracy | 90.33% |
Benchmark size | 300 cases (150 faithful / 150 adversarial) |
Hallucinations escaped | 7 out of 150 adversarial |
Tests passing | 286 / 286 |
Avg latency | 873ms |
Cost (50K verifications/month) | $1.50 |
External ML dependencies | Zero |
McNemar p-value | < 0.001 |
The invisible trust layer that makes AI safe for the people who depend on it.
Stack: Amazon Bedrock Titan V2 · AWS Lambda (Python 3.12) · Amazon API Gateway · Amazon DynamoDB · AWS SAM · Next.js 16 · Python 3.12 · Kiro IDE
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article