AWS Builder Center
ReproLens: Don't Ask an LLM What Happened. Make It Prove What Happened.

ReproLens: Don't Ask an LLM What Happened. Make It Prove What Happened.

Cross-artifact investigation for ML experiments, built on Amazon Bedrock.

1. Idea and Impact

The problem. ML training runs fail in the worst way: quietly. A run can finish with no exception and still be badly wrong, with training accuracy climbing while validation collapses. The cause is almost never next to the symptom. It's scattered across five files: config, dependencies, dataset metadata, training log, and traceback. Researchers cross-reference them by hand, every time.
MLflow and Weights & Biases capture this evidence well, but they don't correlate it. Nothing tells you that your  config.yaml  says 8 classes while your  dataset_metadata.json  says 9.
Who it's for. Students, researchers, and small ML teams who don't have a platform team and lose hours to "why did this run go wrong?"
What changes for them. Upload the five artifacts they already have and get back a structured diagnosis (root cause, confidence, competing hypotheses, recommended checks) where every claim cites the exact artifact and field behind it. Hours of manual cross-referencing become a checkable answer in seconds.

2. Built on AWS

Every service has a job, and Bedrock is the core of the product rather than decoration.
Service
Role
Amazon Bedrock (Converse API)
Reasoning engine, using a forced, schema-constrained tool call (report_experiment_diagnosis) so the model can only answer in validated JSON
AWS Lambda
Normalizes artifacts (safe YAML/JSON parsing, line-numbered logs) and orchestrates the investigation
Amazon API Gateway
Front door for the browser client
Amazon S3
Artifact storage
Amazon DynamoDB
Run history, written asynchronously so a persistence failure never blocks a diagnosis
AWS Amplify Hosting
Frontend hosting
Browser → Amplify → API Gateway → Lambda
├── S3 (artifacts)
├── Preflight engine (deterministic checks)
├── Bedrock (Converse, strict tool use)
└── DynamoDB (history, async)
Two design decisions that make it more than "files → LLM → paragraph":
  1. Facts first, model second. Rule-based checks run before Bedrock sees anything. A class-count mismatch is either true or false, so no model needs to decide that. Verified findings go to Bedrock as pre-established evidence.
  1. Strict tool use instead of free text. The schema requires a non-empty evidence array (artifact, field, observed value, significance), supports competing hypotheses, and treats  insufficient_evidence  as a valid, honest outcome rather than forcing a confident guess.

3. Execution: it works, end to end

The demo runs the real pipeline (S3 → Lambda → Bedrock), not a canned response. It was validated against six failure classes, including:
  • Silent generalization failure. Training accuracy rose from 82% to 99% while validation fell from 0.48 to 0.34, with no exception thrown. ReproLens correlated the diverging metrics with a distribution shift in the dataset metadata and returned a medium-confidence diagnosis with competing hypotheses, because the evidence suggests an explanation without proving one.
  • A healthy-experiment control. It proves the system doesn't manufacture problems that aren't there.
The signature feature is the evidence chain. Every step, claim → evidence → source, is checkable. Click any evidence node and the exact line from your uploaded file opens with its context highlighted. You never have to take the AI's word for it.

4. What I learned

(Edit this so it's true to your experience. Judges score this section, so use your own specifics.)
  • Constraining the model is often more valuable than prompting it. Forcing a schema-constrained tool call through Converse made outputs reliably structured and auditable, which no amount of "please respond in JSON" achieved.
  • Deterministic checks belong in front of the LLM. Establishing facts in code first reduced what the model had to guess, and made its reasoning more trustworthy.
  • Letting a system say "I don't know" is a feature. Designing  insufficient_evidence  as a first-class outcome changed how I thought about AI reliability.
  • Keep non-critical writes off the critical path. Making DynamoDB history async meant a failure there could never hurt the user.
  • (Add: what was new to you this week, such as your first Bedrock tool-use call, first Lambda + API Gateway deploy, or anything that broke and taught you something.)

5. Honest about the edges

Built in four days for First Commit (AWS × WeMakeDevs, Bharat Builds Tour). It's scoped to PyTorch-shaped config and log conventions, validated on synthetic fixtures rather than a real-world failure corpus, and has no authentication layer yet. What I didn't compromise on: every diagnosis is evidence-grounded, schema-validated, and honest about what it doesn't know.

Links

#bharatbuilds #wemakedevs #aws
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article