How We Built AI Jury: A Multi-Agent Pitch Evaluation System on Amazon Nova 2 Lite
We built AI Jury after watching too many great projects lose to bad evaluation processes — exhausted judges, inconsistent feedback, and zero explanation. This is the story of how we used Amazon Nova 2 Lite to build a multi-agent system that gives every pitch a full independent panel, structured scoring, and a leaderboard — in under two minutes.
How We Built AI Jury: A Multi-Agent Pitch Evaluation System on Amazon Nova 2 Lite
Tag: Amazon-Nova
We've both been to enough hackathons to know how the story usually ends.
Not the winner's story. The other one.
A team spends three months building something genuinely interesting. They get five minutes on stage. One judge is scrolling their phone. Another asks a question that's answered on slide four — they clearly didn't read it. The feedback, if it comes at all, is something like "interesting idea, but the market wasn't clear." No score breakdown. No reasoning. No "here's what would have changed our minds."
You walk away not knowing what actually happened. And that's the part that bothered us most.
We've also been on the other side — helping organize events, watching judges burn out by round two, watching evaluation quality drop in real time as the night wore on. By the time you're reviewing project number forty-seven at midnight, you're not evaluating anymore. You're surviving.
So we asked ourselves: what if we could fix both sides of this at once?
That question turned into AI Jury.
The idea: a panel that never gets tired
The concept was simple in theory and genuinely hard in practice.
We wanted a system where every pitch gets evaluated by a full panel of independent judges — not one tired human, not a single AI call that pretends to be objective. A real panel. Multiple perspectives. Independent reasoning. And at the end, a report that actually explains itself.
The key word here is independent. We didn't want judges talking to each other before forming opinions. We didn't want one agent summarizing and another rubber-stamping it. We wanted the CFO to look at the financials through a CFO lens, the CTO to stress-test the architecture through a CTO lens, and neither of them knowing what the other thought until the Summarizer brought it all together.
That's not just a design preference. It's the whole point. If your judges are effectively the same voice with different labels, you haven't built a panel — you've built an echo chamber with extra steps.
The pipeline: seven stages, no shortcuts
We ended up with seven stages. Each one has a job and doesn't do anyone else's job.
Stage one is Input. The system receives a single JSON from the database: project data, weighted criteria, judges with their prompts, participants with pitch links. Clean, versioned, logged.
Stage two is the Decoder. This is where Amazon Nova 2 Lite first enters the picture. It reads each participant's materials — checks link availability, extracts text, normalizes the content. If a pitch deck is missing, the pipeline doesn't stop. It flags it and continues with what it has. No blocking. No crashes.
Stage three is the Judge Orchestrator. Every judge runs in parallel. Each one gets the decoded pitch plus their own technical prompt with three instruction layers: base, default, and advanced. Here's the detail we're most proud of: the project criteria with weights are intentionally not passed to the judges. Each one evaluates through their own lens without knowing how the organizer weighted the dimensions. Pure independent signal.
Stage four is the Summarizer Agent. It takes all judge outputs per participant, eliminates duplicates, resolves conflicts, and builds three development scenarios: Growth, Stable, and Risk. It also calculates success probability. This is the agent that has to do the hard thinking — it's reading five different opinions and finding the honest truth in them, including surfacing where they genuinely disagree.
Stage five is the Scoring Service. This is the only place in the entire pipeline where the organizer's criteria and weights are applied. The final score is calculated by a weighted formula, normalized to the project's scale. Transparent. Auditable. No black box.
Stage six is the Leaderboard. All participants ranked automatically, with per-criteria breakdown and status badges — Winner, Runner-Up, Top 3. No manual spreadsheet reconciliation at three in the morning.
Stage seven is the Comparison Agent. Triggered only on user request. It compares two participants across criteria, shows delta scores, and explains in plain language why one outperformed the other on each dimension.
Why Nova 2 Lite was the right choice — and not just because it was available
When you're running five or more judges in parallel, each processing a full pitch deck with a complex structured prompt and returning strict JSON, the model choice matters enormously.
We needed three things: a large context window, reliable structured output, and speed at reasonable cost.
Nova 2 Lite delivered all three.
The one million token context window meant we could fit an entire pitch deck, the judge profile, scoring instructions, and examples in a single call. No chunking. No context loss between chunks. No "the model forgot what was on slide one by the time it got to slide twelve."
The instruction following was the other critical factor. Our pipeline lives and dies on clean JSON output. One malformed response from one judge breaks the Summarizer's input. We tested with several models and Nova 2 Lite was the most consistent at returning exactly the schema we asked for, every time, under varied prompt conditions.
And the speed. Running a full evaluation across five judges plus summarization and scoring in under two minutes is only possible if each individual call is fast. Nova 2 Lite's price-performance ratio also meant we could run the full pipeline without the economics becoming absurd — which matters when you're evaluating twenty, thirty, forty participants in a single session.
The hardest problem: judge determinism
Here's the thing nobody warns you about when you start building multi-agent systems: getting consistent behavior is genuinely hard.
We hit this early. Same pitch, same judge, same prompt — and the score would vary by two or three points between runs. Not catastrophically, but enough to make the system feel unreliable. If you can't trust that the evaluation is stable, you can't trust the leaderboard. And if you can't trust the leaderboard, the whole thing falls apart.
We spent more time on this than on any other single problem. Temperature settings, prompt structure rewrites, output contract reinforcement, validation layers at every stage. The fix wasn't one thing — it was a combination of tighter prompts, stricter output schemas, and a validation pass that catches anything that drifts outside the expected range before it poisons downstream agents.
We got there. But it humbled us.
What it looks like when it works
An organizer sets up a project in about two minutes. They define criteria, pick their jury board — or build a custom one from our judge library — add participants with their pitch decks, and hit launch.
Two minutes later, every participant has a full evaluation report. Every judge, every score, every reason. Strengths, weaknesses, concrete recommendations. A leaderboard that ranks everyone by weighted criteria. And if you want to understand why first place beat second place on Technical Implementation specifically, the Comparison Agent explains it in three bullets.
No exhausted judges. No inconsistent feedback. No founder walking away not knowing what happened.
What's next
We're building a live pitch decoder — video analysis of real-time presentations. Pace, delivery, argument structure, how you handle pressure. Not just "what's in the deck" but how you sell it in the room.
We're also in early conversations with hackathon organizers to pilot AI Jury as an official evaluation layer. And with VC scouts who review deal flow at volume and need something more reliable than gut instinct at scale.
The goal has always been the same: make good evaluation available to everyone. Not just the projects that happened to land in front of the right judge on the right night.
Built with Amazon Nova 2 Lite
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article