AWS Builder Center
Who's judging your LLM judge? A lifecycle for Agent Evaluation

Who's judging your LLM judge? A lifecycle for Agent Evaluation

Agent evaluation with LLM judges: measure the judge's mistakes, get a failure rate you can defend, and follow nine steps from spec to production.

Series: Agent Evaluation (1 article)

  1. 1
    Who's judging your LLM judge? A lifecycle for Agent Evaluation This article
Most teams grade their AI agents with another AI: an LLM "judge" that reads each conversation and marks it pass or fail. That judge makes mistakes too. This article shows how to measure those mistakes, how to get a failure rate you can defend to your boss or a regulator, and the nine steps that get you there.

You answer for what your agent says

In 2024, a Canadian tribunal ordered Air Canada to compensate a passenger. The airline's website chatbot had told him he could claim a bereavement discount after his trip, which the airline's real policy didn't allow.
Air Canada argued the chatbot was "a separate legal entity that is responsible for its own actions". The tribunal called that "a remarkable submission" and ruled the airline was responsible for what its chatbot said ( Moffatt v. Air Canada, 2024 BCCRT 149 ).
The amount was small. The lesson isn't: if your agent gives customers wrong answers, that's your problem. So you need to know how often it happens, and that number has to hold up when someone questions it.

A pass rate is a guess with a range

Say your agent passes 45 of your 50 test cases. Your dashboard says 90%.
With only 50 tests, the real pass rate could plausibly be anywhere from about 79% to 96%. In plain terms, your agent might fail one customer in twenty, or one in five. The second agent fails four times as often as the first.
Comparing versions has the same problem. If version A passes 88 of 100 tests and version B passes 92, B looks better.
With 100 tests, a 4-point gap like that easily happens by chance. Picking B because it scored higher is a coin toss you didn't know you were making.

The judge makes mistakes too

Nobody can read every conversation by hand, so teams use an LLM judge. Researchers have measured how often these judges get it wrong:
  • Just swapping the order of two answers let a weaker model beat a stronger one in 66 of 80 comparisons ( Wang et al., 2023 ).
  • Judges that agreed with humans more than 90% of the time could still be more than 10 points off on the final score ( Thakur et al., 2024 ). A high agreement number can hide a wrong result.
  • On hard questions with a clear right answer, one leading model did no better than a coin flip as a judge ( Tan et al., 2025 ).
This doesn't mean you should stop using judges. It means you should treat a judge like any measuring tool. Check it against a trusted reference before you rely on it, and account for its error when you report what it measured.
For a pass-or-fail judge, the error comes down to two questions:
  1. Of the conversations that really failed, how many does the judge catch?
  1. Of the conversations that were really fine, how many does it correctly pass?
A judge that passes everything gets the second question perfectly right and is useless. You need both answers to be good.
Getting there takes nine steps. Each one exists because skipping it produces a specific wrong number.

Nine steps from "what's correct?" to production

1. Write down what a correct answer looks like

Before anyone grades anything, write a short spec covering:
  • what the agent handles
  • which policies it applies
  • the rules for each tool it uses
  • what it must never do
This is what lets two reviewers agree on whether a conversation passed. Without it, each person grades against their own idea of the policy, and every number after that inherits the disagreement.
Then list the kinds of input your agent will see, for example customer type, request type and channel. You can't test every combination, but you can cover every pair of values.
Three factors with 18 combinations can be covered pair by pair with nine tests. Most bugs appear when two factors interact, so this finds them cheaply ( Kuhn, Wallace and Gallo, 2004 ).

2. Collect two kinds of conversations, and keep them apart

You need two separate collections, for two different jobs.
Hunted conversations are ones you go looking for: tricky cases, edge cases, the situations where you suspect the agent breaks. They're for finding problems, so they're full of failures on purpose.
Representative conversations are a random sample of real traffic. Only these can tell you how often the agent fails in practice.
If you measure a failure rate on hunted conversations, the number is wrong from the start, because you picked them for being likely to fail. Label each conversation with which kind it is when you collect it, and never mix them in one number.

3. Read the conversations, and grade them yourself

Someone who knows the business reads real conversations and marks each one pass or fail. For every fail, they write one sentence naming the rule the agent broke.
Keep it to pass or fail. A 1-to-5 scale invites people to pick 3, and nobody can say what a 3 means for whether you ship.
Expect your definitions to change as you go.
Researchers who studied this found that people "need criteria to grade outputs, but grading outputs helps users define criteria" ( Shankar et al., 2024 ). Write the definitions down as versions, and don't mix grades made under different versions.
For the grades you'll use to test a judge, have two people grade the same conversations separately, then compare. If your two experts often disagree on what counts as a failure, fix the definitions before going further. A judge can't be checked against experts who don't agree with each other.

4. Group the failures into named problems

Your failure notes are raw. Sort similar ones into groups, and give each group a name and a one-line definition. Phrase each one as a yes-or-no question about a single conversation, such as "Did the agent promise a refund the policy doesn't allow?"
A yes-or-no question can be tested. "Was the response helpful?" can't.
Then decide when to stop reading. Keep going until almost every new failure you find fits a group you already have. A reasonable rule of thumb: at least 30 failures graded, at least two distinct problems found, and new failures that almost never need a new group.

5. Build a judge for each problem, and test it before you trust it

Write one judge per problem, asking exactly that yes-or-no question. Then test it on a fresh set of conversations your experts have graded, with plenty of both failures and passes. For one problem, that's about 55 of each.
Trust the judge only if it does well on both questions from earlier: catching real failures, and passing good conversations. "Does well" should mean at least 80% on both, even at the cautious end of what your test can show, and not just on average.
Use each graded test set once.
If you tweak the judge's prompt, test it again on the same set, and repeat until it passes, you've tuned the judge to those particular answers. The test no longer tells you how it does on conversations it hasn't seen. A revised judge gets a fresh set.

6. Check the judge for bias

A judge can agree with your experts and still be swayed by things that shouldn't matter. LLM judges are known to favor longer answers, to go along with claims made in the conversation, and to prefer writing from their own model family ( Ye et al., 2025 ).
Test for this directly.
Take conversations with a known right answer and change only one thing, such as padding the agent's reply with extra words. Also make a harmless change that shouldn't affect anything. If the judge changes its mind more often after the padding than after the harmless change, it's biased.
Here's an example from a demo of a refund agent, using a recorded judge and synthetic grades. The judge agreed with the grades 97% of the time. It still changed its verdict when the agent's answer was padded, and when the customer claimed something ("it arrived broken").
Refund customers make claims like that constantly. The 97% couldn't reveal the problem, because the test set never varied those things. One prompt revision later, tested on fresh grades, the judge passed.

7. Measure the real failure rate, and gate on the cautious number

Now run the trusted judge on your representative conversations from step 2. Then adjust its result for the mistakes you measured in step 5: it misses some real failures and flags some good conversations. Report the adjusted failure rate with its range, not as a single number.
If the judge is barely better than guessing, don't report a rate at all. A number from a judge like that is worse than no number.
To decide whether to ship, use the cautious end of the range. In the same refund demo, 162 of 200 real conversations passed. That's 81%, and the bar was 80%.
The release still failed the check. With 200 conversations, the cautious end of the range was 76%: the evidence couldn't rule out an agent that fails almost one customer in four. A dashboard showing 81% next to an 80% target would have shipped it.
Make sure your test set is big enough to pass at all. With an 80% bar, 10 out of 10 passing isn't enough evidence. If you expect your agent to pass about 95% of the time, plan on about 28 tests for each problem.

8. Prove the fix worked

When the check fails, don't rerun it hoping for a better result, and don't lower the bar. Fix the agent. But first, write down how big an improvement you expect, for example "failures drop by 10 percentage points", and don't change that target later.
Then run the old and new versions on the same inputs, and compare them input by input. Count only the inputs where the result changed. Accept the fix if:
  • failures went down by at least your target
  • nothing else got worse
When you check many things at once, one will look worse by chance alone, so set a higher bar for each individual check.
Keep the conversations you used while building the fix out of this comparison. Otherwise you're grading the fix on the examples it was tuned to.

9. Keep measuring in production

Passing in testing tells you about your test inputs. Real traffic changes. On a schedule, take a fresh random sample of at least 150 real conversations, run your trusted judge on it, and report the adjusted failure rate with its range.
Watch whether your inputs are changing shape: longer conversations, more turns, more tool calls. A judge tested on short conversations hasn't been tested on long ones.
Re-test the judge whenever something it depends on changes:
  • the judge's model or prompt
  • your definitions of what counts as a failure
  • the agent itself
  • the kind of traffic it sees
  • and at least once a quarter, even if nothing else changed
The third is the easiest to miss. When the agent changes, nothing in the judge's settings changes to remind you.

What it takes

The real cost is expert time. Someone who knows the business writes the spec, groups the failures and grades conversations. A second person grades the judge's test set.
For one problem, testing a judge takes on the order of 55 to 65 graded failures and the same number of passes. Each additional problem with its own judge needs its own grades. Running the judge costs little by comparison.
Spend that time on a few test sets you use once, not on many quick checks. A failure rate from a tested judge, reported with its range, is a number you can act on. A dashboard number from an untested judge isn't.
This approach doesn't cover everything. Tone and style don't reduce to a yes-or-no question with a right answer, so compare those side by side if you need to, but don't block a release on them. And if you're prototyping with no users and nothing at stake, reading a handful of conversations is enough.

Try the lifecycle with an open source tool

We built these nine steps into an open source library and command-line tool,  Calibrated Agent Evaluation . Each step is one command, and the safeguards are built in:
  • a judge can't grade until it has passed its tests and its bias check
  • a test set can't be used to approve a judge twice
  • it refuses to report a rate from too few conversations, or from a judge that's barely better than guessing
  • the release check fails your build when a problem's cautious number falls short
It reads conversation logs from common agent frameworks and observability tools, including OpenTelemetry, Strands, Inspect AI and Langfuse, and has an optional plug-in for  Amazon Bedrock AgentCore . It runs on your own machine. There's no hosted service.
The refund-agent examples above come from its bundled demo. The demo replays recorded judge answers, so it runs offline with no API keys or AWS account.
Its grades are synthetic. A rule applied the store's written return policy in place of a human reviewer, and the tool says so in its output.
What you need: Python 3.11.5 or later, and about ten minutes.
pip install "calibrated-agent-evaluation[stats]"
calibrated-agent-evaluation quickstart --certify
It sets up the demo and prints six commands to run in order:
Seeded 848 OrderBot refund traces into an offline local store
round 1 (label_round=1): 324 hunted traces (discovery frame) carry gold, 162 FAIL for trusted_broken_policy_tool and 162 PASS
round 2 (label_round=2): 324 hunted traces (discovery frame) carry gold, 162 FAIL for trusted_broken_policy_tool and 162 PASS
200 production-representative traces carry none; the rate is measured on them
The  documentation  has a  step-by-step guide  that follows the nine steps on your own agent. If you want the statistics behind each step, the  methods page  names and cites every one.
As of November 2026 it's at version 0.1.0, so expect changes before 1.0.
Try it on your own agent, and tell me in the comments what the release check caught that your dashboard missed.

Series: Agent Evaluation (1 article)

  1. 1
    Who's judging your LLM judge? A lifecycle for Agent Evaluation This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article