AWS Builder Center

Builder Center

Your agent feels better. Prove it.

Your agent feels better. Prove it.

A new Claude Code skill drops a custom rubric, test cases, an LLM-as-judge, and a scoring harness into any agent project in one command — and across four real experiments, it caught regressions and judge drift that vibes and generic benchmarks would have missed.

Erez Weinstein

Here's the scene. You tweak a prompt. Swap Opus for Sonnet. Move from LangGraph to CrewAI because someone tweeted about it. Re-run the demo, squint at the output, mumble "yeah that's better," and hit merge.
Congratulations. You just shipped vibes.
I've done it. You've done it. We've all done it. And the honest truth is: most of the time, we have no idea if the new version is actually better, actually worse, or just different in a way our brain happens to like today.
Generic benchmarks don't save us. MMLU doesn't know what your agent does. HELM doesn't know your tools. The only thing that can tell you whether your thing got better at your task is a rubric you wrote for your task. So I built a Claude Code skill that does exactly that —   eval-layer   — and then I beat it up on four real projects to see what it would catch.
Spoiler: it caught things vibes would have missed every single time.

What you get

Point the skill at any agent project and it drops in four things:
  1. A rubric — 3 to 5 dimensions, each with "what a 1 looks like, what a 5 looks like," no "good"/"bad" hand-waving
  1. A test set — cases across difficulty tiers, a few reference-graded so you know the judge isn't making things up
  1. A judge prompt — LLM-as-a-judge with calibration examples so it doesn't freestyle
  1. An eval harness — runs your agent, sends output to the judge, aggregates, spits out a dashboard
You get two numbers back:
  • Weighted score  [0, 1]  — how good the output is, per your rubric
  • Leniency  [−1, +1]  — whether the judge itself is drifting. This is the sleeper hit. Grading LLMs with LLMs without a second opinion is how you end up gaslit by your own eval.

Four experiments, four "huh, would've missed that" moments

1. The terse prompt that nuked itself —  seo-agent 

Before the numbers, here's what "an eval" actually looks like for this agent so the rest of the post isn't abstract.
The agent. In: a topic like  "ai coding assistants" . Out: a structured SEO brief — target keyword, search intent, 4-8 section outline, word count, CTA.
The rubric (4 dimensions, weights sum to 1.0). Two examples of what level descriptors actually look like:
  •  keyword_relevance  (weight 0.30, 1-5) — does the target keyword match real search demand, not just restate the topic?
  • 1: "Off-topic, tautological ('guide to X' for topic X), or contradicts the trend data"
  • 3: "Reasonable but could be more specific; outline touches some rising queries"
  • 5: "Precisely scoped to a realistic search phrase; outline clearly synthesizes the trend and SERP signal"
  •  outline_actionability  (weight 0.35, 1-5) — can a writer execute the outline without asking clarifying questions?
  • 1: "Vague headings ('Introduction', 'Overview', 'Conclusion') — no writer could start"
  • 3: "Most sections are specific but ordering is weak, or 1-2 sections are placeholder-like"
  • 5: "Each section is a concrete writeable unit; sequence tells a clear narrative; no gaps or overlap"
Plus  completeness  (0.25) and  schema_compliance  (0.10). No "good"/"bad" — every level is a sentence a judge (or a human) can apply.
A test case:
YAML
id: easy-01
input: "ai coding assistants"
context: "trend=rising, volume=22000, competition=high.
          Rising queries: 'claude code vs cursor', 'ai pair programmer'."
reference_scores: { keyword_relevance: 5, outline_actionability: 4,
                    completeness: 4, schema_compliance: 3 }
The  reference_scores  are what makes leniency work — the judge grades this case blind, and we compare its scores to these. If the judge is systematically 1.5 points higher than the reference, that's drift, not quality.
Now the experiment. Same agent, same model, same 8 cases. I just rewrote the system prompt to be "punchier." Spoiler, it was not punchier.
Subject
Weighted
Pass rate
Full prompt
0.931
1.00
Terse prompt
0.213
0.00
From "ships to prod" to "every single case fails" in one prompt rewrite. The rubric told me in about 40 seconds what a human reading the outputs would have taken an afternoon to notice.
For multi-subject runs (comparing prompts, models, or frameworks against the same task), the skill also auto-generates a self-contained HTML dashboard — leaderboard, per-dimension radar, per-case heatmap, latency bars — so you can eyeball the whole comparison at once instead of diffing markdown reports.


2. The framework that looked great until it didn't —  agent-frameworks-bench 

Five frameworks, one task, same model, same prompt, same 8 cases. Who wins?
Framework
Weighted
Pass rate
OpenAI Agents
0.960
0.12
LangGraph
0.953
1.00
PydanticAI
0.949
1.00
Strands
0.945
0.75
CrewAI
0.873
1.00
Read that top line again. OpenAI Agents has the best average dimension score and the worst pass rate. Translation: when it works, it's gorgeous. 7 out of 8 times, structured output on Bedrock just… doesn't. If I'd only tracked weighted score, I would have declared OpenAI Agents the winner and shipped something that fails 88% of the time.
Two numbers. Always two numbers.

3. "Haiku is probably fine, right?" —  incident-triage-agent 

SRE-style triage agent. Real question: can I use Haiku here and save the Opus budget for something that actually needs it?
Model
Weighted
Pass rate
Avg latency
Opus 4.6
0.927
1.00
26.5 s
Haiku 4.5
0.859
1.00
11.9 s
Haiku: ~2× faster, roughly 3× cheaper, 7 points worse, same pass rate. For a triage first-pass where a human reviews anyway? That's a very easy yes. Without the rubric, "Haiku is cheaper" is a vibe. With it, it's a tradeoff I can actually defend in a design doc.

4. The judge threw shade at Sonnet —  langgraph-plan-execute-bench 

Plan-and-execute agent answering quantitative questions over a tiny knowledge graph. Opus vs Sonnet, 8 cases.
Model
Weighted
Pass rate
Leniency
Opus
0.904
0.875
+0.013
Sonnet
0.798
0.875
−0.160
Same pass rate, but Sonnet scores 10 points lower. First instinct: "yeah, Sonnet's worse at this." But look at the leniency column — the judge was measurably harsher on Sonnet's phrasing style. So part of the gap isn't Sonnet being worse, it's the judge being pickier about how Sonnet writes.
Without leniency, I'd have written the wrong conclusion in the report. That signal alone paid for the whole skill.
(Both models choked on the same hard case — a filter-and-aggregate trap that required excluding an employee from another company. Equal-opportunity failure. The rubric showed that clean.)

The pattern

Four projects, same three lessons:
  • One number lies. Weighted score alone hides pass-rate collapses. Pass rate alone hides quality drops. You need two measurements that can disagree with each other, or you're flying blind.
  • The judge is also on trial. Leniency catches "the judge got stricter on this subject" vs "this subject got worse." Those look identical in a single-number report.
  • Rubrics catch regressions fast. 0.931 → 0.213 in under a minute. No stand-up required.

Try it

Bash
git clone https://github.com/erezweinstein5/eval-layer ~/.claude/skills/eval-layer
Then in any agent project:  /eval-layer Add an eval layer to . 
Start small. Five cases. Three dimensions. One reference-graded case for leniency. That's enough to stop shipping on vibes.
If you can't score your agent, you can't ship it. If you can't measure the judge, you can't trust the score. Everything else is storytelling.
# amazon-bedrock# agentic-ai# agents# ai
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article