Amazon Bedrock Evaluations: Measure GenAI Model Quality
A builder's framework for Amazon Bedrock Evaluations: when to use programmatic metrics, LLM-as-a-judge, or human review — plus working judge code and a CI gate.
Series: Best Practices (7 articles)
- …
- 5Amazon Bedrock Evaluations: Measure GenAI Model Quality This article
Lead Data Scientist
You Can't Ship What You Can't Measure: A GenAI Evaluation Framework on Amazon Bedrock
Most teams I talk to can deploy a Bedrock model in an afternoon and then spend the next three months arguing about whether it's "good enough." Prompts get tweaked, someone eyeballs ten outputs, everyone nods, and it ships. That's not evaluation — that's vibes. And vibes don't survive a model swap, a prompt refactor, or a Friday deploy.
Evaluation is the part of the GenAI stack builders skip and regret. So here's a framework for doing it properly on AWS, plus code you can run today.
Three tiers of evaluation — pick per metric, not per project
The mistake is treating "evaluation" as one thing. It's three, and mature systems use all three for different metrics.
- Programmatic / deterministic. Exact match, F1, BERTScore, regex, JSON-schema validation. Cheap, instant, perfectly repeatable. Use these for anything with a checkable answer: classification labels, extracted fields, structured output. If a metric can be deterministic, it should be — never pay an LLM to check what
==can check.
- LLM-as-a-judge. A stronger model scores outputs on fuzzy qualities: correctness, completeness, faithfulness (hallucination detection), tone. Amazon Bedrock Evaluations has this built in with curated metrics, and since 2025 you can define custom metrics — your own judge prompt and rating scale. This is the workhorse for open-ended generation where there's no single right string.
- Human review. The ground truth you calibrate the other two against. Expensive and slow, so spend it deliberately: a periodic golden set, high-stakes categories, and disagreement audits where the judge and programmatic scores diverge.
A decision rule I use: deterministic if you can, judge if you must, human to calibrate.
Offline gate vs. production watch
These are two different jobs, and conflating them is why teams miss regressions.
Offline evaluation runs against a fixed golden dataset — inputs paired with known-good answers — before you ship. This is your regression suite. It answers "did this prompt change make things worse?" and it belongs in CI.
Production evaluation samples live traffic (you have no reference answers here) and scores reference-free metrics like faithfulness, answer refusal, and — for RAG — context relevance and citation precision. It answers "is quality drifting right now?" Bedrock's RAG evaluation (GA in 2025) covers both retrieval metrics (context relevance, coverage) and end-to-end metrics (correctness, faithfulness, citation coverage), which is exactly the split RAG systems need.
Make it a quality gate, not a dashboard
A number nobody blocks a deploy on is decoration. The highest-leverage move is turning offline evaluation into a CI gate: run the golden set on every prompt or model change, fail the build if the aggregate score drops below a threshold. Here's a minimal LLM-as-a-judge you can wire into a pipeline today, using the Converse API so it works across model providers:
python
Python
import json, boto3bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")JUDGE_MODEL = "anthropic.claude-3-5-sonnet-20241022-v2:0" # regional inference profileJUDGE_PROMPT = """You are a strict grader. Compare the ANSWER to the REFERENCE.Score correctness from 1 (wrong) to 5 (fully correct and complete).Respond ONLY as JSON: {{"score": <int>, "reason": "<one sentence>"}}QUESTION: {q}REFERENCE: {ref}ANSWER: {ans}"""def judge(q, ref, ans) -> dict:msg = [{"role": "user","content": [{"text": JUDGE_PROMPT.format(q=q, ref=ref, ans=ans)}]}]resp = bedrock.converse(modelId=JUDGE_MODEL, messages=msg,inferenceConfig={"temperature": 0, "maxTokens": 200})return json.loads(resp["output"]["message"]["content"][0]["text"])# Golden dataset: the regression suite you gate on.GOLDEN = [{"q": "What is the capital of Australia?", "ref": "Canberra"},{"q": "2 + 2 * 3 = ?", "ref": "8"},]def run_gate(get_answer, threshold=4.0) -> None:scores = []for row in GOLDEN:ans = get_answer(row["q"]) # your app under testverdict = judge(row["q"], row["ref"], ans)scores.append(verdict["score"])print(f"{row['q'][:40]:42} -> {verdict['score']} ({verdict['reason']})")avg = sum(scores) / len(scores)print(f"\nAverage: {avg:.2f} (threshold {threshold})")if avg < threshold:raise SystemExit(1) # fail the buildif __name__ == "__main__":run_gate(lambda q: "Canberra" if "Australia" in q else "8")
temperature=0 on the judge is not optional — a nondeterministic grader gives you a flaky test. Pin it, and pin the judge model version too.Cost and gotchas
- The judge costs tokens. A golden set of 50 items on a Sonnet-class judge is pennies per run — trivial next to a bad release. But don't judge 100k production rows; sample.
- Judges are biased. They favor longer answers and their own family's style, and they drift when you upgrade the judge model. Re-baseline against human labels whenever you change judges.
- Don't judge the deterministic stuff. If the output is a SKU or a boolean, assert it. Save the LLM for genuinely fuzzy quality.
- Version everything — prompt, judge model, golden set — or your scores aren't comparable across runs.
Why this matters — what I'd do
If I inherited an undated GenAI project, the first thing I'd build isn't a better prompt — it's a 30-line golden-set gate like the one above, running in CI on day one. Everything downstream (prompt tuning, model swaps, RAG changes) gets safer the moment you can see a regression instead of hearing about it from a user. Start with ten hand-picked examples and a threshold; graduate to Bedrock Evaluations' managed jobs and custom metrics when you need scale, responsible-AI metrics, or auditable reports.
How are you measuring GenAI quality today — golden-set gates, live sampling, or still eyeballing outputs? I'd genuinely like to know what's working for you.
#AWS #Bedrock #GenAI #LLMOps #MachineLearning
Series: Best Practices (7 articles)
- …
- 5Amazon Bedrock Evaluations: Measure GenAI Model Quality This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article