AWS Builder Center

Builder Center

Test Your Prompts Like Code: A Lightweight Evaluation Harness for Amazon Bedrock

Prompt changes ship without tests more often than any other kind of change. This guide shows how to build a small, repeatable evaluation harness with Amazon Bedrock that scores prompt versions on real test cases, so improvements are measured instead of guessed.

The problem with “it looks better”

Most teams iterate on prompts by trying a few examples and judging the output by eye. That works for a demo. It fails in production, because a prompt tweak that fixes one case quietly breaks three others, and nobody notices until a user does.
Software engineering solved this decades ago with automated tests. The same discipline applies to prompts, and you don’t need a heavy framework to start. A small harness with a fixed set of test cases and a pass rate is enough to turn prompt engineering from opinion into measurement.

The idea

An evaluation harness has three parts:
  1. A test set: inputs paired with what a correct output must satisfy
  1. A runner: sends each input through a prompt version and the model
  1. Checks: automated rules that decide pass or fail for each output
Run it before every prompt change, compare pass rates, and ship only what improves or holds the line.

Step 1: Build the test set

Start with 15 to 30 cases. Quality matters more than volume, so include the following:
  • Typical inputs your system sees daily
  • Edge cases: empty input, very long input, ambiguous wording
  • Past failures: every bug you fix becomes a permanent test
Define expectations as checks rather than exact strings, because model output varies in wording:
python
TEST_CASES = [
    {
        "input": "Order #4521 arrived damaged and I want a refund.",
        "must_contain": ["refund"],
        "expected_category": "refund_request",
    },
    {
        "input": "How do I change the email on my account?",
        "must_contain": [],
        "expected_category": "account_support",
    },
    {
        "input": "asdf ???",
        "must_contain": [],
        "expected_category": "unclear",
    },
]

Step 2: Write the runner

This example uses the Bedrock Converse API, which gives one consistent interface across models, so you can swap the model without rewriting the harness.
python
import json
import os
import boto3

bedrock = boto3.client("bedrock-runtime")
MODEL_ID = os.environ["MODEL_ID"]   # any Bedrock model you have access to

def run_prompt(system_prompt: str, user_input: str) -> str:
    response = bedrock.converse(
        modelId=MODEL_ID,
        system=[{"text": system_prompt}],
        messages=[{"role": "user", "content": [{"text": user_input}]}],
        inferenceConfig={"maxTokens": 300, "temperature": 0},
    )
    return response["output"]["message"]["content"][0]["text"]
Setting  temperature  to 0 reduces variation between runs, which makes comparisons between prompt versions fairer.

Step 3: Define the checks

Ask the model for structured output, then verify it with plain code. Deterministic checks are fast, free, and perfectly repeatable:
python
def evaluate(case: dict, output: str) -> dict:
    results = {}

    # Check 1: output is valid JSON
    try:
        parsed = json.loads(output)
        results["valid_json"] = True
    except json.JSONDecodeError:
        return {"valid_json": False, "correct_category": False, "has_keywords": False}

    # Check 2: category matches expectation
    results["correct_category"] = parsed.get("category") == case["expected_category"]

    # Check 3: required keywords appear
    text = output.lower()
    results["has_keywords"] = all(k in text for k in case["must_contain"])

    return results

def score_prompt(system_prompt: str) -> float:
    passed = 0
    for case in TEST_CASES:
        output = run_prompt(system_prompt, case["input"])
        checks = evaluate(case, output)
        if all(checks.values()):
            passed += 1
    return passed / len(TEST_CASES)

Step 4: Compare prompt versions

python
PROMPT_V1 = "Classify the customer message. Reply with JSON: {\"category\": \"...\"}"

PROMPT_V2 = (
    "Classify the customer message into exactly one of: refund_request, "
    "account_support, unclear. Reply with JSON only, no extra text: "
    "{\"category\": \"<one of the three>\"}"
)

print(f"V1 pass rate: {score_prompt(PROMPT_V1):.0%}")
print(f"V2 pass rate: {score_prompt(PROMPT_V2):.0%}")
Now the decision is simple. If V2 scores higher and nothing regressed, ship it. If it scores lower, you caught the problem before your users did.

Going further: model-as-judge

Some qualities can’t be checked with code, such as tone, helpfulness, or factual faithfulness to a source document. For those, a second model call can act as a judge, scoring the output against a written rubric.
Use it carefully. Judge models have biases, so write the rubric explicitly, keep it narrow (one criterion per judgment), and periodically spot-check the judge’s scores against your own reading. Treat it as a complement to deterministic checks, never a replacement.

Making it part of your workflow

  • Store test cases and prompts in version control alongside your code
  • Run the harness automatically in CI whenever a prompt file changes
  • Fail the build if the pass rate drops below a threshold you choose
  • Add every production failure to the test set, so the suite grows smarter over time

Permissions and setup

Your role needs  bedrock:InvokeModel  for the model you use, and the model must be enabled for your account and region. Costs scale with test set size and prompt length, but a 30-case suite is typically very cheap to run.

Takeaway

The teams that improve GenAI applications fastest aren’t the ones with the cleverest prompts. They’re the ones who can measure whether a change helped. A small harness, a good test set, and a habit of running it before every change is the cheapest quality investment in the stack.
# generative-ai# amazon-bedrock# machine-learning# testing
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article