Testing Production AI Applications on AWS

How Do We Test an Agentic System? From Graph Tests to Evals and Judge Calibration.

Elena van Engelen - Maslova

Lead Software Engineer | AWS Community Builder

Published Oct 15, 2026

Last modified Oct 16, 2026

We know how to test traditional software. In a unit test, we give a component a known input and assert the expected output. With white-box testing, we know the implementation and can exercise its branches and conditions. With black-box testing, we test a feature through its inputs and observable results.

Agentic systems challenge these approaches. We cannot inspect and cover the internal branches of a prompt because it does not have executable branches, and the same input does not always produce the same output [8]. In many cases, there is no single exact value we can assert.

So how do we test an agentic system? And how do we determine whether its output is good enough?

Example system under test

Consider a small agent graph from MockNest Serverless [7] that generates API mocks from an API specification and a user scenario.

The graph first parses and compresses the OpenAPI specification, then injects it into the generation prompt together with the user’s scenario. A model generates the mocks via Amazon Bedrock [4], which are then validated against the API specification. If validation succeeds, the graph returns all generated mocks. If errors are found and retries are available, the graph passes the validation errors and invalid mocks to a correction prompt. A model attempts to correct them, and the result is validated again. If errors remain, or the retry limit has been reached, the graph returns only the valid mocks.

Although this graph is relatively small, it contains prompts, deterministic validation, decisions, retries, and several possible outcomes. How do we test that it follows the correct path without calling the model every time?

The Koog strategy graph in code

The graph could be implemented using various available libraries for agentic systems, such as Strands Agents SDK, available for Python and TypeScript [1]. Our graph is implemented with Koog, JetBrains’ open-source framework for building AI agents on the JVM [2, 9]. I chose Koog because the implementation is written in Kotlin, and its type-safe Kotlin DSL allows me to define the nodes, decisions, and retry paths directly in the existing codebase. It serves a similar purpose to the Strands Agents SDK, but is designed for Kotlin and Java applications.

The simplified strategy graph looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
strategy<SpecWithDescriptionRequest, GenerationResult>("mock-generation") {
    val setupNode by node("setup") {
        prepareSpecification()
    }

    val generateNode by node("generate") {
        generateMocks()
    }

    val validateNode by node("validate") {
        validateMocks()
    }

    val correctNode by node("correct") {
        applyCorrections()
    }

    edge(nodeStart forwardTo setupNode)
    edge(setupNode forwardTo generateNode)
    edge(generateNode forwardTo validateNode)

    edge(validateNode forwardTo nodeFinish)
        .onCondition {
            errors.isEmpty() || attempt > maxRetries
        }

    edge(validateNode forwardTo correctNode)
        .onCondition {
            errors.isNotEmpty() && attempt <= maxRetries
        }

    edge(correctNode forwardTo validateNode)
}

Graph test

Koog comes with its own utilities for testing agent graphs [3]. In this article, I focus on topology tests: unit tests of the graph itself. Instead of running a full traversal, they check the wiring directly: which nodes exist, which nodes are reachable, and which edge a node takes for a given output. Koog exposes this through its testGraph API [3]. No real model is called, so these tests run in the normal build without any API cost.

Why test the wiring instead of just running the graph and checking the result? A result-only test observes the destination. A wrong turn inside the graph can still reach the right destination, so a result-only assertion can silently pass even when an edge is miswired. Topology tests check the map. They catch that class of bug, such as accidental rewiring of the generate → validate → correct loop or an off-by-one in a retry condition, directly, and point at the exact edge.

One clarification, because "topology testing doesn't run anything" is easy to overstate.
Topology tests never run the node bodies (no setup/generate/validate/correct
work, no LLM, no parser). But asserting an edge does evaluate that edge's condition
against an output you hand it. So a retry comparison like attempt <= maxRetries really does
run, just in isolation, without the nodes around it. That is exactly what makes it a unit
test of the wiring. The Koog documentation describes the assertEdges API [3]; the library source shows that each goesTo assertion resolves the edge against the output you provide, without executing the node [17]. Koog performs these checks when the agent starts, so the test still calls agent.run(), with a mock LLM executor and stubbed collaborators that exist only to let that run complete [23].

Nodes and reachability

The first check confirms that the expected nodes exist and are connected: the entry flows into setup and generation, the validation loop is intact, and every node can eventually reach finish:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
testGraph<Request, Result>("mock-generation") {
    val start    = startNode()
    val finish   = finishNode()
    val setup    = assertNodeByName<Request, Ctx>("setup")
    val generate = assertNodeByName<Ctx, Ctx>("generate")
    val validate = assertNodeByName<Ctx, Ctx>("validate")
    val correct  = assertNodeByName<Ctx, Ctx>("correct")

    assertReachable(start, setup)
    assertReachable(setup, generate)

    // validation loop: generate -> validate -> correct -> validate
    assertReachable(generate, validate)
    assertReachable(validate, correct)
    assertReachable(correct, validate)

    // no node is a dead end
    assertReachable(validate, finish)
    assertReachable(correct, finish)
}

Happy path

Given that the generated mocks validate successfully, the graph should skip correction and return all mocks. In topology terms: when validate produces an output without errors, it must route to finish, regardless of the attempt number or retry budget.

Retry limit

If errors remain and there are no retries left, the graph must not loop forever. Once the
retry budget is exhausted, the validate node routes to finish instead of back to
correct, so the loop terminates and it returns the mocks that passed validation.

This is a condition on an edge, so it's a perfect fit for a topology test. We assert which
edge validate takes for inputs on either side of the boundary (the last
allowed retry, and one attempt past it), plus the happy path, without running any node logic:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// No model calls, no node bodies run. maxRetries = 1.
// We evaluate the "validate" edge conditions for a given output.
testGraph<Request, Result>("mock-generation") {
    val finish   = finishNode()
    val validate = assertNodeByName<Ctx, Ctx>("validate")
    val correct  = assertNodeByName<Ctx, Ctx>("correct")

    assertEdges {
        // no errors -> happy path, go straight to finish
        validate withOutput ctx(errors = emptyList(), attempt = 1) goesTo finish
        // errors remain, a retry is still available -> go to correction
        validate withOutput ctx(errors = listOf("..."), attempt = 1) goesTo correct
        // errors remain, retries exhausted -> stop, go to finish (no infinite loop)
        validate withOutput ctx(errors = listOf("..."), attempt = 2) goesTo finish
    }
}

In the repository, the retry boundary is a parameterised test that covers several retry budgets, including zero [23].

The graph works. But how do we test the output?

Every graph test can be green while the model still produces wrong results. So how do we test the actual output?

What is an eval?

An evaluation, usually shortened to eval, is a repeatable way to measure how well an AI system performs against defined criteria [10]. Each eval scenario provides an input, describes the expected behaviour, and defines how the generated output will be assessed.

Unlike a traditional test, an eval does not always compare the result with one exact expected value. Different outputs can be valid as long as they satisfy the requirements of the scenario. The assessment can combine deterministic checks with an LLM judge [10].

Evals can target an individual prompt or the complete AI workflow. I started by evaluating the final outcome.

Outcome eval

An outcome eval runs the complete graph and evaluates what the user actually receives.

Each scenario gives the system an API specification and a user request. The complete graph generates the mocks, validates them, and corrects them when necessary. The eval then assesses the final mocks returned to the user.

Wherever code knows the answer, use code. Schema conformance, endpoint selection, response types, status codes and, in many cases, data consistency across related endpoints can be checked deterministically. There is no reason to ask an LLM to judge them. This keeps the checks reproducible and reserves model-based evaluation for semantic questions.

A semantic judge is used only for criteria that are difficult to express as deterministic assertions. For example, does the result satisfy the user’s request? Is the generated data realistic?

A scenario passes only when the required deterministic and semantic checks pass. The eval suite also records latency and cost, with generation and judge costs tracked separately.

Running the eval suite

These evals call a real model through Amazon Bedrock and have associated costs, so I keep them separate from the normal build and run them as "remocal" tests: the test code runs locally while calling a remote model through Bedrock [11]. I only run them when the associated prompts, model, or graph change.

Each scenario defines concrete expected behaviour and applies both deterministic and semantic checks. It measures much more than whether the generated response can be parsed.

Tracking generation and judge cost separately

The agent and the judge both call Bedrock through the same AWS SDK BedrockRuntimeClient, using the Converse API [4]. Every Converse response includes a usage object with the number of input and output tokens consumed by that call [18]. That gives us everything we need to estimate cost, but only if we know which calls belong to the application and which belong to the judge.

In the eval suite, I wrap the Bedrock client in a small decorator. It delegates every call to the real client and records the token usage of each converse call, tagged with the current phase:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class TokenUsageCapturingClient(
    private val delegate: BedrockRuntimeClient,
    private val tokenUsageStore: TokenUsageStore
) : BedrockRuntimeClient by delegate {

    override suspend fun converse(input: ConverseRequest): ConverseResponse {
        val phase = tokenUsageStore.currentPhase   // GENERATION or JUDGE
        val response = delegate.converse(input)
        tokenUsageStore.record(
            TokenUsageRecord(
                inputTokens = response.usage?.inputTokens ?: 0,
                outputTokens = response.usage?.outputTokens ?: 0,
                phase = phase
            )
        )
        return response
    }
}

The eval sets the phase before each step. It switches to GENERATION before running the agent graph, so every call it makes is counted, including correction retries. It then switches to JUDGE before the semantic check:

1
2
3
4
5
6
7
8
9
10
11
tokenUsageStore.currentPhase = TokenPhase.GENERATION
val result = agent.generateFromSpecWithDescription(request)

tokenUsageStore.currentPhase = TokenPhase.JUDGE
val semanticPassed = runSemanticJudge(scenario, mappingsJson)

val judgeRecords = tokenUsageStore.getJudgeRecords()
val judgeCost = CostCalculator.calculateCost(
    inputTokens = judgeRecords.sumOf { it.inputTokens },
    outputTokens = judgeRecords.sumOf { it.outputTokens }
)

Cost is estimated by multiplying the token counts by the per-token on-demand price from the Bedrock pricing page [21]. Prices differ by model and region, so they are configurable. The decorator lives only in the test source set, so the production code path is unchanged. The full implementation is in the MockNest repository [19, 20].

Keeping the two costs apart matters because they answer different questions. Generation cost is what the application costs in production, including the extra calls made by retries. Judge cost is the price of measuring quality, and it is paid only when the eval suite runs.

Example eval results

The eval suite reports first-pass quality, quality after correction, scenario pass rate, cost, and latency.

This run covered 7 REST scenarios based on the Petstore API, generated with Amazon Nova Pro in eu-west-1. 88% of the generated mocks passed structural validation on the first attempt, and 100% after correction; the scenario pass rate, including the semantic check, was also 100%. The whole run cost $0.0449: $0.0393 for generation and $0.0056 for the judge, or $0.0064 per scenario on average. Average generation latency was 3.4 seconds per scenario. These figures describe a single run and will vary between runs.

This distinction between structural and semantic pass is important. A structural pass rate of 100% shows that the mocks conform to the API specification after correction. It does not necessarily mean that they satisfy the user’s request. That is what the complete scenario evaluation measures.

Recording both first-pass and after-retry quality is also important. A final structural pass rate of 100% does not show how often the correction prompt had to repair the initial result.

LLM as a judge – how can we trust it?

A semantic judge makes it possible to evaluate output quality at scale, but it also introduces another probabilistic component. Its decision is not ground truth. Before using its scores, we need to calibrate it against human judgement [12]. AWS supports model-as-judge evaluations, including correctness and completeness metrics, but those scores still need validation against the intended task [5, 6].

At least initially, manual testing is back! Start with a representative set of model outputs that you have reviewed and labelled. Include clear passes, clear failures, borderline cases, and examples of known failure modes. Give the judge the same outputs and compare its decisions with the human labels.

The evaluation criteria must be specific. Instead of asking:

Is this a good mock?

Ask questions such as:

Does the generated data match the scenario requested by the user?
If the user requested bird data, do the mocks actually describe birds?
Does the complete set of mocks represent the requested use case?

When the judge and the human label disagree, inspect the reason. First confirm that the human label is correct. If it is, make the judge instructions or scoring criteria more precise. Pay particular attention to false passes, where the judge accepts output that a human considers incorrect [13].

A simple calibration process looks like this:

Create and label a representative set of outputs.
Let the judge evaluate the same outputs.
Compare the decisions and inspect disagreements.
Refine the scoring criteria or judge instructions.
Run the calibration set again.

Calibration is not a one-time activity. Rerun it after changing the judge model, judge instructions, or scoring criteria. Changes to the generation prompt may also produce different kinds of output, so they can require another calibration run.

The calibration set of scenarios should evolve as well. When a judge makes a new mistake, add that scenario as another labelled example to the calibration set. It then becomes a regression case for future changes [13].

I also use a different model for judging than the model used by the agent: Amazon Nova Pro generates the mocks, and OpenAI GPT-OSS 120B on Bedrock acts as the judge [20]. This is a precaution against some shared failure patterns, not a guarantee of independence: research has found that LLM judges may favor their own generations [12, 14]. It does not replace calibration.

Only after calibration do the judge scores become useful engineering feedback rather than an unverified opinion from another model.

Was the initial generation good, or did correction fix the result?

An outcome eval can show that the final result is correct, but it may hide where the quality came from. A weak generation prompt could produce invalid mocks that the correction step successfully repairs. The scenario passes, but the additional model call increases latency and cost.

The first-pass result helps identify this problem. In the earlier example, 88% of the REST mocks passed structural validation on the first attempt, while 100% passed after correction. The difference shows how much work the correction step is doing.

Focused prompt evals help assess and improve individual AI nodes.

A generation prompt eval provides an API specification and a user request, invokes the generation node, and checks whether the generated mocks:

Match the requested endpoints
Use response types and status codes from the specification
Do not invent parameters or operations
Contain realistic and consistent data

Evaluating the correction node is more complicated because the correction step continues the conversation started by the generation step. Both nodes append their prompt to the same Koog write session, which retains the conversation history across LLM requests [15, 22]. Its behaviour depends on the original prompt, the first model response, and the validation errors. Calling only the correction prompt would remove that context and test something different from the production workflow.

A focused correction eval must therefore reproduce the conversation up to that point. It starts with a representative original request and invalid model response, adds the validation feedback, and then invokes the correction node in the same session context. The eval checks whether the corrected result:

Fixes the reported errors
Preserves content that was already correct
Avoids introducing new validation errors
Still satisfies the original user request

These focused evals make problems easier to diagnose. Generation evals measure the quality of the initial result. Correction evals measure whether the system can repair known errors when given the same context it receives in the real graph.

Outcome evals tell us whether the complete agentic workflow produces what we want. Focused prompt evals help tune AI nodes individually.

Testing pyramid for agentic systems

I think about testing AI agents in three layers. This is my adaptation of the traditional testing pyramid, rather than the conventional pyramid of unit, integration, and end-to-end tests [16]:

Bottom — Graph tests and prompt evals. Topology tests verify the expected nodes, reachability, and edge conditions, including retry and failure routing, without real model calls [3]. Focused prompt evals call a real model to assess individual AI nodes.

Middle — Evals of the full flow. Outcome evals run the complete AI workflow with a real model and measure whether the final output meets the user's requirements. They combine deterministic checks with semantic assessment and track latency and cost.

Top — Judge calibration. Before trusting semantic eval scores, compare the LLM judge's verdicts with human-labelled cases. Use disagreements and new failure modes to refine the rubric and extend the calibration dataset [12, 13].

The top layer does not mean judge calibration is performed on every execution. It is a quality check on the evaluator itself, repeated whenever its model or rubric changes and when new failure modes emerge.

Takeaways

1. Test the graph without model API cost

Use topology tests to verify the graph wiring: the expected nodes, their reachability, and the edge conditions that drive decisions, retries, and failure paths.

2. Evaluate the prompt nodes

Use focused evals to determine whether individual prompts produce the required behaviour with a real model.

3. Evaluate the final outcome

Run the complete workflow and assess the result that the user actually receives.

4. Calibrate the judge

Compare the judge with human-labelled examples and refine its instructions when they disagree.

These layers complement each other. A graph can be wired correctly while its prompts produce poor results. Individual prompts can perform well while the complete workflow fails to meet the user's request. Even when the final outcome appears correct, the judge may accept mistakes unless its decisions have been calibrated against human-labelled examples.

AI quality is not something we test once. Every new failure should improve the graph tests, eval scenarios, or judge calibration set.

References

[1] Strands Agents — SDK quickstart overview, language support (Python and TypeScript). https://strandsagents.com/docs/user-guide/sdk/quickstart/overview/

[2] Koog — Custom strategy graphs. https://docs.koog.ai/custom-strategy-graphs/

[3] Koog — Testing agent pipelines, graph structures, and mocked model interactions. https://docs.koog.ai/testing/

[4] Amazon Bedrock — Inference using the Converse API. https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html

[5] Amazon Bedrock — Evaluate model performance using another LLM as a judge. https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html

[6] Amazon Bedrock — Evaluation metrics, including correctness and completeness. https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-metrics.html

[7] MockNest Serverless — Project source repository. https://github.com/elenavanengelenmaslova/mocknest-serverless

[8] Amazon Bedrock — Influence response generation with inference parameters. https://docs.aws.amazon.com/bedrock/latest/userguide/inference-parameters.html

[9] JetBrains — Koog source repository. https://github.com/JetBrains/koog

[10] Amazon Bedrock — Evaluate the performance of Amazon Bedrock resources. https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html

[11] Yan Cui — My testing strategy for serverless applications (remocal testing). https://theburningmonk.com/2022/05/my-testing-strategy-for-serverless-applications/

[12] Zheng et al. (2023) — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685

[13] Shankar et al. (2024) — Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. https://arxiv.org/abs/2404.12272

[14] Panickssery et al. (2024) — LLM Evaluators Recognize and Favor Their Own Generations. https://arxiv.org/abs/2404.13076

[15] Koog — LLM sessions and manual history management. https://docs.koog.ai/sessions/

[16] Ham Vocke — The Practical Test Pyramid (Martin Fowler). https://martinfowler.com/articles/practical-test-pyramid.html

[17] JetBrains — Koog TestingFeature.kt, graph and edge assertion implementation. https://github.com/JetBrains/koog/blob/main/agents/agents-test/src/commonMain/kotlin/ai/koog/agents/testing/feature/TestingFeature.kt

[18] Amazon Bedrock API Reference — Converse (response usage: input and output token counts). https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html

[19] MockNest Serverless — TokenUsageCapturingClient.kt, Bedrock client decorator that captures token usage per phase. https://github.com/elenavanengelenmaslova/mocknest-serverless/blob/main/software/infra/aws/generation/src/test/kotlin/nl/vintik/mocknest/infra/aws/generation/ai/eval/TokenUsageCapturingClient.kt

[20] MockNest Serverless — BedrockPromptEvalTest.kt, eval suite with judge model configuration and separate generation and judge cost. https://github.com/elenavanengelenmaslova/mocknest-serverless/blob/main/software/infra/aws/generation/src/test/kotlin/nl/vintik/mocknest/infra/aws/generation/ai/eval/BedrockPromptEvalTest.kt

[21] Amazon Bedrock — Pricing. https://aws.amazon.com/bedrock/pricing/

[22] MockNest Serverless — MockGenerationFunctionalAgent.kt, Koog strategy graph with generation and correction nodes. https://github.com/elenavanengelenmaslova/mocknest-serverless/blob/main/software/application/src/main/kotlin/nl/vintik/mocknest/application/generation/agent/MockGenerationFunctionalAgent.kt

[23] MockNest Serverless — MockGenerationGraphStructureTest.kt, Koog topology tests for the mock generation graph. https://github.com/elenavanengelenmaslova/mocknest-serverless/blob/main/software/application/src/test/kotlin/nl/vintik/mocknest/application/generation/agent/MockGenerationGraphStructureTest.kt

# ai-agents # amazon-bedrock # testing # integration-testing # unit-testing

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Testing Production AI Applications on AWS

Elena van Engelen - Maslova

Example system under test

The Koog strategy graph in code

Graph test

Nodes and reachability

Happy path

Retry limit

The graph works. But how do we test the output?

What is an eval?

Outcome eval

Running the eval suite

Tracking generation and judge cost separately

Example eval results

LLM as a judge – how can we trust it?

Was the initial generation good, or did correction fix the result?

Testing pyramid for agentic systems

Takeaways

1. Test the graph without model API cost

2. Evaluate the prompt nodes

3. Evaluate the final outcome

4. Calibrate the judge

References