Beyond Correctness with Kiro & Strands Evals
Agent Evaluation using Strands Eval SDK Evaluators
In Part 1, we implemented the first half of the 7-Step Agent Evaluation Strategy, focusing on prompt quality, functional correctness, and execution traces. These stages answered an important question:
Did the agent produce the expected response?
For many LLM applications, that level of evaluation may be sufficient.
Agentic AI changes the equation.
Agents maintain conversations, invoke tools, retrieve information, follow instructions, and make autonomous decisions. Evaluating only the final response no longer provides enough confidence to deploy these systems into production.
In this article, we continue our evaluation journey by implementing Stages 4–6 of the evaluation strategy using Strands Agent Evaluators. Rather than evaluating a single response, we now evaluate the overall interaction—measuring conversation quality, Responsible AI behaviors, and whether the user ultimately achieved their goal.
🔗 Quick Recap: Where Part 1 Left Off
In Part 1, we implemented the foundation of the evaluation pyramid.
These stages primarily evaluate individual interactions.
By the end of Part 1, we had confidence that the agent could generate correct responses and execute the expected workflow.
Now we move beyond evaluating responses and begin evaluating agent behavior.
Why Evaluation Changes After Stage 3
As agents become more autonomous, evaluating only the final response becomes increasingly insufficient.
Consider a research assistant.
The final answer may be correct, but important questions still remain:
- Did the agent maintain context across multiple interactions?
- Did it behave safely when presented with harmful or biased prompts?
- Did the user actually accomplish their objective?
- Would this interaction be considered successful in production?
These questions cannot be answered by evaluating a single prompt-response pair.
Instead, they require evaluating the entire interaction, including conversations, execution traces, safety behavior, and business outcomes.
This is precisely where Stages 4–6 of the evaluation strategy come into play.
Telemetry: The Foundation for Higher-Level Evaluation
Stages 1 and 2 primarily rely on the agent's output.
Higher-level evaluations require much more.
They require visibility into how the agent executed the task.
This is where OpenTelemetry becomes the foundation of the evaluation pipeline.
Every LLM invocation, tool call, reasoning step, and response is captured as telemetry spans. Using the Strands In-Memory Session Mapper, these spans are transformed into a structured conversation session that higher-level evaluators can analyze. During execution, the completed spans are mapped into a conversation trajectory.
Conceptually, the evaluation pipeline looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
flowchart TD
A[User] --> B[Strands Agent]
B --> C1[LLM Calls]
B --> C2[Tool Calls]
B --> C3[Agent Events]
C1 --> D[OpenTelemetry<br/>Execution]
C2 --> D
C3 --> D
D --> E[Strands In-Memory Session Mapper]
E --> F[Structured Conversation Session]
F --> G1[Trace Evaluations]
F --> G2[Session Evaluations]
F --> G3[Responsible AI Evaluations]
G1 --> H[Outcome Evaluation]
G2 --> H
G3 --> H
H --> I[Evaluation Report / Metrics]Without telemetry, higher-level evaluators such as CoherenceEvaluator, GoalSuccessRateEvaluator, or TrajectoryEvaluator would have no visibility into how the agent behaved during execution.
Telemetry becomes the bridge between agent execution and agent evaluation.
📊 Complete Evaluator Coverage
The quickstart notebook demonstrates 17 evaluators across 7 scenarios, covering five stages of the evaluation strategy.
| Stage | Evaluator | Measures |
|---|---|---|
| 2. Functional | OutputEvaluator | Rubric-based output quality |
CorrectnessEvaluator | Factual accuracy | |
HelpfulnessEvaluator | User-perceived usefulness | |
Contains | Deterministic keyword validation | |
| 3. Trace | FaithfulnessEvaluator | Grounded in retrieved content |
ResponseRelevanceEvaluator | Response matches the query | |
InstructionFollowingEvaluator | Follows format constraints | |
ConcisenessEvaluator | Not overly verbose | |
RefusalEvaluator | Appropriately declines out-of-scope | |
ToolSelectionAccuracyEvaluator | Correct tool selection | |
ToolParameterAccuracyEvaluator | Correct tool parameters | |
TrajectoryEvaluator | Tool call sequence is logical | |
| 4. Session | CoherenceEvaluator | Context maintained across turns |
InteractionsEvaluator | Multi-turn interaction quality | |
| 5. Responsible AI | HarmfulnessEvaluator | Resists harmful content |
StereotypingEvaluator | Avoids bias and stereotypes | |
| 6. Outcome | GoalSuccessRateEvaluator | User achieved their goal |
One notebook. Seven scenarios. Seventeen evaluators. Five stages of the evaluation strategy.
Self-Hosted Evaluation Makes Continuous Testing Practical
One of the biggest advantages of this implementation is that the complete evaluation workflow runs entirely on self-hosted Ollama models.
- Application Model:
llama3.2:3b - Evaluation Judge:
qwen3.5:4b
Running evaluations locally provides several advantages:
- No API keys
- No token charges
- No external rate limits
- Fully reproducible evaluation pipelines
- Suitable for enterprise environments
This makes comprehensive evaluation practical throughout the development lifecycle instead of limiting it to pre-release validation.
Architecture NoteA common enterprise strategy is:* Run deterministic evaluators on every commit.
- Execute LLM-as-Judge evaluations during nightly builds.
- Perform session and outcome evaluations before production releases.This balances evaluation confidence with execution cost while providing rapid feedback during development.
🚀 Try It Yourself
📚 Jupyter Notebook: ai_research_assistant_eval_quickstart.ipynb
The notebook demonstrates:
- ✅ 17 Strands Agent Evaluators
- ✅ 7 evaluation scenarios
- ✅ OpenTelemetry-based session capture
- ✅ Self-hosted evaluation using Ollama
- ✅ Functional, Trace, Session, Responsible AI, and Outcome evaluations
Run the notebook:
1
2
3
4
ollama pull llama3.2:3b
ollama pull qwen3.5:4b
jupyter notebook ai_research_assistant_eval_quickstart.ipynbKey Takeaways
- Evaluating only the final response is not enough for production-grade AI agents.
- Higher-level evaluations rely on telemetry to understand how an agent behaves throughout an interaction.
- Session, Responsible AI, and Outcome evaluations complement functional and trace evaluations, providing a more complete assessment of agent quality.
- Strands Agent Evaluators offer a composable framework that enables multiple evaluation dimensions to be executed within a single evaluation pipeline.
- Self-hosted evaluation makes continuous enterprise-scale testing both practical and cost-effective.
What's Next
With the evaluation foundation now in place, the next phase of this series shifts from measuring agent quality to strengthening agent resilience.
In the upcoming articles, we'll explore AI Security Architecture for Agentic AI, including:
- Red Teaming
- Adversarial Testing
- Prompt Injection
- Jailbreak Techniques
- Building secure and trustworthy enterprise AI agents
Conclusion
In Part 1, we established confidence in the agent's prompts, responses, and execution traces.
In Part 2, we extended that confidence by evaluating conversations, Responsible AI behaviors, and business outcomes using Strands Agent Evaluators.
Together, these stages demonstrate that agent evaluation is not a single metric—it is a layered strategy. As agents become more autonomous, evaluation must evolve beyond response quality to include execution, safety, conversations, and user success.
By combining OpenTelemetry, Strands Agent Evaluators, and a structured 7-Step Evaluation Strategy, we can build AI systems that are not only intelligent but also observable, trustworthy, and production-ready.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article