I Built and Evaluated an AWS AI Agent: What Actually Improved the Results?
Building an AI agent is surprisingly easy. Building one that is reliable, measurable, and actually better after every change is a different problem. That became the focus of my latest AWS experiment.
Building an AI agent is surprisingly easy.
Building one that is reliable, measurable, and actually better after every change is a different problem.
That became the focus of my latest AWS experiment.
Instead of simply creating an agent, asking it a few questions, and deciding that it “worked,” I wanted to build a small agent on AWS, establish a baseline, introduce improvements one at a time, and measure whether those changes actually helped.
The goal was simple:
Don't optimize the feeling that the agent is better. Measure it.
My experiment uses Strands Agents with Amazon Bedrock, a small evaluation dataset, controlled experiments, and Amazon Bedrock/Strands evaluation capabilities.
The Experiment
The project is a small AWS Cloud Architecture Assistant.
I want the agent to answer questions such as:
- Which AWS service should I use for this requirement?
- Should this EC2 workload be public or private?
- When should I use NAT Gateway versus VPC endpoints?
- How should I design a highly available application?
- What security risks exist in this architecture?
- Which AWS service is responsible for a particular capability?
The first version is intentionally simple.
1
2
3
4
5
6
7
8
9
10
11
12
User
|
v
Strands Agent
|
v
Amazon Bedrock
|
+---- AWS knowledge/tools
|
v
Answer
The important part is that I don't stop here.
I need to know whether the agent's answers are actually improving.
IMAGE 1 — Baseline Architecture
Place an architecture diagram here showing:
1
2
3
4
5
6
7
8
9
10
11
User
↓
Strands Agent
↓
Amazon Bedrock
↓
AWS knowledge/tools
↓
Response
↓
Evaluation
1. Why I Started With a Baseline
Before adding tools, memory, complicated prompts, or retrieval, I created a baseline agent.
This is important because otherwise it becomes difficult to answer a simple question:
Did the new feature improve the agent, or did I just make the architecture more complicated?
The baseline agent receives a question and sends it to a Bedrock model through Strands.
Strands supports Amazon Bedrock as a model provider and can work with multimodal inputs, tool calling, guardrails, and prompt caching.
A minimal Python example looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from strands import Agent
from strands.models import BedrockModel
model = BedrockModel(
model_id="YOUR_BEDROCK_MODEL_ID"
)
agent = Agent(
model=model,
system_prompt="""
You are an AWS Solutions Architect assistant.
Give technically accurate answers.
Explain why you recommend an AWS service.
Prefer secure and production-oriented designs.
Clearly identify assumptions and limitations.
"""
)
response = agent(
"Should my application EC2 instances be public or private?"
)
print(response)
This gives us our first measurable system.
2. The First Problem: Accuracy Is Not Enough
Suppose the agent answers:
“Use a public EC2 instance because it needs internet access.”
That answer sounds reasonable if we only check whether the model responded.
But architecturally, it can be a poor recommendation.
A better answer might be:
Keep the EC2 instance in a private subnet and provide controlled outbound connectivity through a NAT Gateway or appropriate VPC endpoints, depending on the workload.
This illustrates an important point:
An AI response can be fluent and still be technically wrong.
So I needed an evaluation strategy.
3. Building a Small Evaluation Dataset
I created a small dataset of AWS architecture questions.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[
{
"question": "Which service distributes HTTP traffic across multiple EC2 instances?",
"expected": "Application Load Balancer"
},
{
"question": "Should a database normally be directly accessible from the internet?",
"expected": "No"
},
{
"question": "Which AWS service provides object storage?",
"expected": "Amazon S3"
},
{
"question": "Why would an EC2 instance in a private subnet use a NAT Gateway?",
"expected": "For outbound internet access without giving the instance a public IP"
}
]
For a real project, I would expand this into multiple categories:
1
2
3
4
5
6
7
8
9
Networking
Security
Compute
Storage
Databases
High Availability
Cost Optimization
Architecture Design
Troubleshooting
This is much more useful than randomly asking questions.
4. Defining the Metrics Before Testing
I wanted to measure more than one thing.
My evaluation dimensions were:
Correctness
Is the technical answer correct?
Relevance
Does the response actually answer the question?
Completeness
Does it cover the important architectural considerations?
Safety
Does the recommendation avoid introducing unnecessary security risks?
Clarity
Can another builder understand the recommendation?
Hallucination
Does the agent invent AWS services, features, limits, or configurations?
Amazon Bedrock's evaluation capabilities include metrics such as correctness, completeness, faithfulness, helpfulness, coherence, relevance, and instruction following. Custom metrics can also be defined for use cases that need more specific evaluation criteria.
5. Experiment 1 — Better System Prompt
My first experiment was intentionally simple.
I improved the system prompt.
Instead of:
1
Answer AWS questions.
I used:
1
2
3
4
5
6
7
8
9
10
11
12
13
You are an AWS Solutions Architect assistant.
Follow these rules:
1. Prioritize security.
2. Prefer least-privilege designs.
3. Explain architectural trade-offs.
4. Distinguish public and private resources.
5. Mention availability considerations.
6. Consider cost when recommending architecture.
7. Never invent AWS services or capabilities.
8. If information is uncertain, clearly state the uncertainty.
9. Explain why the recommended architecture is appropriate.
Then I ran exactly the same evaluation dataset.
This gives us a controlled experiment:
1
2
3
4
Same model
Same questions
Same evaluation
Different system prompt
That makes the comparison meaningful.
IMAGE 2 — Experiment Pipeline
Place an image here showing:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Evaluation Dataset
|
+----------+----------+
| |
v v
Baseline Agent Improved Agent
| |
v v
Responses Responses
| |
+----------+----------+
|
v
Evaluation
|
v
Score Comparison
6. Experiment 2 — Give the Agent Tools
The next question was:
What happens if the agent can retrieve information instead of relying entirely on the model's internal knowledge?
This is where agent architecture becomes more interesting.
A tool-enabled design looks like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
User
|
v
Strands Agent
|
+------> Bedrock Model
|
+------> AWS Documentation Tool
|
+------> Search / Knowledge Tool
|
+------> Calculator
|
v
Final Answer
Strands has a tool ecosystem covering capabilities such as file operations, HTTP requests, Python execution, AWS integration, memory, and other agent tools.
The key principle is:
The model should decide when a tool is necessary instead of blindly calling every tool.
7. Why Tools Can Improve Accuracy
Consider this question:
“What is the current AWS service limit for X?”
A model may have outdated information.
Instead of guessing, the agent can retrieve authoritative information.
This creates:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Question
|
v
Agent
|
| Does this require external/current information?
|
YES
|
v
Tool / Documentation
|
v
Evidence
|
v
Model
|
v
Answer
This pattern is particularly important for cloud architecture because AWS services, features, quotas, pricing, and capabilities change over time.
8. Experiment 3 — Retrieval
The next improvement is giving the agent a controlled knowledge source.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
AWS Documentation
|
v
Knowledge Base
|
v
Retrieval
|
v
Agent
|
v
Grounded Response
Instead of asking the model to remember everything, we give it access to relevant information.
The goal isn't:
“Give the model more information.”
The goal is:
“Give the model the right information at the right time.”
9. But Retrieval Creates Another Problem
Retrieval can also fail.
Imagine the user asks:
“How should I secure a three-tier AWS application?”
The retrieval system might return:
- VPC documentation
- Security Groups
- IAM
- WAF
- RDS
- CloudWatch
- unrelated networking documentation
More context doesn't automatically mean a better answer.
Too much irrelevant context can make the model less focused.
So retrieval itself needs evaluation.
10. Evaluating Retrieval
For a RAG-style workflow, I would measure:
Context Relevance
Did we retrieve information relevant to the question?
Context Coverage
Did we retrieve the information necessary to answer?
Faithfulness
Did the final answer stay grounded in the retrieved context?
Correctness
Did the final answer actually solve the user's problem?
Amazon Bedrock supports evaluation of RAG workflows, including retrieval and retrieve-plus-generation scenarios.
That means we can evaluate not only the model but also the entire retrieval pipeline.
11. The Evaluation Loop
At this point the architecture becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
User Question
|
v
Strands Agent
|
+--------+--------+
| |
v v
Bedrock Model Knowledge Tool
| |
+--------+--------+
|
v
Final Answer
|
v
Evaluator
|
+------------+-------------+
| | |
Correctness Relevance Faithfulness
| | |
+------------+-------------+
|
v
Results
This is the part I find most valuable.
We're no longer simply building an AI application.
We're building an AI engineering feedback loop.
12. Experiment 4 — Model-as-a-Judge
Manual evaluation works for ten questions.
It becomes painful with hundreds or thousands.
Amazon Bedrock supports model evaluation using an LLM as a judge, where a second model evaluates generated responses against selected metrics.
A simplified flow is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Question
|
v
Generator Model
|
v
Generated Answer
|
v
Judge Model
|
+--> Correctness
+--> Completeness
+--> Relevance
+--> Helpfulness
+--> Faithfulness
We can also create custom metrics.
For an AWS architecture assistant, a useful custom metric could be:
Architecture Safety
Score:
1
2
3
4
0 = Unsafe
1 = Needs major changes
2 = Mostly safe
3 = Production-oriented
The evaluator could check:
- Public exposure
- IAM permissions
- Encryption
- Network isolation
- Secrets management
- Availability
- Failure handling
This is much closer to how I actually want to judge an architecture assistant.
13. A Custom Evaluation Metric
A custom evaluator could use instructions similar to:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Evaluate the AWS architecture response.
Check whether the recommendation:
1. Avoids unnecessary public exposure.
2. Uses least privilege.
3. Separates public and private workloads.
4. Protects sensitive data.
5. Considers high availability.
6. Identifies important trade-offs.
7. Avoids unsupported AWS claims.
Return:
score: 0-3
reason:
Short explanation of the score.
Amazon Bedrock custom metrics support an evaluator model, custom instructions, and a rating scale.
14. The Experiment That I Would Not Skip
One of the most important lessons from reading engineering-focused AI experiments is this:
Don't only report the successful run.
If a change makes the architecture worse, that result is valuable.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Experiment A
Baseline prompt
Result: baseline
Experiment B
Longer prompt
Result: +2%
Experiment C
Large retrieval context
Result: -5%
Experiment D
Filtered retrieval
Result: +8%
Experiment E
Tool + filtered retrieval
Result: +12%
The failed experiment tells us something.
In this example:
More context ≠ better context.
15. What I Would Record for Every Experiment
For reproducibility, I would keep:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Experiment ID
Model ID
Model version
System prompt version
Dataset version
Tool configuration
Retrieval configuration
Temperature
Max tokens
Evaluation metrics
Timestamp
Cost
Latency
Result
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
{
"experiment": "exp-004",
"model": "bedrock-model-id",
"dataset": "aws-architecture-v1",
"prompt_version": "v3",
"retrieval": "filtered",
"temperature": 0.2,
"metrics": {
"correctness": 0.91,
"relevance": 0.94,
"faithfulness": 0.93
}
}
This turns experimentation into something another engineer can reproduce.
16. The Architecture I Would Keep
After the experiments, I would aim for a simple architecture rather than the most complicated one.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
USER
|
v
+----------------+
| Strands Agent |
+-------+--------+
|
+-----------+-----------+
| |
v v
Amazon Bedrock Knowledge / Tools
| |
+-----------+-----------+
|
v
Final Response
|
v
Evaluation
|
+---------+---------+
| |
v v
Metrics Experiment Store
The important design principle is separation:
Generation → Retrieval/Tools → Evaluation
Each part can be improved independently.
17. What I Learned
The biggest lesson from this experiment wasn't a particular model or prompt.
It was the engineering process.
Lesson 1 — Establish a baseline
Without a baseline, improvement is difficult to prove.
Lesson 2 — Change one important thing at a time
If I change the model, prompt, retrieval system, and tools simultaneously, I don't know what caused the improvement.
Lesson 3 — Accuracy isn't enough
A technically correct answer can still be incomplete, unsafe, or poorly explained.
Lesson 4 — Retrieval needs evaluation
Adding a knowledge base doesn't automatically make an agent better.
Lesson 5 — Failed experiments are useful
A negative result can tell us which direction not to take.
Lesson 6 — Keep the agent architecture simple
Every new tool introduces another failure mode.
Lesson 7 — Measure cost and latency
A model that is slightly more accurate but dramatically more expensive may not be the right production choice.
18. From Demo to Production
A demo can look like:
1
2
3
4
5
Prompt
↓
LLM
↓
Answer
A production AI application needs much more:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
User
|
v
Authentication
|
v
Agent
|
+---- Model
|
+---- Tools
|
+---- Retrieval
|
+---- Guardrails
|
v
Response
|
v
Evaluation
|
v
Observability
|
v
Cost / Quality Monitoring
That's the difference between:
“I built an AI demo.”
and
“I built an AI system that I can measure and improve.”
19. The Next Experiment
My next step would be to turn this small architecture assistant into a proper benchmark.
I would create approximately:
1
2
3
4
5
6
100 Networking questions
100 Security questions
100 Compute questions
100 Storage questions
100 Database questions
100 Architecture questions
Then divide them into:
1
2
3
4
5
Training / development
|
+---- Evaluation set
|
+---- Unseen scenarios
The unseen set is particularly important.
If the agent only performs well on questions similar to what I used during development, I haven't demonstrated strong generalization.
The real test is:
Can the architecture assistant solve AWS scenarios it has never seen before?
20. Final Takeaway
The most interesting part of building AI applications isn't simply getting the model to answer.
It's understanding why it answered correctly, why it failed, and whether the next version is genuinely better.
My current mental model is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
BUILD
↓
BASELINE
↓
MEASURE
↓
CHANGE ONE THING
↓
EVALUATE
↓
ANALYZE FAILURE
↓
IMPROVE
↓
REPEAT
AWS gives us the building blocks to do this with services such as Amazon Bedrock, while Strands Agents provides an agent framework that can connect models with tools and workflows. Strands' evaluation framework similarly organizes experiments around cases, tasks, evaluators, and detectors, which fits naturally into this iterative approach.
For me, this is the bigger lesson:
AI engineering isn't just about making an agent smarter. It's about building a system that lets you prove whether it actually became better.
And that's the direction I'm continuing to explore in my AWS learning journey: build → measure → learn → improve. 🚀☁️🤖
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article