AWS Builder Center

Builder Center

I Built an NBA AI Agent. Then I Tried to Break It

I Built an NBA AI Agent. Then I Tried to Break It

Week 1 of my transition from Data Engineering to AI Engineering. I built an NBA preseason agent with Strands SDK and Amazon Bedrock. Then I started testing what happens when things break. These are my field notes on the failures, tradeoffs, and engineering decisions behind the build. No hype, no perfect demo. Just the work of figuring out what makes an AI agent more reliable, one failure at a time.

Alexander Bolaño Cervantes

Senior Data Engineer | Building the path toward AI Engineering | Entrepreneur
Coming from Data Engineering, I thought building an AI agent was simple: connect a prompt to a few tools, and you're done.
 
Well, not quite.
 
I built an NBA preseason agent using Claude Haiku, Amazon Bedrock, and the Strands Agents SDK. Then I tried to break it.
 
Not to prove agents don't work, but to understand what makes them reliable.

Why I Tried to Break It

In Data Engineering, I was used to testing expected outputs. With AI agents, a convincing answer doesn't necessarily mean everything worked correctly. I wanted to find out what happens when things go wrong.
 
It broke in ways a pipeline never would. Each time it did, I faced the same choice: trust the model a little more, or move the risky part into deterministic code. That choice, repeated across five failures, is what this article is about. The end to end diagram of how this agent actually works is in ARCHITECTURE.md , if you want the full picture before the failures.
 
 

Five Moments Where Data Engineering and AI Engineering Disagreed

ProblemPipeline assumptionAgent reality
The agent explored on its own. It queried six dates without being asked and, when a call failed, invented an infrastructure cause without evidence.A missing parameter is rejected or handled with a fixed default. A failure cause is never improvised.The model can take initiative and produce plausible explanations without sufficient evidence. The fix used two layers: the prompt asks before taking that action, and the code enforces a hard limit of fourteen days that no instruction can override.
The original cost estimate came out ten times lower than the cost measured in CloudWatch.Cost is predictable from the code and remains stable if the functionality does not change.Every turn can resend the prompt and tool schemas. Token usage can grow with each reasoning step. Measuring actual usage revealed what the initial estimate missed.
The agent had to calculate standings from real game data. Three different models produced three different incorrect answers.Once a calculation passes a test, trust the implementation and keep regression tests around it.One successful run is not enough to establish reliability. The fix was to move the arithmetic out of the model and into deterministic code.
The model contradicted the user's explicit premise about the current roster, asserting information from its training instead.The source of truth is explicit, and conflicting assumptions should not silently override it.The model can give prior knowledge too much weight, even when the prompt provides contrary context. A prompt change improved accuracy from zero to seventy-five percent in my test, but it did not turn the behavior into a guarantee.
Lambda stopped returning real data after the timeout fixes had appeared to work, while the same request worked from my laptop.If the failure looks the same everywhere, the external source is probably down. Retry later.The same request can behave differently across execution environments. The comparison narrowed the investigation, but the root cause still needed to be established. The request's network path and origin became part of the investigation. To be continued in the next article.
 
These are observations from one project, not universal laws about every model or agent. But each one changed how I approached the system. Row three's full evaluation, the same question run five times with five different answers, and three models each wrong in a different way on the exact same real data, is written up with the raw transcripts in EVALUATION.md . The same five rows above, with nothing trimmed, are also in PARADIGM_SHIFTS.md .

Two of These, in Code

1. Prompts are not hard limits

Once I saw the model improvise beyond the intended scope, a better prompt alone was not enough. The tool needed a boundary that no instruction could talk its way past.
1
2
3
4
5
if span_days > MAX_RANGE_DAYS:
raise InvalidToolArgument(
f"date range spans {span_days} days, but at most "
f"{MAX_RANGE_DAYS} days are allowed per call"
)
 
The prompt helps guide behavior. The code enforces the constraint.
 
That distinction matters whenever a tool call can trigger expensive work, retrieve too much data, or do something the user did not ask for.

2. Don't ask the model to do deterministic arithmetic

When three models returned incorrect standings from the same real data (Haiku, Nova Lite and Sonnet), switching to a larger model was not the first fix I needed. The arithmetic belonged in code.
 
1
2
3
4
5
6
7
8
9
10
11
12
def _compute_standings(games):
records = {}

for game in games:
if game["status"] != "Final":
continue

home_won = game["home_score"] > game["away_score"]
# Update wins and losses deterministically.
# The model does not calculate the standings.

return records
 
The snippet leaves the record update intentionally abbreviated; the important design choice is that the calculation is performed by deterministic code and can be tested independently.
 
Neither fix is clever. Both are the same move: identify the part of the system that must behave consistently, then stop asking the model to be that part. Three more fragments built the same way, the time budget guard, the SDK hook that catches duplicate tool calls, and the client side fix for row five below, are in SELECTED_IMPLEMENTATION.md .
 

What I Did Not Reach For

  1. I did not reach for Amazon Bedrock AgentCore, a larger model, or a vector database to solve these five problems. That does not mean these tools are not useful. It means they were not the right solutions for the specific failures I observed.
  2. The fixes were usually smaller and less glamorous: explicit constraints, deterministic calculations, measured costs, and clearer handling of uncertainty.
 
It is tempting to respond to every agent failure by adding another layer of AI infrastructure. Sometimes that is appropriate. Sometimes the right answer is a few lines of ordinary code. The engineering work is figuring out which problem you actually have.

The Failure I Have Not Solved Yet

The agent's Lambda function could not reliably retrieve data from the NBA Stats API, while the same request worked from my laptop at the same time.
 
I have a few hypotheses about what might be causing this, and there are several approaches I could try to work around it. But I deliberately left this failure unresolved.
 
Why? Because I wanted to document what actually happened, not turn every failure into a success story just to make the architecture look good.
 
In real engineering, knowing how to fix something and proving why it failed are two different things. I would rather be transparent about an unresolved issue than claim a root cause I have not verified.
 
Deployed Lambda answering the same question honestly instead of inventing data
 
That comparison made a general API outage less likely, but it did not prove the root cause. The remaining investigation focused on differences between the execution environments and their network paths.
 
Same question, asked twice, through the full agent, one environment at a time. Locally, real data came back.
 
1
2
3
4
5
6
7
Tool #1: compare_players
status: retrying, retry_count: 0, error_type: ReadTimeout
status: retrying, retry_count: 1, error_type: ReadTimeout
status: failed, retry_count: 2, error_type: ReadTimeout
I wasn't able to retrieve preseason stats for either player. The tool
encountered issues fetching data for Giannis Antetokounmpo, and it ran
out of time to check Klay Thompson's data.
 
 
Three real, timed retries, all the same error. The second player was never even attempted, the time budget guard skipped it on purpose rather than let the invocation run past Lambda's own limit. The whole call finished in thirty six seconds, well inside the sixty second ceiling, with a plain answer and zero invented numbers.
 
This was a useful reminder: application code can be correct while the system still fails for reasons outside the agent itself.
 
In Data Engineering, I was used to investigating pipelines, transformations, permissions, and infrastructure. Here, I was also investigating how the model behaved, what it inferred, when it called tools, and how those tools behaved in different environments.
 

What I Learned

  • The biggest lesson was not that AI agents are unreliable. It was that reliability cannot depend on the model getting everything right.
  • Some decisions belong in the model. Others belong in deterministic code, explicit constraints, measurements, and infrastructure.
  • I started this project to build an agent. I ended up learning how much engineering it takes to make one trustworthy. And I still have one failure left to understand.
 
 
This is part of my field notes as I transition from Data Engineering to AI Engineering. The goal is not to present a perfect demo, but to document the failures, tradeoffs, and engineering decisions behind building a more reliable agent.
 
The companion repo has the rest: nba-agent-field-notes 
 
 

Let's Connect

I’m always happy to connect with people building at the intersection of AI, data, and engineering. Whether you want to discuss this project, exchange ideas, or explore a collaboration, feel free to reach out.
 
# agentic-ai# strands-agents# aws-community-builders# ai# agents
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article