AWS Builder Center
Your Agent's Worst Bug Is Its Prompt. Here's How to Prove the Fix

Your Agent's Worst Bug Is Its Prompt. Here's How to Prove the Fix

A hands-on guide to improving an AI agent using Amazon Bedrock AgentCore Optimization. You analyze real traces, let the service recommend a better prompt from what actually failed, version the before and after as Configuration Bundles, and prove the improvement with a real A/B test. The running example is a code-security agent that reviews pull requests.

Your code-security agent is in production and mostly does its job. It also cries wolf: it flags safe code as vulnerable, occasionally misses a real SQL injection, and marks minor issues as critical. Nearly all of that behavior comes from the system prompt, and yet rewriting the prompt by hand is guesswork. Which instruction is causing the over-flagging? There's usually no way to tell just by reading it.
This guide shows you how to answer that with data instead of intuition, using Amazon Bedrock AgentCore Optimization. You never touch the agent's code. Instead, you let the service read your production traces, recommend an improved prompt based on what actually failed, and then prove the new prompt is better with a real experiment before promoting it. We use a code-security agent that scans pull requests as the running example, but the loop works for any Strands or agents developed using other frameworks.

Who Is This For?

You should read this if:
  • You run an agent in production and its quality is uneven (false positives, missed cases, inconsistent severity).
  • You want to improve a system prompt with evidence, driven by what actually failed in real traces, rather than rewriting it on a hunch.
  • You need to A/B test agent configurations with statistical rigor before rolling out a change.
  • You're building on AWS using Strands Agents and Amazon Bedrock AgentCore Runtime and want to use the Observability data you're already collecting.
If your agent isn't instrumented for tracing yet, start there. The entire optimization loop is fueled by traces.

The Core Idea: The Prompt Is a Config Value, Not Code

The mental shift that makes this whole workflow click:
Your agent's behavior is a configuration, not a code path. The system prompt, the model ID, and the tool descriptions are parameters you can version, swap, and test without redeploying the container.
In this lab the agent is one Strands  Agent  with five tools. What separates "v1, the original prompt that over-flags" from "v2, the improved prompt" is entirely external configuration. Same model, same tools, same code. Only the prompt changes.

v1 (control)
v2 (treatment)
System prompt
original (verbose, over-flags)
AI-optimized prompt from Recommendations
Model
Same model
Same model
Tools
Same 5 tools
Same 5 tools
Code
Identical
Identical
Holding the model constant is deliberate. It isolates the value of the prompt itself, so any change in quality you measure comes from the recommendation and not from a different model. That is what this guide is really about: what a better prompt is worth on its own.
AgentCore packages those parameters as Configuration Bundles, which are immutable, versioned snapshots of agent behavior. Think of them as git tags for how the agent thinks. The Runtime resolves which bundle to use per request, so a single deployed container can serve v1 and v2 at the same time. That is what makes a live A/B test possible without a blue/green deploy.
The five tools the agent orchestrates (the workflow AgentCore Optimization learns to tighten):
Tool
Purpose
get_pr_diff
Fetch the code diff for a PR
scan_dependencies
Check manifests for known CVEs
check_secrets
Detect hardcoded secrets and credentials
analyze_code_patterns
Find insecure patterns (SQL injection, XSS, unsafe calls)
generate_report
Aggregate findings into a structured report

The Optimization Loop, Step by Step

The whole workflow is five stages: Explore, Recommend, Bundle, A/B Test, and Deploy the winner.

Step 1: Explore the Traces

You can't optimize what you can't see. Every agent invocation on AgentCore Runtime is auto-traced to CloudWatch as OpenTelemetry spans, with no instrumentation code from you beyond enabling telemetry. Each session produces:
  • Spans that record the shape of the run: the top-level agent invocation, each model call, and each tool call, along with token usage, latency, and which tools ran.
  • Event records that capture the actual prompt and completion content, shipped alongside the spans.
Before optimizing, you look at what's failing. The v1 baseline is deliberately suboptimal and exhibits classic first-version failure modes:
  • Over-classification, where it flags safe patterns as vulnerabilities (false positives).
  • Missed patterns, where it misses obvious SQL injection or XSS.
  • Verbose tool calls, where it calls all five tools even when the diff doesn't warrant it.
  • Severity inflation, where it marks low-severity issues as critical.
Those patterns, sitting in the traces, are exactly what the Recommendations API reads.

Step 2: Generate a Recommendation

You hand your traces to the AgentCore Recommendations API and it writes you a better system prompt. You pass in the original prompt and a batch of your recorded sessions (a single recommendation caps at 20 sessions), and start a system-prompt recommendation job. In skeleton form:
client = boto3.client("bedrock-agentcore", region_name="us-east-1")

started = client.start_recommendation(
name="codesec-prompt-rec",
type="SYSTEM_PROMPT_RECOMMENDATION",
recommendationConfig={ ... }, # original prompt + session traces + evaluator
)

The job scores each session against a target evaluator you specify, which acts as the reward signal (the AWS examples use  Builtin.GoalSuccessRate , which measures whether the agent achieved the user's goals). It learns which behaviors correlate with failure, and returns a rewritten system prompt plus a plain-language explanation of what it changed and why. A single recommendation samples up to 20 sessions, so expect it to take several minutes.
Redaction note: The sample PRs contain fake secrets and exploit code, which is exactly what a security agent is meant to catch. Uploaded as-is, those traces can look like a real attack to the API's safety filter, which then rejects the batch. The fix is to blank out the fake secrets and defang the risky code before upload. Nothing is lost by doing so: the recommendation learns from how the agent behaved (the tools it ran, how wordy it was, how it rated severity), not from the secret text itself.

Step 3: Create Configuration Bundles

Now version the before and after. A Configuration Bundle binds the model, system prompt, and tool descriptions into an immutable, versioned artifact. For this experiment you create two:
  • v1 (control), the original prompt.
  • v2 (treatment), the recommended prompt, on the same model.
Because the model is held constant, the only variable between the two bundles is the prompt. When you later see v2 outperform v1, you can say exactly why: the improved prompt did it.

Step 4: A/B Test Through the Gateway

Split live traffic between the two bundles using an AgentCore Gateway Rule that routes each request to a bundle by weight. A weighted split holds exactly two variants whose weights add up to 100, which fits this experiment perfectly: for example, 70 percent of traffic to the original prompt and 30 percent to the improved one while you gather evidence.
In parallel, an online evaluation config scores live sessions continuously. Two things are happening across the loop, and it helps to keep them separate:
  • The recommendation (Step 2) points a target evaluator (the reward signal,  Builtin.GoalSuccessRate  in the AWS examples) at your recorded traces to diagnose failures and drive the rewrite.
  • The online evaluation (Steps 4 and 5) scores live traffic. It combines built-in quality evaluators (such as Helpfulness and Faithfulness) with a custom evaluator that checks the agent's verdict against known ground truth for each PR and returns a simple pass or fail.

Step 5: Analyze and Deploy the Winner

Score both variants against a held-out set of PRs (some genuinely vulnerable, some clean), run a test for statistical significance, and pick the winner: the prompt that is significantly better, or at least no worse, on quality. Then promote it by removing the traffic-split rule so all traffic flows to the winning bundle as the new default.
Here's how the results changed when moving from v1 (the original prompt) to v2 (the improved prompt), with the same model used throughout:
Metric
v1 (control)
v2 (treatment)
Read
False-positive rate
high (over-flags)
down
only flags what actually matter
Missed vulnerabilities
some
fewer or none
tighter instructions catch the real issues
Severity accuracy
inflated
calibrated
criticals are actually critical
Detection quality
baseline
holds or improves
no regression on genuine threats
Watch the shape of the result: the same model, guided by a better prompt, catches the real vulnerabilities while raising far fewer false alarms. Since the prompt is the only thing that changed, it gets all the credit.

Quick Reference

Stage
What it does
Evaluator
Output
Explore
Read the traces your Runtime already emits
(none)
Failure-mode signal in traces
Recommend
Let the service rewrite the prompt from what failed
Goal-based evaluator
Improved prompt + explanation
Bundle
Version the before/after (same model, different prompt)
(none)
Immutable v1 and v2 bundles
A/B Test
Split live traffic and score each session
Quality evaluators + custom ground-truth check
Live split + scored sessions
Deploy
Promote the winning prompt as the new default
Significance test on results
Winner as new default
Two evaluation stages, don't confuse them. The recommendation stage diagnoses failures in recorded traces to drive the rewrite. The online evaluation stage scores live sessions to decide the A/B test.

Practical Tips

  • Enable telemetry correctly, or you have nothing to optimize. Make sure both the run structure (spans) and the actual prompt and completion content (event records) are being exported. Recommendations and evaluations need the content, not just the structure.
  • Hold the model constant to isolate the prompt. Keeping the same model on both sides is what lets you say the prompt earned the improvement. If you also want to explore a cheaper model later, do that as a separate experiment so you never confound the two effects.
  • Make runs deterministic for evaluation. For a clean comparison, make sure each scan maps to exactly one configuration and one trace, rather than relying on random assignment while you're validating the setup.
  • Redact before you upload traces. Mask secrets and neutralize exploit tokens before sending traces to the Recommendations API, or the safety classifier may reject the batch.
  • Cap and cache. A recommendation caps at 20 sessions and takes minutes, so cache completed recommendations to avoid waiting on a live job during a demo or re-run.

A Note on Cost

Once you've proven that a disciplined prompt holds quality, you've also earned an option: a well-instructed prompt often lets a smaller, cheaper model do the same job. That is a natural follow-on experiment, run the exact same way, with a smaller model as the only change and the same evaluators deciding whether quality holds. But that is a second question. This guide is about the first one: proving the prompt itself made the agent better.

Conclusion

You don't improve an agent by rewriting its code. You improve it by treating the prompt, model, and tool descriptions as configuration you can measure and swap. The loop is always the same:
  1. Explore the traces your Runtime already emits.
  1. Recommend a better prompt from what actually failed.
  1. Bundle the before and after as immutable versions on the same model.
  1. A/B test the two prompts on live traffic, scored by real evaluators.
  1. Deploy the winner and start the next iteration from the new baseline.
For a code-security workload, that means fewer false alarms and better detection on every pull request, with no prompt engineer spending a week hand-tuning wording. The service read the traces, diagnosed the failures, wrote the fix, and gave you the tooling to prove it was safe before shipping. The agent stays constant. The prompt is the parameter you tune.

Resources

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article