AWS Builder Center

Builder Center

When 200 OK Isn’t Success: Evaluating AI Agents in Production

When 200 OK Isn’t Success: Evaluating AI Agents in Production

A production AI agent can execute every tool call successfully and still produce the wrong result. This article explores a practical framework for measuring agent quality through task completion, groundedness, tool-use reliability, observability, and cost per successful task.

Traditional applications give us familiar signals of success.
The API returned 200 OK.
The database query completed.
The function executed without an exception.
Latency stayed within the expected range.
For an AI agent, all of those things can happen and the final result can still be wrong.
Imagine an agent that receives a request, retrieves information from a knowledge base, calls two tools, generates a response, and returns it successfully. From an infrastructure perspective, the workflow looks healthy.
But what if the agent selected the wrong document? What if it misunderstood the tool output? What if the response confidently included information that was never present in the retrieved evidence?
The execution succeeded.
The task did not.
This distinction changes how we need to think about monitoring agentic systems in production.

Operational Success Is Only the First Layer

For a traditional service, metrics such as availability, latency, throughput, and error rate tell us a great deal about system health.
They remain important for AI agents, but they answer only one question:
Did the system execute correctly?
For an agent, we also need to ask:
Did the system accomplish the user's goal correctly?
I find it useful to separate production evaluation into three layers:
  1. Operational reliability
  2. Agent quality
  3. Business outcome
Each layer answers a different question.

Layer 1: Operational Reliability

This is the layer most engineering teams already know how to monitor.
Useful signals include:
  • invocation failures
  • latency
  • throttling
  • tool errors
  • retries
  • model response time
  • token consumption
If an external API is unavailable or an agent repeatedly times out, infrastructure monitoring should expose it.
But consider an agent that completes all of its calls successfully and produces an unsupported answer.
None of those operational metrics necessarily indicate a failure.
That is where the second layer becomes essential.

Layer 2: Agent Quality

Agent quality measures whether the behavior and output of the agent were actually useful and correct.
Several metrics become important here.

Task Completion

Did the agent accomplish what the user asked it to do?
This sounds straightforward, but defining completion can be surprisingly difficult.
For example, consider an agent asked to:
Find the highest-risk compliance issue in this document and explain why it should be prioritized.
A response existing is not evidence that the task was completed.
A better evaluation might check whether the agent:
  1. analyzed the relevant document,
  2. identified a valid risk,
  3. ranked it correctly,
  4. supported the conclusion with evidence, and
  5. returned the expected output.
This means task completion should be defined around the intended outcome rather than around execution.

Groundedness

Groundedness asks whether the response is supported by the information available to the agent.
This becomes especially important when an agent uses retrieval, enterprise documents, databases, or external tools.
Suppose an agent retrieves five policy documents but generates a statement that none of those documents support.
The response may sound convincing. It may even be well structured.
But it is not grounded.
For higher-risk workflows, I would treat groundedness less like a secondary metric and more like a quality gate.
If the evidence does not support the answer, successful task execution should not be enough for the result to pass.

Tool-Use Reliability

Agentic systems introduce another source of failure: tools.
It is not enough to ask whether a tool call succeeded.
We should also evaluate:
  • Was the correct tool selected?
  • Were the correct arguments supplied?
  • Were tools called in the appropriate order?
  • Did the agent interpret the result correctly?
  • Did it call a tool unnecessarily?
  • Did it stop when sufficient information was available?
A tool can return a perfectly valid response to the wrong query.
From an infrastructure perspective, that interaction is successful. From an agent-quality perspective, it is a failure.

Layer 3: Business Outcome

Once an agent is technically reliable and produces acceptable results, another question becomes important:
Is operating this agent actually worthwhile?
This is where metrics such as cost become meaningful.
A metric I find particularly useful is:
Cost per successful task
rather than simply:
Cost per request
Suppose Agent A costs $0.03 per request and Agent B costs $0.05.
At first glance, Agent A appears more efficient.
But imagine that Agent A successfully completes only 60% of tasks while Agent B successfully completes 95%.
Optimizing request cost without considering outcome quality could lead us toward the cheaper but less effective system.
A more useful relationship is:
Cost per successful task = Total agent cost / Number of successfully completed tasks
The same principle can be extended to metrics such as:
  • human escalation rate
  • successful tasks per session
  • time saved per workflow
  • retries per successful task
  • cost of failed tasks
This connects technical evaluation to actual business value.

So Which Metric Comes First?

If I had to prioritize groundedness, task completion, and cost, I would not treat them as three competing metrics.
Instead, I would give them different roles.
Groundedness → Quality gate
For workflows where unsupported information creates meaningful risk, an answer should meet a minimum groundedness threshold before being considered successful.
Task completion → Primary outcome
Once basic quality requirements are satisfied, task completion tells us whether the agent actually accomplished the user's objective.
Cost per successful task → Optimization metric
After reliability and quality reach acceptable levels, cost becomes an optimization target.
This produces a hierarchy:
Grounded enough? → Task completed? → What did successful completion cost?
The exact thresholds should depend on the application.
A creative assistant and an enterprise compliance agent should not necessarily have the same evaluation policy.

Observability Needs Both Traces and Evaluations

Operational metrics tell us that something happened.
Traces help explain how it happened.
Evaluations help determine whether the result was good.
A useful production loop therefore looks like this:
User Request
↓
Agent Execution
↓
Model + Tool Calls
↓
Trace and Operational Metrics
↓
Quality Evaluation
↓
Failure Analysis
↓
Prompt / Model / Tool Improvement
↓
Re-evaluation
This is important because aggregate metrics alone can hide the reason an agent failed.
Suppose task completion drops from 92% to 81%.
That tells us something changed.
A trace can help answer the next question: why?
Perhaps the agent selected the wrong tool. Perhaps retrieval quality deteriorated. Perhaps a prompt change altered the reasoning path. Perhaps an external service returned unexpected data.
Evaluation tells us where quality declined; traces help us investigate the execution that produced it.

Applying This on AWS

On AWS, this idea maps naturally to the observability and evaluation capabilities around Amazon Bedrock AgentCore and Amazon CloudWatch.
Agent traces can provide visibility into sessions, spans, model interactions, and tool execution. Operational telemetry can expose signals such as latency, errors, duration, and token usage.
Quality evaluation adds another dimension.
Instead of monitoring only whether an invocation failed, teams can evaluate agent interactions against criteria such as response quality, task completion, and tool use.
Evaluation results can then become metrics that are monitored over time alongside traditional operational signals.
This creates an important shift:
Monitoring infrastructure health
becomes
Monitoring infrastructure health + agent behavior + outcome quality
For production AI systems, I believe all three are necessary.

Build a Feedback Loop, Not Just a Dashboard

A dashboard is useful only if the information leads to improvement.
Production evaluation should therefore become a continuous feedback loop.
A practical process might look like this:
1. Observe
Collect traces and operational metrics from real agent interactions.
2. Evaluate
Score representative interactions against criteria such as correctness, groundedness, task completion, and tool-use quality.
3. Diagnose
Inspect failed sessions and traces to determine why the quality metric declined.
4. Improve
Modify the prompt, retrieval strategy, model, tool definitions, orchestration logic, or guardrails.
5. Re-evaluate
Run the same evaluation criteria again and compare the results.
The final step is particularly important.
Without consistent evaluation, an improvement to one behavior can quietly introduce a regression somewhere else.

A Practical Production Scorecard

For an agent moving toward production, I would monitor a scorecard across three dimensions.

Reliability

  • availability
  • latency
  • invocation errors
  • tool failures
  • retries

Quality

  • task completion rate
  • groundedness
  • correctness
  • tool-selection accuracy
  • escalation rate

Efficiency

  • token consumption
  • cost per request
  • cost per successful task
  • latency per successful task
The goal is not necessarily to maximize every metric.
The goal is to define acceptable quality thresholds and then optimize the system without crossing them.

Final Thought

The most dangerous agent failures may not look like failures at all.
There may be no exception.
No timeout.
No failed API call.
No red indicator on the infrastructure dashboard.
The agent simply returns the wrong answer successfully.
That is why production readiness for agentic AI requires more than monitoring whether the system is running.
We also need to measure whether it is working.
As agents take on longer workflows and interact with more tools and enterprise data, I expect the distinction between execution monitoring and outcome evaluation to become increasingly important.
A 200 OK tells us that a request completed.
For an AI agent, that should be where evaluation begins—not where it ends.
# amazon-bedrock# generative-ai# ai-agents# amazon-cloudwatch# observability
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article