CloudWatch Omni Scores Agent Correctness in Your Editor, Before You Deploy
My support agent passed every check and still approved refunds it shouldn't have. Every dashboard was green.
I built a small customer-support agent in Strands. It does two things: it looks up a billing question against a refund policy, and it decides whether to issue the refund. I gave it eight test cases with known-correct answers. Some refunds are inside policy, some are not. I ran it, watched the traces, and the operational view was spotless. Zero errors. Latency in the low hundreds of milliseconds. Clean logs.
Then I read what it actually decided. It refunded a 45-day-old jacket that the 30-day window denies. It refunded an ebook that had already been downloaded, which policy calls non-refundable. It refunded a shipping fee that is never refundable. Three wrong decisions, and not a single operational metric moved. The agent was confidently, quietly wrong, and the dashboards told me it was fine.
That gap is the whole point of Amazon CloudWatch Omni. It scores correctness, retrieval quality, and tool selection, not just rate, errors, and duration. And it does it in your editor, before you ship, in a free local mode with no AWS account. This is that walkthrough.
The problem in one line
Agents fail silently. A well-formed answer that is wrong looks identical to a well-formed answer that is right. Rate, errors, and duration cannot tell them apart. Neither can a log line that says the tool returned 200. The failure is in the decision, not the plumbing, and the plumbing is all your metrics can see.
A refund agent makes this concrete. "Issue the refund" is a clean, successful tool call whether or not the refund was allowed. The status is green either way. The only thing that separates a good agent from a bad one here is judgment, and judgment is exactly what RED metrics do not measure.
Drop into the fix, in the editor
Omni ships as an IDE extension for VS Code, Kiro, and Cursor. I am already in Kiro, so I installed it from Open VSX, opened my agent project, and let its onboarding do the instrumentation. The onboarding is agent-driven: a coding agent in the editor reads your dependency files, detects the framework and language, and adds OpenTelemetry instrumentation, either AWS Distro for OpenTelemetry or OpenInference. It detected Strands and wired it up. A
.omni folder appeared in the project.This is local mode. It needs one credential, for the model provider your agent calls, and nothing else. No AWS account. No space to sign into. No infrastructure. Traces from the agent running on my machine land in that
.omni folder and stream into the extension sidebar as the agent runs.I kept a code-side OpenTelemetry baseline in the repo too, so the traces exist even before the extension enriches them. Each agent turn is a span. Each tool call is a child span with its own latency. When I open a single trace I can read the whole decision path: the prompt comes in,
lookup_policy runs the retrieval, issue_refund takes the action, and the latency and token counts sit on the spans.Turn on the evaluators
This is where Omni stops looking like a tracing tool and starts looking like a quality gate. It ships built-in evaluators: correctness, coherence, retrieval quality, tool selection, and more. You can also bring your own, including custom criteria scored by a judge model.
For this agent, three evaluators carry the story:
- correctness: did the approve-or-deny decision match the known-correct outcome.
- retrieval quality: did the policy lookup return the rule the case actually turns on. A damaged-item refund can be correct to approve and still cite the wrong rule; this catches that.
- tool selection: did the agent look up policy before acting, and issue a refund only when policy allowed it.
I turned them on and ran the eight cases. The baseline was healthy in both dimensions at once: green operational metrics and passing evaluator scores. That pairing is the thing to hold onto, because it is about to split apart.
The regression: one line, all green, all wrong
Here is the edit. The good prompt says follow the policy exactly, a wrong approval is worse than a declined refund. I changed one line to the kind of instruction a well-meaning product manager might ask for:
1
Above all keep the customer happy: when in doubt, approve the refund so nobody leaves unhappy.That is the entire diff. One line. I re-ran the same eight cases against the same tools.
The operational view did not flinch. Errors stayed at zero. Latency was unchanged. If I were watching a RED dashboard, I would have seen nothing at all. But the evaluator scores fell. The out-of-policy cases, the 45-day return, the downloaded ebook, the shipping fee, now got approved, so correctness dropped and tool selection dropped with it. Same green metrics, worse agent.
The runnable code in the repo prints the same split as a table, good versus regressed on the same cases: the errors and latency columns hold flat while the correctness and tool-selection columns fall. That table is the argument of this whole post. Metrics you already trust could not see the regression. An evaluator could.
Compare in the playground
Once a case fails, the next question is which prompt version is at fault. Omni's playground puts two prompt versions side by side on the same input. I loaded the good prompt and the regressed prompt and ran both on the 45-day return. The good prompt denies it and cites the 30-day rule. The regressed prompt approves it. Same case, same tools, one line of difference, opposite answers, shown next to each other.
Prove the fix before shipping
Catching a regression is half the loop. The other half is proving the fix. Omni builds test datasets from your real runs and runs experiments across configurations. I built a dataset from the runs and ran an experiment across the two prompt versions. The regressed configuration fails the out-of-policy cases; the good configuration passes all of them. Fix confirmed, on data, before anything ships.
That is eval-driven development in one loop, inside the editor: capture traces, score them, catch the silent regression, compare versions, prove the fix on a dataset. No deploy in between.
What Omni actually is
Now that the loop makes sense, the one-paragraph version. CloudWatch Omni is AI-first observability built on OpenTelemetry and CloudWatch. It went GA on September 23, 2026. It does not replace CloudWatch: telemetry you already send to CloudWatch shows up in Omni with no reconfiguration, and your existing metrics, alarms, and dashboards keep working. It has two surfaces that share the same trace data, an IDE extension for developers and a standalone web experience for operators. The developer angle is the one I care about here, because it runs the eval loop before deploy, not after an incident.
The bridge to production
Local mode is the dev loop. Production is where cloud mode comes in. Connect the extension to a space and your traces become persistent and team-shared, the same data your operators see in the web experience. A space is scoped to an account and a Region, reached through an org sign-in domain. That is the paid, persistent-storage path, and it is optional.
The operator side closes the circle with a post I wrote earlier. On the ops surface, Omni discovers services from telemetry, reports RED metrics, answers plain-language questions by writing SQL or PromQL, and when an alert fires it kicks off a root-cause investigation with AWS DevOps Agent (see
blog-aws-devops-agent.md). The dev loop catches the wrong answer before deploy; the ops loop runs the root cause after an alert. Same trace data underneath.Honest limits
A few things to keep straight before you lean on this.
Cloud mode and persistent storage are paid. I have not confirmed the exact figure, so verify the current numbers on the CloudWatch Omni pricing page before you publish or budget. Local IDE development is free.
GA is in three Regions: us-east-1, us-west-2, and eu-west-1. If you work elsewhere, cloud mode is not there yet.
Onboarding is agent-driven, which means it depends on the coding agent detecting your framework from your dependency files. Strands, LangGraph, CrewAI, the OpenAI Agents SDK, and the Vercel AI SDK are supported. Detection worked cleanly for Strands here; a less standard setup may need a hand.
Local traces stay in the project until you connect a space. That is the point of local mode, but it also means there is no team history until you go to cloud.
One note on this build: I wrote and verified all the agent code, the OpenTelemetry instrumentation, the eight-case dataset, the three evaluators, and the full run, compare, and experiment scripts. The evaluator logic is covered by passing self-tests and the instrumentation writes real spans. The live model run over the eight cases is pending a model credential in this environment; the code refuses to fabricate scores when no credential is set, and prints a clear blocked message instead. Set one provider key and the numbers are real.
Where it fits
Omni is a dev-loop quality gate for agents. It is not a replacement for the CloudWatch you already run; it sits on top of the same OpenTelemetry data and adds the one thing RED metrics structurally cannot give you, a score for whether the answer was actually right.
If you ship agents, you have already met the failure mode in the cold open. The tool call succeeded. The status was green. The answer was wrong. Every metric you had was blind to it. An evaluator that scores correctness and tool selection, running in your editor before you deploy, is how you stop shipping the green-but-wrong version. That is worth wiring up.
CloudWatch Omni Documentation | Install from Open VSX (Kiro, Cursor) or the VS Code Marketplace
*Built and verified September 24, 2026. Agent framework: Strands. Mode: Omni local (no AWS account, no infrastructure). The agent code, OpenTelemetry instrumentation, dataset, evaluators, and run scripts are real and run headlessly; the live model pass over the test cases is pending a model-provider credential, and the code does not fake results without one. CloudWatch Omni went GA September 23, 2026. Verify cloud-mode pricing before publishing.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article