AWS Builder Center

Builder Center

CloudWatch Omni: For the First Time, AWS Gives the Observability Leaders a Real Reason to Worry

Golden Jacket Field Notes · Week 21

Series: Golden Jacket Field Notes (22 articles)

  1. 1
    The Cheapest Price Per Token Can Be the Most Expensive Way to Run an LLM
  2. …
  3. 22
    CloudWatch Omni: For the First Time, AWS Gives the Observability Leaders a Real Reason to Worry This article

Erick Mancz

AWS Golden Jacket | Co-founder Golden Jackets Brazil Community | 40 AWS Credentials
The first thing I opened when CloudWatch Omni went GA on September 22 was the Region list. I always do that first, because most of what I build lands in São Paulo, and a launch that doesn’t reach sa-east-1 is a launch my clients will read about for a while before they can touch it. Three Regions at launch: US East (N. Virginia), US West (Oregon), Europe (Ireland). Then I kept reading anyway, because the rest of the announcement was about the one problem I keep running into in agent reviews.
So let me be upfront about where this piece comes from. I haven’t run Omni in production. Nobody has run it in production for long; it’s been out for five days. Everything below comes from reading the launch posts and a few dozen documentation pages closely, cross checking them against each other, and comparing what they describe with the agent workloads I review for a living. Where I’m inferring, I’ll say so.
What caught me is a line in AWS’s own tutorial, describing an agent that answered the wrong question promptly and successfully: “Performance monitoring cannot see that.” I’ve been trying to get that sentence into steering committees for a year.

The problem

Here’s a pattern I’ve watched repeat across agent projects, and I’ll describe it as a pattern, because the same incident keeps arriving in different clothes.
A team ships a customer facing agent. Support, internal knowledge, account servicing, pick your flavor. They do everything the operations playbook says. Tracing is on. There’s a dashboard with latency percentiles, error rate, invocation count, token consumption per session. Alarms fire on 5xx spikes and on p99 latency. The on call rotation knows where the runbook lives. By every measure the platform team was trained to care about, this is a well run service.
Then someone outside engineering finds the problem. A customer posts a screenshot. A compliance reviewer pulls a sample of transcripts for an unrelated audit. A product manager tries the agent on a Saturday. The agent has been answering questions it was never supposed to answer. A banking agent explaining medication interactions. A telecom support bot writing a cover letter. An internal HR agent confidently quoting a policy that was retired two quarters ago. Every one of those responses returned a 200. Every one had normal latency. Several of them were faster than average, because the agent skipped the retrieval step entirely and answered from the model’s general knowledge.
The first reaction in the room is almost always the wrong one. More monitoring. Somebody proposes an alarm on response length, another on token spikes, another on the ratio of tool calls to model calls. All of them are proxies, and every proxy fails the same way: a wrong answer that looks like a right answer produces identical telemetry. You can’t alarm your way out of a semantic failure with operational signals, because the signal you need lives in the content, and in a judgment about the content, where operational telemetry never looks.
The second reaction is better and more expensive. The team builds an evaluation harness. A notebook grows into a script, the script grows into a pipeline, somebody wires a model as a judge, somebody else builds a small labeling interface, and three months later there’s a homegrown quality system that one engineer understands and nobody else wants to own. In the projects where I’ve watched this happen, it quietly consumed the better part of a quarter. The harness usually lives outside the observability stack, so when a quality regression shows up nobody can correlate it with the deploy, the retrieval latency, or the knowledge base sync that ran the night before. The quality people and the operations people are looking at two different tools that don’t share a trace ID.
That’s the gap Omni is aimed at. The cost of leaving it open is concrete: a failed compliance review, a public screenshot, or a quarter of engineering time spent rebuilding something that should have been a platform capability.

Why this matters beyond the specific case

AWS observability has moved through this same shape before, and each time the move was toward a higher unit of meaning.
CloudWatch started as host metrics. CPU, disk, network. The unit was the machine. X-Ray moved the unit to the request, following a call across services. Application Signals moved it again, to the service and its SLOs (Service Level Objectives), with RED metrics (request rate, errors, duration) discovered automatically instead of hand built. Each step kept the previous layer and put a new question on top. Is the box alive. Is the request fast. Is the service meeting its promise.
Omni adds the next question: is the answer right. The unit becomes the judgment an agent made, and the signal becomes an evaluation score sitting in the same data store as the span that produced it.
There’s a second familiar shape here, and it’s the one I’d pay more attention to. Security Hub sat on top of GuardDuty and Inspector, consolidated their findings, and made correlation cheap, while the detection and the cost stayed in the underlying services. Omni follows the same pattern. The documentation is explicit that Omni is an interface and workflow layer built on CloudWatch, reading the same data stores the traditional console reads, with ingestion, retention and configuration still managed in CloudWatch. So the value lives in correlation, and the bill lives where it always lived.
The stakes are higher this time for one reason. Every previous layer failed loudly. A dead host stops sending metrics. A slow request shows up in p99. A broken service throws errors. Agent failures are quiet by construction. The model is designed to produce fluent, well formed output, which means its failures are fluent and well formed too. For teams shipping into regulated environments, a quiet failure is the worst kind, because the first person to detect it is often the auditor.

Technical deep dive

What Omni actually is

Strip the launch language away and Omni is three things. A data model that organizes CloudWatch telemetry around teams and applications. Two new surfaces to work with that data, a web experience outside the AWS console and an IDE (Integrated Development Environment) extension. And an evaluation workflow for AI agents that plugs into Amazon Bedrock AgentCore Evaluations.
The data model has three levels. A domain is your organization’s entry point, with its own sign in URL in the format <domain-name>.cloudwatch-omni.global.app.aws, and the identity provider connects at the domain level through IAM (Identity and Access Management) Identity Center. A space is where work happens, and a space maps to exactly one AWS account and Region combination. Only one space can exist per account and Region. A CloudWatch Dataset is created automatically for each space and makes logs and traces queryable together.
The space boundary is a hard separation. The documentation says it plainly: each space sees the telemetry centralized into its own account and Region, and cross space queries aren’t supported. If you need to query across accounts, you centralize telemetry into one account instead of creating more spaces.
Engineers sign in with their organizational identity, so the people using Omni don’t need AWS console access at all. For organizations, the domain is created once from the AWS Organizations management account, and member accounts create their own spaces, which link to the domain automatically. One constraint to catch early: your IAM Identity Center instance and your domain must live in the same Region, or you can’t configure sign in.

The data path

Omni reads what CloudWatch already has. Existing metrics, alarms, dashboards and Logs Insights queries keep working, and existing CloudWatch agent instrumentation carries forward without changes.
Traces are the primary signal, and traces depend on Transaction Search. When Transaction Search is enabled, spans land as structured logs in a log group called aws/spans, in OpenTelemetry semantic convention format with W3C trace IDs. It's an account level, per Region setting, and the documentation is unusually blunt about ordering: until it's enabled, spans don't arrive, and traces sent before span ingestion is enabled are never stored.
Ingestion runs through OTLP (OpenTelemetry Protocol) endpoints, one per signal. The traces endpoint accepts SigV4 only. The metrics and logs endpoints also accept bearer tokens for environments where short lived credentials aren’t available. For on premises collectors, the documented path is IAM Roles Anywhere, so the collector still signs with temporary credentials.
There are four collection paths. The CloudWatch agent for applications on EC2, ECS, EKS, and Azure VMs and AKS. ADOT (AWS Distro for OpenTelemetry) exporting directly from the SDK, with no collector, which is the path for Lambda and for agents. A self managed OpenTelemetry Collector pointed at the OTLP endpoints with SigV4 configured on each exporter. And AgentCore Runtime, which exports traces natively.
Getting logs into the Dataset takes one more step that’s easy to miss. You create an execution role trusted by logs.amazonaws.com, grant it logs:IntegrateWithDataset on the log groups and cloudwatch:PutRecords, and create the integration with aws observabilityadmin create-dataset-integration. There's one integration per account and Region, and forwarding is same account, same Region. Metrics don't get forwarded to the Dataset at all. They're queried directly from CloudWatch Metrics, including through PromQL (Prometheus Query Language).

The agent loop

This is the part of Omni that has no real precedent in CloudWatch, and it’s the part worth your time.
The documentation frames agent monitoring as a closed loop. Instrument, examine, evaluate, improve, monitor, iterate. In practice it goes like this. Your agent emits OpenTelemetry traces where each run is a trace, each model call, tool call and retrieval is a span, and related traces group into a session. Omni renders those as agent traces, sessions and an agent topology. You select runs, score them with evaluators, package the interesting ones into a dataset, change something, run the change against the dataset, and deploy. Then online evaluation scores a sample of live traffic continuously, and a regression starts the loop again.
Supported frameworks at launch include LangGraph and LangChain, Strands Agents, CrewAI, LlamaIndex, the OpenAI Agents SDK and the Vercel AI SDK, with Python and Node.js as the documented languages. The supported environments page lists minimum versions for each and a few rough edges worth reading before you start. CrewAI needs its own telemetry disabled. The OpenAI Agents Node.js path needs a manual instrumentation call. Strands has built in tracing and OpenInference both available, and the documentation says to use one path only.
Evaluators come in three kinds. Built in evaluators, which the News Blog launch post counts at seventeen, covering things like coherence, helpfulness, faithfulness and routing correctness. Third party evaluators, with the launch materials naming DeepEval, AutoEval, Ragas and Braintrust in different places. And custom model as judge evaluators, where you write a rubric, pick a judge model, and define a rating scale. Evaluations run through Amazon Bedrock AgentCore Evaluations.
The official tutorial is the best piece of documentation in the launch, and I’d read it even if you never deploy Omni. It walks through a customer support agent that answers a cookie recipe request and a question about back pain. Both requests complete successfully. Both look healthy on the performance dashboards. And a built in evaluator rates the failing response as acceptable, which makes sense once you think about it: a clear, friendly cookie recipe is, in fact, helpful. The fix is a custom evaluator with a binary rubric: on topic scores one, off topic scores zero, and a response that acknowledges the request is off topic but answers anyway still scores zero. Then a prompt change is tested against a dataset built from the failing production traces, and online evaluation confirms the success rate recovers on live traffic.

Two surfaces, split on purpose

The IDE extension runs in VS Code and Kiro, with the What’s New post and the product page adding Cursor. It works in local mode without an AWS account, storing traces and configuration in a .omni folder, and switches to cloud mode when you sign in to a space. The web experience is where operators and fleet views live.
The split between the two isn’t symmetric, and it matters when you plan who does what.
CapabilityWeb experienceIDE extensionAgent traces, sessions, topologyYesYesDatasets and evaluationsYesYesPrompt playgroundYesYesOnline evaluationsYesNoExperimentsNoYesPrompt management and versioningNoYesAggregate fleet viewsYesNoLocal testingNoYes
Read the table as an org chart. Production quality monitoring belongs to whoever lives in the web experience. Controlled experiments and prompt versioning belong to whoever lives in the IDE. The handoff between them is a dataset. Datasets downloaded from the space into the IDE become independent local copies with no sync back, so a dataset is a snapshot you hand across, and the team has to agree on which copy is the source of truth.

Investigation

Omni has two AI layers that are easy to confuse. The Omni agent answers questions about telemetry in plain language and turns them into SQL or PromQL, with the evidence visible. AWS DevOps Agent, which I dissected in Week 5, handles root cause investigation. You can ask for an investigation in a thread, use an /investigation new command, or choose Investigate on a firing alert. Alerts don't trigger investigations automatically.
Threads are the unit of collaboration. Each one keeps the conversation, the queries, and every canvas you visited, auto saved, so an investigation can move between people and devices. The durable way to share is to save a canvas as a dashboard, which every space member can see.

Multi account and multi Region

Omni doesn’t aggregate across accounts or Regions on its own. You use CloudWatch centralization rules, created from the management account or a delegated administrator, which name source accounts and source Regions and a destination account and Region. Point the destination at the account and Region that hosts your space.
Three details decide whether this works on day one. Centralization doesn’t backfill, so only telemetry generated after the rule is active gets replicated. Trace centralization requires Transaction Search to be enabled in every source account. And the application map covers services from all source accounts only when Context Graph is enabled.

The billing model

I’ll describe the model and leave the unit prices to the pricing page, because figures age badly. As published at launch in September 2026, Omni bills across three dimensions. Ingestion is charged per GB at standard CloudWatch rates. Storage is charged per GB month. A third dimension covers usage of Omni features. Each month, scans up to five times your eligible log and span ingestion are free, and scanning beyond that is billed per GB. PromQL metric queries are billed per million samples. Omni dashboards, alerts, the application topology map and the investigation agent are included within service quotas. Agent evaluations are billed at AgentCore Evaluations rates. Centralized telemetry is billed at standard CloudWatch rates in the destination account. The launch trial is thirty days of credits covering OpenTelemetry ingestion only, for up to ten accounts per AWS Organization.

What the official documentation does not tell you

The variable that turns content capture on is called OPT_OUT. To make prompts and responses visible in spans, which evaluators need in order to score anything, you set AWS_GENAI_CONTENT_EXTRACTION_OPT_OUT=true. Read that name out loud in a change review and watch the room. "Opt out equals true" sounds like the privacy safe setting. It does the opposite: it opts out of redaction and keeps prompt and response text in your traces. The documentation explains it correctly, but the name will get somebody in trouble, and I'd put a comment next to it in every deployment file.
The same agent has opposite privacy defaults in Python and Node.js. In Python, model payloads and tool responses are redacted by default, and you need the variable above to keep them. The Node.js path captures GenAI message content by default, and the variable isn’t required. A team with a Python agent and a TypeScript agent calling the same model with the same users will have one agent storing prompts and one agent not storing them, unless someone reads a single sentence on the telemetry page. In a regulated shop, that’s the kind of discrepancy an auditor finds before you do.
Your CloudWatch Logs masking policies do not protect the Dataset. The data protection page states that masking policies configured on log groups don’t carry over to a space’s CloudWatch Dataset, and that Omni doesn’t detect or redact PII (Personally Identifiable Information) automatically. Many teams already have data protection policies on their log groups and would reasonably believe they’re covered. They aren’t, for this path. The documented options are redaction at capture time, through AWS_REDACT_SPAN_ATTRIBUTES, the OpenInference hide flags, or a custom span processor, or filtering in a collector. Capture time redaction is irreversible, which means the content you hide from the auditor is also hidden from the engineer debugging the incident. No configuration gives you both. That decision belongs in a design review, and an environment variable someone sets on a Friday is the wrong place to make it.
Evaluation is a data flow, and the sampling percentage is a data exposure dial. Model as judge evaluators send the evaluator prompt and the trace content to whatever judge model you configured. The documentation says to choose the judge model accordingly, or restrict evaluation to non sensitive traces. It also says the judge is invoked for every sampled trace, which makes the sampling percentage the cost lever. Put those two sentences together and the same slider controls your bill and how much customer content leaves the trace store for a second model. In a Week 7 context, where the governance layer is supposed to be deterministic, you’ll want the judge model choice written down and approved like any other data processor.
Three Regions, and centralization is a cross Region replication engine. This one matters most to me, and I’ll mark clearly where the documentation ends and my inference begins. Documented: Omni is available in N. Virginia, Oregon and Ireland. Documented: centralization rules replicate telemetry from source Regions into a destination Region, and the destination should host your space. My inference: for a workload running in São Paulo today, using Omni means replicating its telemetry, possibly including prompt and response content if capture is on, into a Region outside Brazil. For many organizations that’s fine. For a bank, a telecom, or a health provider operating under LGPD (Lei Geral de Proteção de Dados, Brazil’s General Data Protection Law) and sector regulation, that’s a conversation with the DPO (Data Protection Officer) before it’s an architecture decision. I’m not a lawyer and I won’t pretend to know how each regulator will read it. I do know that “we turned on the new observability console” is a bad way for a data protection team to find out telemetry changed jurisdiction.
The healthy dashboard lie is in AWS’s own tutorial, and so is the evaluator that misses it. Step six of the tutorial says it directly: volume, errors and latency all look healthy, because the agent answered promptly and successfully, and it answered the wrong question. Step seven has you run a built in evaluator on the failing trace, and the expected checkpoint is a score “showing that existing checks rate the failing run as acceptable.” That’s AWS telling you, in its own onboarding, that general purpose evaluators won’t catch the failure your business cares about. The practical rule from the tutorial is worth stealing: if the failure completes without an error, general purpose evaluators miss it, and the rule fits in one sentence, you need a custom evaluator with a binary scale, and you need it before the incident. Keep the rubric in version control next to the prompt.
Order of operations will cost you data you can’t get back. Traces sent before span ingestion is enabled are never stored. Centralization doesn’t backfill. Trace centralization needs Transaction Search in every source account. Stack those three and the failure mode is quiet: a team enables Omni, deploys agents, turns on centralization a week later, and the first week of traces either never existed or never left the source account. Nothing errors. The data just isn’t there when you go looking for the week the regression started. Enable Transaction Search in every account first, create centralization rules second, deploy agents third.
Changing an online evaluation opens a gap. While an online evaluation is enabled, its evaluator list is locked, and you have to disable it to change evaluators. It’s a small line in the tutorial. In production it means every evaluator change creates a window with no continuous scoring on that agent. If you’re about to add a new rubric right after an incident, which is exactly when you’ll want to, plan for that gap or run the new evaluator as a separate online evaluation first.
The space boundary will force an account topology decision you may not have made yet. One space per account and Region, hard boundaries between spaces, no cross space queries. If your agents are spread across workload accounts the way a Landing Zone is supposed to spread them, you’ll end up centralizing into a dedicated observability account, much like the Log Archive account pattern in Control Tower. That’s my recommendation, and AWS doesn’t prescribe it. It’s worth deciding before the first team creates a space in their own workload account and gets comfortable there.
I couldn’t find whether queries the Omni agent runs for you count against the free scan allowance. The free allowance is five times eligible ingestion. The Omni agent turns natural language into SQL and PromQL and runs it. Nothing I read says those generated queries are exempt, so my working hypothesis is that they count like any other scan. That makes the allowance a function of how much you ingest, while scan volume becomes a function of how curious your engineers are, and a conversational interface makes people a lot more curious. Watch scan volume in the first month before you roll it out to the whole engineering org.
The launch materials disagree with each other, so trust the documentation pages. The News Blog post lists the IDE extension for VS Code and Kiro, while the What’s New post and the product page add Cursor. One launch post says DevOps Agent is enabled by default in investigation sessions. The documentation says you need an existing DevOps Agent space in the same account, the AIDevOpsAgentFullAccess policy attached to CloudWatchOmniOperatorRole, and a sign in method on the agent space that matches the Omni domain. Evaluator partners appear in different combinations across pages. None of this is unusual for a launch week. It's a reason to build from the docs pages and treat blog posts as orientation.

The bigger picture

For fifteen years, observability answered one question in increasingly sophisticated ways: is the system working. Omni is AWS putting a second question into the same product: is the system right. I think that second question is going to reorganize who does what in platform teams, and faster than most org charts will adapt.
Here’s the position I’ll take. Evaluation engineering is becoming an operations discipline. For the last two years it lived with data scientists, in notebooks and offline benchmarks, run before release and mostly forgotten after. Omni puts evaluators next to alerts, online evaluation next to dashboards, and quality regressions in the same thread as latency spikes. Once that’s where it lives, the person who writes the one sentence rubric for “our agent must never answer outside its scope” is doing on call work. That person needs to understand the business rule, the model’s failure modes, and the telemetry path. SRE (Site Reliability Engineering) teams that learn to write and maintain evaluators will own agent reliability. Teams that treat evaluation as someone else’s job will keep watching green dashboards while customers post screenshots.
The second signal is quieter and I think more strategic. AWS built Omni with a sign in URL outside the AWS console, enterprise SSO (Single Sign-On), an IDE extension that works with no AWS account, Azure support at launch, and OpenTelemetry as the ingestion standard. That’s AWS conceding that the people operating agents aren’t primarily console users, and that telemetry ownership has to feel portable before a serious buyer will commit to it. The OpenTelemetry bet is a real escape hatch for customers, and it’s also AWS competing on the ground where independent observability and LLM tooling vendors have been winning. I’d expect a lot of vendor comparison decks in the next quarter. The one question I’d put in every one of them is where your trace content goes when a judge model scores it.
For the Brazilian market specifically, the practical reality is waiting. Regulated organizations running in sa-east-1 will read about Omni, prototype it in a sandbox account in N. Virginia with synthetic traffic, and wait for local availability or for a data protection position they’re comfortable with. That’s a reasonable choice. I’d use the waiting time well: write the custom evaluators now, build datasets from real failures now, and get the rubrics reviewed by compliance now. None of that depends on where Omni runs, and all of it is the hard part.

Takeaway

A wrong answer with a 200 status and good latency is invisible to every dashboard you already have. Omni’s real contribution is putting evaluation scores in the same place as spans, so quality and operations finally share a trace ID.
Before you turn it on, settle three things: where your traces will live, whether prompt content will be captured and redacted, and who owns the one sentence rubric. Enable Transaction Search first, because traces sent before it are gone for good.
The healthy dashboard answers whether it’s working. The evaluator answers whether it’s right. You need both, and only one of them was ever on call.
I keep going back and forth on one question. When an online evaluator flags a quality regression at two in the morning, who gets paged? The SRE who owns the service, the product team who owns the prompt, or the compliance function who approved the rubric? I don’t have a clean answer, and I suspect most teams haven’t been forced to decide yet. If you’re running agents with continuous evaluation in production, I’d like to hear how your team settled it, or whether you’ve settled it at all.

References

About the author

I am an AWS Solutions Architect with almost 15 years in cloud, focused on production ML and GenAI workloads for regulated organizations. I hold every active AWS certification, including the Generative AI Developer Professional as an Early Adopter, the status AWS calls Golden Jacket. I am the sixth Brazilian Golden Jacket holder according to the Awstronauts list, and co-founder of Golden Jackets Brazil.
Golden Jacket Field Notes is a weekly practitioner series. No re:Invent recaps. No tutorials. Field notes from workloads where outages, audit gaps, and cost surprises are not tolerated. The full index lives at golden-jacket-field-notes .
Opinions expressed in articles and code are my own and do not represent the views of my employer or any affiliated organization.
#AWS #CloudWatch #Observability #OpenTelemetry #GenerativeAI #AIAgents #AgentCore #SRE #LGPD #GoldenJacket
# aws-cloudwatch# amazon-cloudwatch# agentic-ai# agents# observability

Series: Golden Jacket Field Notes (22 articles)

  1. 1
    The Cheapest Price Per Token Can Be the Most Expensive Way to Run an LLM
  2. …
  3. 22
    CloudWatch Omni: For the First Time, AWS Gives the Observability Leaders a Real Reason to Worry This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article