AWS Builder Center
I Gave the Same Incident to CloudWatch Omni and to My Grafana Stack

I Gave the Same Incident to CloudWatch Omni and to My Grafana Stack

I broke my own online shop on purpose and gave the incident to CloudWatch Omni’s AI assistant, to Omni with Amazon DevOps Agent, and to my Grafana dashboards. Here’s what each one found, and how to build the lab yourself.

Cloud Architect
At 11:10 almost a quarter of the checkouts in my shop started failing.
I had just set up Amazon CloudWatch Omni, the new observability service from AWS. One of its main selling points is an AI assistant built into the console: you describe the problem and it investigates for you. So I typed one sentence and decided not to help it after that:
"Storefront errors and latency increased since about 11:10. What is the root cause?"
The answer took almost four minutes. It blamed the wrong service, and one line in it said:
"No exception text was recorded in traces or logs for this window."
That wasn't true. CloudWatch Logs was receiving about 70 lines a minute that said  inventory timeout after 800 ms tenant=tenant-b . The cause was written in the logs in plain English, and the assistant hadn't found it.
I knew the line was there because I had written the bug myself. That's the one advantage of breaking your own shop: you're the only person in the room who already knows the answer. This article explains why I did it, and what happened when I gave the same incident to three different investigators.

What I wanted to find out

My team runs its own observability stack: Grafana for dashboards, Tempo for traces, Loki for logs and Prometheus for metrics. We install it, upgrade it and fix it when it runs out of memory. This year we wanted to modernize that setup, and Omni was the obvious candidate: the same kind of tooling as a managed service, plus an AI that investigates incidents for you.
So the question was practical: if we moved to Omni, would we find the cause of an outage at least as quickly and as accurately as we do today, and would that be worth what Omni costs? The biggest difference between the two setups turned out to be the AI, so most of this article is about the investigations. At the end I'll tell you what we decided.
If you run your own Grafana stack and are asking yourself the same thing, this article is for you. I assume you know Kubernetes and have looked at traces, logs and metrics before. Everything specific to Omni and DevOps Agent, I explain when it comes up.
A feature list can't answer that question. You need an incident, and to judge the answers you need to know the correct one in advance. Real incidents don't give you that, so I made my own:
  • I built a small online shop and wrote two bugs into it, each in its own release.
  • The shop sent exactly the same telemetry to Omni and to my Grafana stack.
  • I deployed a broken release and gave the incident to three investigators.

The three investigators were:
  1. Omni's built-in assistant, the chat window in the Omni console.
  1. Omni with Amazon DevOps Agent. DevOps Agent is a separate AWS service that runs a whole investigation on its own: it reads metrics, logs and traces, tests hypotheses and writes a report. You connect it to Omni, and the assistant hands the work over to it.
  1. Me, with my Grafana dashboards. I knew the answer, so my time is only a reference point, not a fair result.
Each investigation got one point for each of five things I'd want in a good incident report:
Point
Question
Service
Which service is broken?
Release
Which deployment caused it?
Mechanism
What exactly goes wrong in the code?
Customer
Which customer is affected?
Noise
Did it ignore a warning that is always in the logs and has nothing to do with the incident?
I also timed each one, from the first question until the answer contained everything it was ever going to find. For the assistant, that includes the time I spent clicking its suggestions.
Is that a fair test? Partly, and you should know where it isn't. Each investigator had one run. There were two bugs in one small app, and since I wrote them, they are simpler than most real incidents. The assistant only saw the inventory bug. DevOps Agent saw the same bug on its second deployment, so it had more data, which is why I also gave it a bug chosen at random. Treat the scores as a field report from one lab, not as a benchmark.
All the code is on GitHub at  github.com/romanceresnak/omni-vs-grafana-lab . If you only want the outcome, skip to "Results". If you want to run the lab yourself, "Build it yourself" at the end takes you from an empty AWS account to a working lab, and back to an empty account.

The experiment setup

The shop

All times in this article are local time (UTC+1), unless a screenshot says UTC.
The shop is four small Python services, and every checkout goes through all of them:
  • storefront receives the customer's request and passes it to orders.
  • orders creates the order. It asks inventory for prices and stock, and asks payments to charge the card.
  • inventory keeps price rules in a Redis cache.
  • payments charges the card.
A load generator sends five checkouts a second. Every checkout belongs to one of three customers. The shop is multi-tenant, like many SaaS products, so I call them tenants: tenant-a sends half of the traffic, tenant-b and tenant-c a quarter each. Every request carries the tenant's name, and so does everything the services record about it, in an attribute called  tenant.id .

The two bugs

  • inventory 2.0.0 adds a new cache for price rules. It writes under one key and reads under another, so it never finds anything and loads every rule from scratch. tenant-b has 40 price rules, so its requests take about a second. orders waits at most 800 ms for inventory, then gives up, and the checkout fails.
  • payments 1.4.0 adds a fraud check that runs synchronously, only for tenant-c, and takes up to 2.6 seconds. orders waits at most two seconds for payments.
I also added a distraction. payments logs  card network latency high, retrying  every few seconds, before, during and after both incidents. Most production systems have a warning like that, and everybody has learned to ignore it. A good investigator should ignore it too.

Sending the same data to both sides

The comparison is only fair if both sides see exactly the same data. So the services don’t know about Grafana or CloudWatch at all. They are instrumented with OpenTelemetry, the open standard for collecting telemetry, and send everything to one OpenTelemetry Collector: their logs, their metrics and their traces. A trace follows one request through all the services, and it is made of spans, one for each step the request takes in one service.
The collector then sends every signal twice: to my stack, at the top right of the diagram, and to the CloudWatch endpoints that Omni reads from, at the bottom right.

Architecture: a load generator and four services (storefront, orders, inventory with Redis, payments

The lab on Amazon EKS: the shop, one OpenTelemetry Collector, and the two places the data goes.

For traces, the important part of the collector configuration looks like this:
exporters:
otlp_grpc/tempo: # the stack I run today
endpoint: tempo.observability.svc.cluster.local:4317
otlp_http/cw-traces: # CloudWatch, signed with SigV4
traces_endpoint: https://xray.eu-west-1.amazonaws.com/v1/traces
auth: { authenticator: sigv4auth/traces }

service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp_grpc/tempo, otlp_http/cw-traces]
Logs and metrics are configured the same way, with Loki and Prometheus on one side and the CloudWatch endpoints for logs and metrics on the other.

Do both sides really get the same data?

Before comparing anything, I wanted proof. The collector counts what each exporter sends, and both Prometheus and Omni count requests. Over the same 30 minutes:

Grafana side
Omni side
Spans sent by the collector
177,455
177,455
Storefront requests
9,320
9,297
Storefront errors
95
96
The span counts are identical. The small difference in requests comes from two time windows that didn't start on exactly the same second. So any difference in the answers later on comes from the tools, not from the data.

First impression: getting started

Turning Omni on took about 30 seconds in the console. It then drew a map of the application from the traces, with no configuration:

Omni's application map, drawn from the traces without any configuration.

It found all four services and the direction of every call, and it recognized Redis as a database. All it needed were two attributes my services already had:  service.name  and  service.namespace=shop . The numbers in the tooltip match the table above.
Grafana draws a similar map from the same traces:

My Grafana’s map of the same traces. The extra arrows from “user“ are Kubernetes health checks.

Mine shows more, and not all of it is useful. The arrows from  user  straight to inventory, orders and payments are Kubernetes health checks, which my configuration counts as customer traffic. The  unknown  node comes from calls that the collector couldn't match up in time. Both can be fixed, but fixing them is my job. Omni's map didn't show them.
Deploying my stack takes about a minute, but it needs a Kubernetes cluster, which took 20 minutes to create, and a few hundred lines of configuration that I have to maintain. For getting started, Omni wins clearly.

Investigator 1: Omni's built-in assistant

The incident from the opening was the inventory bug. After deploying inventory 2.0.0, I asked the assistant my one question. After that I only clicked the follow-up options it suggested itself.
Its first answer, after 3 minutes 45 seconds, named orders as the origin of the problem. That's a reasonable first step: every failed storefront request was a failed call to orders, and it counted them correctly (1,251). But orders was only passing on an error from further down. The same answer contained the sentence about no exception text in the logs, while CloudWatch Logs held both the inventory timeouts and the line  price rule cache miss tenant=tenant-b misses=40 of 40 .
It offered to "Start investigation", which turned out to require DevOps Agent, and I hadn't connected it yet. So I chose its other suggestion, a closer look at orders. Twenty-one seconds later it said that orders was "also a victim". One more click and a minute of waiting, and it reached inventory, using the error recorded on the trace:  ReadTimeout … read timeout=0.8 .
Then came its final answer:


The assistant's final answer: the right service, but the wrong conclusion.

Its own numbers contain the answer. The median response time of inventory (p50) is 52 ms, while the slowest 1% of requests (p99) take 978 ms. When most requests are fast and some take nearly a second, you are usually looking at two different groups of requests. Here they were tenant-b, with 40 cache misses per request, and everyone else. The assistant suggested a slow database, a slow dependency or a busy CPU, and added that "the 0.8 s timeout on orders is tight".
Raising the timeout would have been the wrong fix. Before the release, inventory answered in under 100 ms.
After 15 minutes 37 seconds, one question and four clicks on its own suggestions, the result was 2 out of 5. It found the right service and didn't blame the card network warnings. It didn't find the release, the cache bug or the affected customer.

Investigator 2: Omni with DevOps Agent

The assistant itself had asked for DevOps Agent, so I set it up: an agent space in the DevOps Agent console with a read-only role, connected to Omni in its integration settings. I left "Agent Actions" switched off, because I wanted the agent to investigate, not to roll anything back on its own.
There was one oddity. Even after I connected the agent, the assistant kept saying it wasn't connected. What worked was starting the investigation with a command:
/investigation new storefront error rate is critical since about 12:55
To give it the same problem, I deployed the broken inventory release again, opened a new conversation and sent only that line. The report was complete 8 minutes 49 seconds later:
DevOps Agent summary: inventory v2.0.0 regression redeployed, confirmed root cause of cache misses o

DevOps Agent's report on the inventory bug, from the same data the assistant had.

The title already names the release. The root cause is marked as confirmed: inventory 2.0.0 misses the cache for all 40 of tenant-b's price rules and runs past the 800 ms timeout. It found both times the bug had been deployed, almost two hours apart, and marked them on its timeline together with the rollbacks. It also noticed the payments errors in between, from my other experiments, and labelled them as unrelated, which was correct. The three tiles at the bottom show 74 failed checkouts a minute against the usual 2 to 4, inventory at 900 ms against 45 ms on the previous version, and 40 cache misses out of 40 per request.
It recommended rolling back to version 1.0.0. The assistant on its own had recommended a longer timeout.
Same incident, same data, same question, and this time 5 out of 5. The agent used the log lines that the assistant had said didn't exist.

A bug it hadn't seen before

One objection is fair: by the time DevOps Agent ran, the account already contained an earlier investigation of the same bug. So I ran another test. A script picked one of the two broken releases at random and didn't tell the agent which one. It picked payments. The report came 7 minutes 10 seconds later:
DevOps Agent root cause: payments v1.4.0 deployed with sync_fraud_check=True at 11:20 UTC, fraud_che

The root cause of the payments bug, with the deployment time and the line of code.

payments 1.4.0, deployed between 12:20:16 and 12:20:22 (the report shows UTC, so 11:20). A synchronous fraud check in  payments.py  on line 24. Only tenant-c affected, 330 failed checkouts, orders giving up after 2,000 ms. Nothing about the card network warnings. 5 out of 5 again.
The line number surprised me, because the agent had no access to the code. It came from the logs: OpenTelemetry's logging instrumentation adds the file name, function and line number to every log record, and the agent used them.
I checked every number in the report against Grafana, and they all matched.
The report also has a section I didn't expect, a list of what the agent could not check:
DevOps Agent investigation gaps: kubectl access to the cluster denied, no code repository connected,

The agent's list of what it couldn't check, and why.

It couldn't look inside the Kubernetes cluster, because its role had no access to EKS. It couldn't see the code change, because no repository was connected. It couldn't tell who deployed the release, because EKS audit logs were turned off. And it noticed that only tenant-c was slow, but said it didn't know why instead of guessing.
That list doubles as a to-do list. Give the agent read access to the cluster, connect the repository and turn on audit logs, and the next report should have fewer gaps. The "Build it yourself" section shows how to give it cluster access, but I haven't tested that step yet.
DevOps Agent costs $0.0083 per second of agent work, so each of my investigations would have cost about $3.50 to $4.50. The free trial covered them. The CloudWatch queries the agent runs are billed separately.

Investigator 3: me, with Grafana

I can't give myself a fair score, because I wrote the bug. What is still useful is how quickly the dashboards point to the answer when they have the right labels:

Grafana dashboard: inventory p95 by tenant and version, tenant-b on v2.0.0 at 990 ms, and Loki log

inventory’s response time split by tenant and version, with the matching log lines below.

The top panel shows how long inventory takes for 95% of requests (p95), with one line for every combination of tenant and version. Before the release they were all around 24 ms. After it, tenant-a and tenant-c were at about 95 ms, slower but still fine. They miss the cache too, but they have only two price rules each, so it costs them about 70 ms instead of a second. tenant-b on version 2.0.0 was at 990 ms, above the timeout. This one chart shows the service, the release and the customer. The log panel below it shows the mechanism, in the bug's own words.
It took me 1 minute 24 seconds. That isn't because Grafana is cleverer than Omni. I had made  tenant.id  and  service.version  into labels myself, for exactly this kind of question. My stack answers the questions I prepared it for, and doesn't help with the ones I didn't think of.

The problem the investigations didn't cover

The tenant label also mattered outside the investigations, in alerting. I set up the same alert on both sides: storefront error rate above 5% for one minute.
On my side the alert works per tenant, because one line in the collector configuration ( dimensions: [tenant.id] ) turns the tenant into a label on the metrics. Omni alerts also use PromQL, the query language of Prometheus, so I looked for the same label there:
Omni PromQL query split by span.kind, status.code and tenant.id; the legend shows only span.kind and

In Omni, the request metrics carry the service, span kind and status, but not the tenant.

I asked for the request rate split by span kind, status and tenant, and got four lines: client or server, with or without an error. The tenant part of the query was quietly ignored. The tenant is still in CloudWatch, on every trace and in every log line. But the metrics that Omni’s alerts use are produced by the CloudWatch plugin for OpenTelemetry, and the plugin decides which attributes become labels. My custom  tenant.id  isn’t one of them.
The payments bug showed what that means in practice:
Bar chart of error rate during the payments incident: tenant-a 1.1%, tenant-b 0.7%, tenant-c 23.3%,

Error rate per tenant during the payments incident (lower is better), and the overall rate that Omni’s alert could see.

tenant-c was losing almost one checkout in four, while tenant-a and tenant-b stayed at their usual 1% or less. Because tenant-c sends only a quarter of the traffic, its 23% turned into 5.6% in the overall error rate, and the overall rate was the only number Omni’s alert could see.
The alert fired, but only just. With a slightly milder bug, or a threshold of 6%, it would have stayed quiet while one customer was losing a quarter of its orders. My per-tenant alert fired at 23%.
If one of your customers can have a problem the others don’t, make sure your alerts can see each customer separately. Otherwise the overall average hides exactly that kind of outage.

Results

Horizontal bars: Omni assistant alone 15:37 min, score 2/5; Omni with DevOps Agent on the inventory

The red bar is Omni's assistant on its own: the slowest result, and the only one below 5. The blue bars are Omni with DevOps Agent, on the inventory bug and on the payments bug it hadn't seen before. The dashed outline is me with Grafana.

My Grafana stack
CloudWatch Omni
Ready to use
20 minutes for the cluster, plus my configuration
about 30 seconds
Application map
automatic, but needs tuning
automatic
Alert per customer
one label in the collector
not in the plugin's metrics (needs a custom metric)
Root cause, assistant only
–
2 of 5 in 15 min 37 s
Root cause, with DevOps Agent
–
5 of 5 in 8 min 49 s and 7 min 10 s
Root cause, by hand
5 of 5 in 1 min 24 s (I knew the answer)
–
What I maintain
Tempo, Loki, Prometheus, Grafana, alerting
one collector

Build it yourself

From here on it’s your turn. Reading my scores tells you how three investigators did on my bugs. Running the lab tells you something more useful: how quickly you find a bug you didn’t write, compared with an AI that gets the same data. And if you’re considering Omni yourself, it shows you which of your own labels and alerts would survive the move.
All the code is in the repository github.com/romanceresnak/omni-vs-grafana-lab. There are two ways to use this section:
  • Option A runs the whole lab from AWS CloudShell, including the bugs, and ends with a challenge: find the bug before the agent does.
  • Option B covers the four things you need to get right when you add Omni and DevOps Agent to a cluster you already have. If you use Option A, you don’t need to run anything from Option B, because the scripts already do it.

What you need

  • An AWS account where you can create an EKS cluster, IAM roles, a VPC and CloudWatch resources. A test account with administrator access is the easiest.
  • A Region where both CloudWatch Omni and Amazon DevOps Agent are available. I used  eu-west-1 , which is also the scripts' default. Check the  DevOps Agent Regions  before you start.
  • Room in your account's quotas for one more VPC and one Elastic IP address.
  • About an hour and a half.
  • Some money. The lab costs roughly USD 0.40 an hour in  eu-west-1  (EKS control plane, two t3.large nodes, a NAT gateway and a load balancer), plus CloudWatch ingestion. Each DevOps Agent investigation costs about USD 3.50 to 4.50, unless you are eligible for its free trial.

Option A: run the whole lab from CloudShell

1. Create the cluster

Open AWS CloudShell in your Region, then:
git clone https://github.com/romanceresnak/omni-vs-grafana-lab && cd omni-vs-grafana-lab
chmod +x scripts/*.sh incident/*.sh
echo <YOUR-LAPTOP-PUBLIC-IP> > ~/.omni-lab-ip # Grafana will be reachable only from this IP
source env.sh
./scripts/01-tools.sh # installs eksctl
./scripts/02-cluster.sh # EKS cluster + IAM role for the collector, about 20 minutes
./scripts/03-aws.sh # log group, log stream, Transaction Search check
You can find your public IP at  https://checkip.amazonaws.com .
CloudShell disconnects after about 20 minutes without any input, which is about as long as the cluster takes. Keep the tab active. If it disconnects anyway, run  cd omni-vs-grafana-lab && source env.sh  and start  02-cluster.sh  again. Run  source env.sh  again in every new CloudShell session too.

2. Set up Omni in the console

This is the only part without a script:
  1. In the CloudWatch console, open Omni and start the setup with an Account domain.
  1. Choose a domain name, for example  my-lab . Your Omni sign-in URL becomes  https://my-lab.cloudwatch-omni.global.app.aws .
  1. Let the wizard create its IAM roles.
  1. Create a space, for example  shop-lab , for your account and Region.
  1. Turn on OTel span ingestion (Transaction Search) and OTel metrics enrichment, then finish.
Run  ./scripts/03-aws.sh  again. Continue only when it reports Transaction Search as  CloudWatchLogs  and  ACTIVE . Without that, no traces arrive in CloudWatch.
If you want the second investigator, connect DevOps Agent now, as described in Option B, step 3.

3. Deploy the shop and the Grafana stack

./scripts/04-deploy.sh # Grafana stack, collector and shop
At the end, the script prints a Grafana URL and a password. The load balancer behind the URL can take two or three minutes to answer. Log in as  admin  and open the dashboard Shop – RED, tenants, pipeline. Grafana runs on plain HTTP and accepts connections only from the IP address you saved in step 1. For a lab that holds a few hours of fake orders, that's enough. For real data you would put it behind HTTPS and single sign-on.

4. Verify that both sides see the same data

Don't compare two tools until you've proven they get the same input. Wait five minutes after the deploy, then run:
./scripts/05-verify.sh
It checks the collector's log for export errors, asks Prometheus, Tempo and Loki for the shop's data, and checks CloudWatch for the log stream and for the  aws/spans  log group that Transaction Search creates. Every line must say  [OK] . Then open the Application map in Omni: the application  shop  should show four services and Redis.
If a line says  [FAIL] , it is almost always one of these three:
What you see
What to do
 log stream does not exist  in the collector's log
Run  ./scripts/03-aws.sh  again. It creates the stream.
No  aws/spans  log group
Transaction Search is off. Turn it on (step 2).
Traces arrive, but there is no map and there are no metrics in Omni
Check that the services run with  OTEL_METRICS_EXPORTER=otlp  (Option B, step 2).

5. Your turn: find the bug before the agent does

Start one of the two bugs at random. You don't know which one, and neither does the agent:
# deploys one of the two bugs at random; the answer goes to incident/.answer
./incident/start.sh blind
Within about two minutes, the error rate of one tenant in Grafana climbs well above 20% and the Grafana alert for that tenant fires. Wait five more minutes so that there is enough data, then:
  1. Start a stopwatch and find the cause yourself in the Grafana dashboard. Write down four things: the service, the release, what goes wrong, and which customer is affected.
  1. Ask Omni's assistant the question I used, with your own time: "Storefront errors and latency increased since about HH:MM. What is the root cause?" Follow only the suggestions it offers itself.
  1. Start DevOps Agent with  /investigation new storefront error rate is critical since about HH:MM .
  1. Score all three with the sheet in  EXPERIMENT.md , and only then open  incident/.answer .
If you were faster than the agent, ask yourself why. Which panel or label pointed you to the answer? That is exactly the list of labels you would have to rebuild before you could trust Omni's alerts. If the agent was faster, look at which log lines it quoted. Those are the lines your own dashboards don't show you yet.
Roll back with:
./incident/stop.sh

6. Clean up

./scripts/99-destroy.sh
The script deletes the cluster, the VPC and the load balancer through  eksctl , and then the log group. If CloudShell's credentials expire during the long  eksctl  delete, the comment in  99-destroy.sh  shows how to refresh them. Running it again is safe.
Then, in the console, delete the DevOps Agent space and the Omni space, and turn off Transaction Search if nothing else uses it.
To check that nothing is left behind:

aws eks list-clusters --region eu-west-1
aws cloudformation list-stacks --region eu-west-1 \
--stack-status-filter DELETE_FAILED
aws ec2 describe-addresses --region eu-west-1 --query 'Addresses[].PublicIp'

All three should return empty lists. If a CloudFormation stack shows  DELETE_FAILED , delete it in the console. That's usually the VPC, held back by a leftover load balancer or network interface.

Option B: add Omni and DevOps Agent to your own cluster

If you ran Option A, skip this part. The full collector configuration is in  k8s/10-collector.yaml , so here are only the four things that cost me time and that the documentation doesn't put in front of you.

1. Create the log stream before the collector starts

The collector signs its requests to CloudWatch with SigV4, through one  sigv4auth  extension per signal ( service: xray ,  logs  and  monitoring ). For that, its Kubernetes service account needs an IAM role with  CloudWatchAgentServerPolicy  and  AWSXrayWriteOnlyAccess .  scripts/02-cluster.sh  creates it with  eksctl .
The trap is in the logs. The CloudWatch OTLP logs endpoint writes into the log stream named in the  x-aws-log-stream  header, but it never creates that stream. Without it, every request fails with HTTP 400, and the only place you see that is the collector's own log. Your application keeps answering 200. Create the group and the stream once:
aws logs create-log-group --log-group-name /omni-lab/shop
aws logs create-log-stream --log-group-name /omni-lab/shop \
--log-stream-name otel-collector
and point the exporter at them:
otlp_http/cw-logs:
logs_endpoint: https://logs.eu-west-1.amazonaws.com/v1/logs
headers:
x-aws-log-group: /omni-lab/shop
x-aws-log-stream: otel-collector
auth: { authenticator: sigv4auth/logs }
Traces have a similar trap: they only arrive once Transaction Search sends them to CloudWatch Logs. Check that  aws xray get-trace-segment-destination  says  CloudWatchLogs  and  ACTIVE  before you go looking for missing spans.

2. Give Omni the metrics it needs

Omni's application map and its request, error and duration metrics come from the CloudWatch plugin for OpenTelemetry, which calculates them from the traces inside your application. For Python, it's one more package next to the usual auto-instrumentation. The plugin doesn't export anything itself, so the metrics exporter has to be switched on:
pip install opentelemetry-distro \
opentelemetry-exporter-otlp-proto-http \
cloudwatch-plugin-otel
opentelemetry-bootstrap -a install # instrumentations for Flask, requests, Redis...
export OTEL_SERVICE_NAME=inventory
export OTEL_RESOURCE_ATTRIBUTES=service.namespace=shop,service.version=2.0.0
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4318
export OTEL_METRICS_EXPORTER=otlp
opentelemetry-instrument python inventory.py
 service.namespace  is what groups the services into one application on Omni's map.  service.version  is what allowed both investigators, and my Grafana, to name the bad release.

3. Connect DevOps Agent before you need it

  1. In the AWS DevOps Agent console, create an agent space. Let it create its role, and leave Agent Actions off until you trust it.
  1. In Omni, open Settings → Integrations → Sub agents and connect the space.
  1. Start investigations with  /investigation new <what you see> . In my lab that worked even when the assistant said the agent wasn't connected.
To close the first gap from the agent's report, give its role read access to the cluster. This step wasn't part of my test runs, so if the next report still says kubectl access was denied, check that you used the role ARN of the agent space, not your own:
aws eks create-access-entry --cluster-name omni-lab \
--principal-arn <agent-space-role-arn>
aws eks associate-access-policy --cluster-name omni-lab \
--principal-arn <agent-space-role-arn> \
--policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \
--access-scope type=cluster

4. Write alerts in Omni's PromQL

In Omni's PromQL, attributes of the OpenTelemetry resource, such as the service name, get an  @resource.  prefix, and names with dots need quotes. This is the error-rate alert that fired correctly in both incidents:
100 * sum(rate({"traces.span.metrics.calls",
"@resource.service.name"="storefront",
"span.kind"="SERVER", "status.code"="ERROR"}[2m]))
/
sum(rate({"traces.span.metrics.calls",
"@resource.service.name"="storefront",
"span.kind"="SERVER"}[2m]))
It returns a percentage on purpose, because the threshold field turned my  0.03  into  3 . If one customer matters on its own, you'll need a custom metric that carries the tenant, or a second system that has it.

So, are we switching?

Not now.
Omni does well at the work I like least. There's no Tempo to resize and no Loki to keep alive, and the application map was there in 30 seconds. With DevOps Agent connected, it found the cause of both bugs, including the one it hadn't seen before, and it told me what it couldn't check. That part impressed me.
But we were looking for a reason to modernize, and the benefits we would actually get didn't add up to one. The best result in this test didn't come from Omni alone. It came from DevOps Agent on top, a second service billed for every second it works. The assistant that comes with Omni scored 2 out of 5 and suggested a fix that wouldn't have helped. And our per-tenant alerts, which is how we notice when one customer is in trouble, wouldn't work with the plugin's metrics without extra custom metrics. When we put the price of all that next to what we would gain, it didn't convince us.
So we'll keep our stack, and I'll try Omni with DevOps Agent on one non-production environment, where the agent's investigations can prove themselves on incidents nobody wrote on purpose.
If you're making the same decision, two things from this lab are worth doing before you look at any price list. Connect DevOps Agent before your first real incident, not in the middle of one. And list the labels your alerts depend on and look for each one in Omni first. My tenant was on every trace and in every log line, and Omni's alerts still couldn't use it.
One more thing surprised me: my latency alert in Omni went back to green in the middle of an incident, and kept firing after the rollback. That one needs its own article.
Have you let an AI investigator work on a real incident yet? I'd like to hear whether it found the release, or stopped one service too early.

Sources

  1. Send application telemetry to Omni (CloudWatch plugin for OpenTelemetry):  https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/omni-send-application-telemetry.html 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article