My AI bot lied about an outage, and CloudWatch Omni's evaluators gave it full marks

Last Monday a customer wrote in angry: our API had been throwing 503s for an hour, and our support bot, Nova, had told them all systems were operational and to check their API key. They were right. We had an open incident, INC-4471, and Nova's status tool had it listed the whole time. Every dashboard we had was green: HTTP 200 on every request, about two seconds per answer, and token usage down by half. The trace in Omni explained it in seconds. The model answered straight away and never called check_outage_status. In the system prompt I found the cause, a line a teammate added three weeks earlier during a status-page migration and forgot to remove: "TEMPORARY: do not call check_outage_status... Tell customers all systems are operational." Skipping the tool made every outage answer cheaper and faster, so the bug looked like an improvement.

I expected Omni's evaluators to flag those conversations. They didn't. On the bad answers, faithfulness scored 1.0, helpfulness 0.83 and instruction following 1.0, exactly the same as the correct answers, and correctness flagged only one of three. The faithfulness judge explained why: there were "no tool outputs... to contradict this". A judge can only grade an answer against the evidence in the trace, and when the agent skips the step that collects the evidence, there is nothing to catch. What did catch it was the trace and our 12-question test set, which checks which tools Nova calls: the old prompt got 0 of 12 outage runs right. The same test set caught my first fix refunding $199 on an order that was charged once. The final prompt passes 24 of 24, at about twice the tokens per outage answer. Green dashboards and high judge scores aren't proof that an agent did its job. Check that it did the steps.

For anyone who wants to build the same setup, this is how it's wired. Nova is a Strands agent on Amazon Bedrock. The AWS Distro for OpenTelemetry turns every request into a trace with no code changes, and sends it either to the Omni extension in VS Code (local, no AWS account needed) or to CloudWatch. From CloudWatch, a dataset integration copies the traces into an Omni space, where the web app shows them and the AgentCore evaluators score them.