AI agent observability: what a green dashboard misses | Agentic ERP Quickstart
Three operational questions, the CloudWatch signals that answer each, and four limits that catch people out. Plus the rehearsal to run before you trust it.
Series: Building Production-Grade ERP Agents on AWS (18 articles)
- …
- 15AI agent observability: what a green dashboard misses | Agentic ERP Quickstart This article
AI agent observability: what a green dashboard misses | Agentic ERP Quickstart
A green dashboard tells you the components you thought to instrument are behaving. It does not tell you the work got done. An agent can process every case it was handed while the cases that never arrived sit invisibly upstream, and the panel that would have caught it looks identical either way. Three operational questions are worth answering before you let a run go unattended, and the useful part of each answer is knowing exactly where it stops being trustworthy.
Series: Building governed ERP agents on AWS · Series guide →
This post: Which signal answers which operational question, where each one stops being trustworthy, and the rehearsal that turns a dashboard into a procedure.
Prereq: A deployed non-production environment with approved test data. If you intend to rely on notification, confirm that an alarm email subscription is configured and confirmed before you begin.
What you get from this post
An operator's reading of what the deployment gives you. Three questions, the signals that answer each one, and four limits that regularly catch people out. A dashboard being green is not the same as your workflow being observed, and the gap between those two is where overnight incidents live.
You also get:
- The path a case takes through the system, so you know where to look.
- Every custom metric the runtime emits, and the alarm thresholds that ship with it.
- Why an existing alarm does not mean anyone gets notified.
- A five-step rehearsal to run before you rely on any of this unattended.
Time: about 12 minutes. This is an operations explainer plus a rehearsal plan.
New to this series? Start here
The Agentic ERP Automation Quickstart is an open source reference implementation that deploys an AI agent for work your team does by hand from a written procedure, on Amazon Bedrock AgentCore. Accounts payable is the worked example, where a case looks like an invoice that does not match its purchase order. A new process type is a skill directory and your own SOPs. Seven terms recur below:
- A case is how the project represents one ERP exception the agent works on.
- A poller is the scheduled component that reads SAP and turns records into cases.
- SQS is Amazon Simple Queue Service. FIFO means first in, first out. A DLQ is a dead-letter queue, where messages land after delivery keeps failing.
- Amazon CloudWatch collects metrics, logs, and alarms. An alarm watches a metric and changes state when a threshold is crossed.
- Amazon SNS is the notification service. An alarm needs a topic and a confirmed subscription before a human hears about it.
- A trace here is the stored record of what the agent did on one case.
trigger-modeis this project's setting for whether unattended work may start. See the autonomy control post .
Follow the work through the system
Unattended work starts when an Amazon EventBridge schedule invokes the OData poller and
trigger-mode permits an enqueue. Inbound webhook and email paths can produce work too. SQS FIFO delivers the case to the agent invoker, which calls the agent runtime. The agent loads the routed skill and its procedure, calls Gateway tools, and persists case traces.Four independent signal groups let you inspect that path:
| Signal group | What it shows |
|---|---|
Custom ERPAgent metrics | Turns, tokens, estimated cost, latency, success |
| AWS service metrics | AWS Lambda errors and queue depth |
| Observability API health | Alarm state and recent Lambda errors |
| Observability API traces | Recent case traces with case context |
Independent is the useful word. When one of them looks wrong, the others tell you whether the problem is real.
Question 1: Did work arrive and complete?
The dashboard uses the
AgentSuccess sample count for cases processed, and its average for a success-rate view. Read those next to the main queue depth and the DLQ depth.Rising main-queue depth usually means arrivals are outpacing consumer throughput. A visible DLQ message means something to investigate, not something that will retry itself.
The DLQ alarm that ships with the deployment watches
ApproximateNumberOfMessagesVisible, with a threshold above zero over a five-minute period, and it treats missing data as not breaching.IMPORTANT
That alarm existing does not mean anyone will hear it. Alarm creation and notification are separate, and the next section covers why.
Question 2: Is the agent getting slower, pricier, or less predictable?
The runtime emits these custom metrics in the
ERPAgent namespace, dimensioned by ProcessType and ModelTier:AgentTurnsAgentInputTokens,AgentOutputTokens,AgentCacheReadTokens,AgentCacheWriteTokensAgentLatencyMsAgentEstimatedCostUSDAgentSuccess
The dashboard summarizes 24-hour cases, success rate, average estimated cost, and average turns. It graphs token use, estimated cost, p50 and p90 latency, turn distribution, cache activity, Lambda errors, and queue depth.
Two alarms ship alongside it:
| Alarm | Deployed threshold |
|---|---|
| Failure rate | AgentSuccess below 0.90 over one hour |
| Cost | p90 AgentEstimatedCostUSD above $0.50 over one hour |
Treat both as starting points from a reference implementation, not as recommendations. Whether they fit depends on your process types, your traffic pattern, your model configuration, and what your operators are expected to do when one fires.
IMPORTANTAgentEstimatedCostUSDis derived from token counts and a local pricing table. It is an estimate, not an AWS bill. Use an AWS Cost Explorer report for billed infrastructure cost, and do not add the two sources together without a defined methodology.
Question 3: Can an operator investigate a case safely?
The observability API exposes three read-only paths:
| Path | Returns |
|---|---|
/observability/metrics | Performance metrics for a requested time window |
/observability/health | Lambda error counts, queue state, alarms, and a small set of recent log errors |
/observability/traces | Recent case traces with case context |
It is a convenience layer, not a telemetry warehouse, and it has four properties worth knowing before you read a number off it:
- The merged CloudWatch average across dimension series is a simple average, not a weighted one.
- When CloudWatch metrics come back empty, it can derive a fallback from Amazon DynamoDB traces. That is a different data source, so label it as such in any report.
- Trace lookup scans cases and returns at most 50 recent traces.
- Per-case trace persistence is capped too, so the displayed trace list is deliberately bounded.
None of those are defects. They are the reason you should not treat an API response as a complete history of what happened.
An alarm is not a notification
This is the limit that turns into an incident.
The observability construct creates the three core alarms whether or not an email address is configured. It creates an SNS topic and an email subscription only when
alarm_email is set. So a green dashboard and a healthy-looking alarm resource tell you nothing about whether a human would be told.WARNING
Before an overnight run, confirm the subscription is confirmed, then test the whole notification chain in a non-production environment. An unconfirmed subscription is silent in exactly the way that matters.
Record the target environment, the contact path, and the response owner in your operating procedure. Keep personal addresses out of anything published.
Autonomy-change auditing is optional
The CloudTrail trail, metric filter, and alarm that watch autonomy changes exist only when
security.audit_trail_enabled is turned on. The default is off.That default is deliberate. CloudTrail has a hard limit of five trails per Region, and spending one of them on a sample deployment is not the right call in every account.
When you do enable it, the metric filter watches
PutParameter calls against the stack's autonomy parameters. The alarm tells you a parameter changed, including a perfectly authorized change. It does not decide whether the change was authorized, so the alarm is the start of a conversation rather than the end of one.One clarification worth repeating:
trigger-mode controls whether unattended work starts. It is not a SAP write authorization control.
Rehearse before you need it
In an approved non-production environment, run a narrow operational exercise:
- Confirm the selected
trigger-modeand whether the deployment can run unattended at all. - Confirm the CloudWatch dashboard, the health API, the alarm states, and the notification subscription.
- Process approved test cases and inspect success, latency, turns, token, and estimated-cost signals.
- Exercise a controlled failure path, then verify the main queue, the DLQ, the logs, the traces, and the operator response procedure.
- Record the commit, the Region, the thresholds, the data sources, the response owners, and the gaps you did not close.
WARNING
Do not present any of this as a proven overnight runbook until that exercise has been performed and reviewed in the environment you actually run.
Trade-offs
- What you gain. Six layers of visibility that ship with the deployment: business outcome, agent behavior, queues, compute health, alarms, and traces.
- What you give up. Each layer has a stated boundary, and reading one without the others produces confident wrong conclusions.
- What the design does not promise. A billed cost figure, a weighted average, a delivered notification, or universal audit coverage.
Four limits to state out loud
Whenever you hand this to another operator, say all four:
- An estimated cost is not a bill.
- A simple average is not a weighted average.
- An alarm is not a notification subscription.
- Optional autonomy auditing is not universal coverage.
Monitor at more than one layer, and name the limits as clearly as the signals. That is the difference between a dashboard and an operations practice.
Where to go next
- Agent autonomy and unattended mode . What
trigger-modestarts, and what it deliberately leaves alone. - Cedar policy for AI agents . The three conditions that decide whether a request was actually blocked.
Series: Building Production-Grade ERP Agents on AWS (18 articles)
- …
- 15AI agent observability: what a green dashboard misses | Agentic ERP Quickstart This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article