How to build the risk ledger
The register, pipeline, nightly rules engine and storage discipline that sit above CloudWatch Omni and turn raw telemetry into six honest statuses.
Part 7 of the series "Containing the agent, counting the value"
Article 6 named the six questions the ledger answers for every production workload: is it owned, costed, proven, contained, stoppable and accountable. Naming them is not the same as answering them. This article builds the machine that does: the register, the pipeline, the nightly rules engine and the storage discipline. It starts with the thing that keeps the whole design small, which is knowing what to build and what to leave to the platform you already run.
Build on the engineering layer, do not rebuild it
The most expensive mistake available here is to build another observability platform. Amazon CloudWatch Omni already stores agent traces, evaluator scores, token counts and service topology. Datadog bills per span, Grafana shipped agent observability in 2026, and Arize and Langfuse hold evaluations. That layer exists and teams already pay for it.
The ledger is not another one of those. It is the layer above them that none of them is: the one that records who owns a workload, what a call costs in dollars, which business outcome it produced, and whether its controls are enforcing. Omni answers "what did this agent do?". The ledger answers "is this agent worth running, and is it under control?", and links back to Omni by trace ID rather than copying the trace. The rule that keeps the build tractable is to add the value-and-risk layer and reference the engineering layer, never duplicate trace storage, span search or evaluation.
The shape of the system
Underneath, the ledger is six layers, each with one job.
- Sources — where AI work happens: AgentCore and Bedrock, SaaS copilots, direct provider APIs, non-AWS platforms.
- Adapters — small plug-ins that receive or pull telemetry from each source and map it to OpenTelemetry.
- Pipeline — a collector that normalises, enriches, redacts, prices, flags and samples every record.
- Workload Register — the source of truth for owner, cost class, P&L line, value case and outcome definition.
- Stores — an engineering store (CloudWatch and Omni, for traces) and a business store (daily facts, cost per outcome, test results) in S3 Tables queried with Athena.
- Dashboards — Omni for engineers, and the ledger's own views for the business.
The register and the pipeline are where the ledger's own work lives. The stores and the engineering dashboards are mostly the platform's. Keep that division clear and the thing you have to build shrinks to a register, a set of adapters, a pipeline and a rules engine.
The register: one source of truth for who and why
Everything starts with the Workload Register. It is the system of record for the facts no telemetry carries: the accountable owner and the other four roles, the cost class (capital, operating or cost of goods sold), the P&L line, the value case with its ceiling and outcome definition, the data classification, and the containment fields from Part B. Each record links to the agent and tool records the platform detects, but the register, not the platform, owns the ownership and value facts.
It is deliberately small and heavily governed: a change to an owner or a cost class is reviewed by the owner and Finance, and every version is kept. This is the opposite of the telemetry stores, which are large and append-only. The register is small, authoritative and slow-changing, and it is what turns a stream of anonymous usage into "this spend belongs to this workload, owned by this person, against this business case."
The pipeline: six processors, and cost is always derived
Between the sources and the stores sits a collector pipeline that runs six processors on every record, in order.
- Normalise — map each vendor's attributes to the OpenTelemetry GenAI convention, through a versioned mapping that also accepts Arize's OpenInference. The conventions are still in development with no official release, so the mapping is pinned and tested before any upgrade.
- Enrich — look up the workload ID in the register and attach owner, cost centre, P&L line, cost class and environment. This is the step that makes everything downstream ownable.
- Redact — drop or hash prompt and response content unless the workload has explicitly opted in, hash the user ID, and run a PII scrubber, so content never reaches the business store.
- Price — derive the cost from token counts and a versioned price catalogue, never trust the cost a source reports. Price input, output, cache-read and cache-write tokens separately, at the right tier and Region. Keep any vendor-reported cost beside it only for reconciliation.
- Flag — mark a record orphan when no register match exists, and mark a frontier model used on a low-complexity workload as a misrouting candidate.
- Sample — keep every error, slow trace, low evaluation score and high-cost trace, and sample the rest, so evaluation and storage costs stay bounded.
Then the record fans out: to CloudWatch for the engineering view, and to the business store as an enriched usage event with no content. Deriving cost rather than trusting the bill is what lets the ledger reconcile its own figure against the invoice later. The most common cause of a mismatch is forgetting the cache-token rates, not a flaw in the method.
The nightly rules engine: raw measures to six statuses
The register and pipeline produce measures. The six tests need statuses, and that is a separate step, run once a night.
The rules engine takes a snapshot of the inputs for each workload (spend, outcomes, evidenced tier, drill results, exposure windows) and evaluates the rules for each of the six tests against it, writing a status, the reason codes behind it, and the rule version that produced it. Two design choices make it trustworthy. The rules are pure functions of the snapshot and a clock, so any night can be re-run and audited rather than taken on faith. And every result records its rule version, so tightening a threshold never silently rewrites yesterday's history, so you can see that the estate looks worse because the rule changed, not because the agents did.
This is also where the five states from article 6 are assigned, including the grey
unknown when an input is missing, and the ageing that turns a stale unknown into a fail after 30 days. The engine is the piece that turns sixty-odd raw measures into the six honest answers a CTO reads in the morning.Store, derive, reference: the discipline that keeps it small
The last design rule is about what the ledger keeps. Sort every piece of data into one of three classes.
- Store — the ledger is the only authoritative copy: the register, value cases, prices, rule config, test results, exposure windows, decisions, benefits and the Evidence Pack manifests. These exist nowhere else, so the ledger owns them.
- Derive — daily facts computed from sources and kept for trend and audit: workload-day cost and tokens, cost per success, pass shares, reconciled spend.
- Reference — point at it, do not copy it: full traces, raw logs and complete security findings stay in their source systems, and the ledger keeps a one-row summary with an ID to follow.
This is the storage expression of the first rule in this article. Everything the ledger stores is something no other system holds. Everything large and already stored, such as the traces, the raw logs and the findings, is referenced by ID. That is what keeps a value-and-risk layer from quietly turning into a second, redundant copy of your observability stack.
What article 8 adds
The architecture here is the skeleton: register, adapters, pipeline, rules engine, stores, and a clear line between what the ledger owns and what it points at. What it does not yet have is its feedstock. A pipeline is only as good as the signals reaching it, and the hardest part of this whole build is not the collector or the rules engine. It is resolving every source to one workload, and generating the four inputs that no platform emits at all.
That is the final article. Article 8 is about the data: the one join key everything hangs on, how to connect an AWS account once rather than per workload, how to reach non-AWS platforms, and the human review time, business outcomes, benefit sign-off and drill records that no telemetry will ever hand you — the inputs that decide whether Costed and Proven can be answered at all. It also shows how to start on a single workload instead of boiling the ocean.
References
- AWS, "CloudWatch Omni (overview)." https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-omni.html
- AWS, "Send telemetry to CloudWatch Omni." https://docs.aws.amazon.com/en_us/AmazonCloudWatch/latest/monitoring/omni-send-telemetry.html
- AWS, "Supported frameworks and telemetry (Amazon Bedrock AgentCore)." https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/supported-frameworks-telemetry.html
- AWS, "Track usage and costs in Amazon Bedrock." https://docs.aws.amazon.com/bedrock/latest/userguide/cost-management.html
- OpenTelemetry, "Semantic conventions for generative AI (in development)." https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article