Building an Agentic, Real-Time Fraud Detection System on AWS
A proof-of-concept that takes a card transaction from raw signals to an auditable, human-reviewable decision: using Amazon SageMaker, Amazon Bedrock, and Aurora.
Most fraud-detection write-ups end at a confusion matrix. A model gets trained, a ROC-AUC number gets reported, and the story stops right where the hard part begins: what happens when that score has to become a decision, in real time, inside a system a human can trust, audit, and interrupt.
A card transaction scorer has three pressures pulling against each other:
- Fast - a customer is standing at a terminal waiting.
- Accurate - a reviewer shouldn't drown in false positives.
- Humble - when the system isn't confident, it should stop and ask, not silently guess.
Most demos optimize one and quietly drop the other two. This proof-of-concept holds all three at once. That's why it ended up as three cooperating agents plus a decision layer, rather than one model doing everything, and why the whole thing is built to run natively on AWS, from data preparation all the way through to an agentic conversation layer.
This post is the overview: what the POC does, how the pieces fit, and which AWS services carry each part of the workload.
What the system does :
A transaction arrives. Two independent agents score it in parallel one using a trained ML model, one using a live statistical comparison. Their scores are blended into a single number. If that number is low, the transaction is auto-approved and the system moves on cheaply. If it's high enough to matter, a third agent retrieves the relevant fraud-policy passages and a large language model writes a plain-language, citation-backed rationale. If the score lands in a genuinely uncertain zone, the workflow pauses and waits for a human to approve or reject, then resumes exactly where it left off. Every step is wrapped in guardrails, and every decision is logged for audit.
The architecture
The design keeps a clean seam between the agentic application (the orchestration, the agents, the guardrails) and the managed AWS services that do the heavy lifting (training, inference, generation, storage, identity). That seam is the whole point: the same system runs locally for quick iteration, or fully on AWS for production, by switching configuration, not by rewriting code.
Agentic Real-Time Fraud Detection on AWS
The three agents
Agent A - Propensity scoring (the trained model)
A classic machine-learning classifier trained on anonymized transaction features. It's deliberately tuned to be a sensitive, cheap first pass**,** it catches the large majority of real fraud, accepting some false positives in exchange, because everything downstream exists to add the context a bare score can't carry. In production, this model lives behind an Amazon SageMaker endpoint rather than inside the application, so it can be versioned, scaled, and rolled back independently.
Agent B - Behavioral scoring (no model at all)
This is the one that surprises people. Agent B isn't a trained model. It's a live statistical comparison: how unusual is this transaction's amount compared to the typical spend for its merchant category? "Is $1,900 at a restaurant unusual?" is a different question from "does this look statistically like fraud," and a static model answers it poorly because normal shifts by category and over time. Agent B computes it on the fly from data in the database, no training pipeline to maintain.
Agent C - Policy retrieval (the explainer)
Once a transaction clears the review threshold, an approve/deny flag isn't enough, a reviewer needs a reason, grounded in an actual policy document, not an LLM's improvisation. Agent C performs a semantic search over indexed policy documents and hands the most relevant passages to a language model, which writes a cited rationale. A downstream check verifies that every citation the model references was actually one of the retrieved passages, catching hallucinated citations before a human ever sees them.
How they combine
The two scores are blended (weighted toward the trained model). Below a low threshold, the transaction is auto-approved and the expensive policy-retrieval step never runs, which matters for cost and latency, because most transactions are unremarkable. At or above the threshold, Agent C runs and, where the score is ambiguous, the human-in-the-loop gate engages.
Inside the RAG layer
Retrieval-augmented generation (RAG) is what turns a risk score into an explanation a reviewer can act on. Without it, the language model would be asked to justify a fraud decision from its general training data. That is exactly how you get confident, plausible, and unverifiable rationales. With RAG, the model can only reason over passages pulled from your own policy documents, and every claim it makes can be traced back to a source.
The RAG layer has four stages.
1. Ingestion: policy documents become searchable vectors
This step runs offline, as an admin task. It isn't part of the live request path.
- Source documents (card-fraud policies, handbooks, escalation guidelines) are stored in Amazon S3.
- Each document is split into overlapping chunks of roughly a thousand characters. The overlap keeps a sentence that straddles a boundary from losing its meaning.
- Each chunk is converted into a vector with Amazon Bedrock Titan Text Embeddings.
- The vectors are stored in an Aurora PostgreSQL table using the pgvector extension, along with the chunk text, its source document, and page metadata. A vector index (HNSW, cosine distance) keeps lookups fast as the corpus grows.
Re-running ingestion is how the system learns a new policy. No retraining, no redeploy.
2. Retrieval: find the passages that matter for this transaction
When a transaction clears the review threshold, Agent C builds a query from the transaction's risk signals: the category, the amount, how unusual the amount is for its peer group, and the combined score. That query is embedded with the same Titan model and matched against the policy vectors with a nearest-neighbour search. Only the top few passages (three, in the POC) are returned. A short, highly relevant context gives better rationales and costs fewer tokens than a long, noisy one.
3. Generation: a rationale that has to cite its sources
The retrieved passages, the transaction details, and both agent scores go to Claude on Amazon Bedrock. The prompt tells the model to explain the decision using only the supplied passages and to cite the document each point comes from. The output is a short, plain-language rationale written for an analyst, not a data scientist.
4. Verification: trust, but check
Retrieval alone doesn't guarantee a grounded answer, so the output guardrails check the rationale before anyone sees it:
- Citation grounding. Every document the rationale cites must be one of the passages that was actually retrieved. A citation to anything else is treated as a hallucination and flagged.
- Citation presence. If relevant passages were retrieved but the rationale cites none of them, that's suspicious too, and it's flagged.
- PII and harmful-content screening. This catches anything that shouldn't reach the reviewer's screen.
One RAG layer, two uses
The same retrieval pipeline also powers the policy chat tab, where an analyst can ask questions like "what's our escalation rule for high-value restaurant transactions?" Both paths share one index, one embedding model, and one set of guardrails, so the answers in chat and the rationales in scoring stay consistent.
Why pgvector on Aurora, and not a separate vector database?
Keeping vectors in the same Aurora cluster as transactions, memory, and workflow checkpoints means one database to secure, back up, and monitor. Retrieval can also be combined with ordinary SQL filters, such as limiting results to a document type or effective date. For larger corpora or managed ingestion, Amazon OpenSearch Serverless or Amazon Bedrock Knowledge Bases are natural alternatives. The agent code doesn't change, only the retrieval backend does.
One operational detail: the vector column width has to match the embedding model (Titan v2 produces 1024-dimension vectors). If you switch embedding models, re-embed the corpus. Vectors from different models aren't comparable.
The parts that only show up in production
A notebook doesn't need these. A running service does.
- A real human-in-the-loop pause. When the system is uncertain, the workflow genuinely pauses, not a spinner, an actual checkpoint, and waits for a human decision, then resumes. The checkpoint is stored durably so a pause survives a restart and works even when requests land on different instances behind a load balancer.
- Layered guardrails. Checks run before the workflow starts (prompt-injection and jailbreak detection, PII scanning, role-based authorization), during execution (each agent may only use its assigned tool, with rate limits), and after (screening the generated rationale for harmful content, PII leakage, citation grounding, and a valid decision shape).
- Memory. The system keeps a short-term working memory of the current session and a long-term, searchable memory across sessions, so a conversation about a flagged transaction has context.
- Identity and access. Everything past a health check sits behind authentication with a three-tier role hierarchy (viewer, analyst, admin). Roles are re-checked on every request, so revoking access takes effect immediately.
- Observability. Every agent step, tool call, and model invocation is traced, with latency, token usage, and cost captured, plus audit records for events that have no in-flight trace, like a blocked guardrail or a denied access attempt.
The AWS service map
The system covers the full ML lifecycle on AWS: data prep, training, deployment, inference, and the agentic conversation layer on top. It's easiest to read the service map in five stages: Build, Deploy, Run, Govern, Operate.
Build: data and model development
| Service | Role in this system | Why this choice |
|---|---|---|
| Amazon S3 | Data lake for raw transactions, curated train/validation splits, policy documents, and model artifacts | Durable, cheap, versionable storage. Every other ML service reads from and writes to it natively |
| SageMaker Processing | Cleans raw data and produces stratified train/validation splits | Repeatable data prep jobs that run on demand, not on a laptop |
| SageMaker Feature Store | Holds one shared definition of model features | Prevents training/serving skew: the features the model learned on are the features it's scored on |
| SageMaker Training Job | Trains Agent A's propensity model | Reproducible runs with logged hyperparameters, inputs, and metrics |
| SageMaker Model Registry | Versions each trained model with its metrics (ROC-AUC, PR-AUC, recall) and an approval status | A model only reaches production after someone approves a specific, auditable version |
Deploy: packaging and serving
| Service | Role in this system | Why this choice |
|---|---|---|
| SageMaker Endpoint (real-time or serverless) | Serves Agent A's fraud probability | Scale and roll back the model without touching the application. Serverless suits spiky POC traffic, real-time suits steady volume |
| SageMaker Batch Transform | Scores historical files or nightly backfills | Same model, no always-on endpoint cost for offline work |
| Amazon ECR | Stores the application container image | Private, scanned, versioned images |
| AWS App Runner | Runs the FastAPI backend and LangGraph orchestration | The shortest path from container to HTTPS endpoint, with autoscaling and health checks built in. ECS Fargate + ALB is the step up when you need full VPC control or sticky sessions |
Run: agents, generation, and data
| Service | Role in this system | Why this choice |
|---|---|---|
| Amazon Bedrock: Claude | Writes cited rationales, answers policy chat, summarizes session memory | Managed access to strong models. Prompts and responses stay in your AWS account and aren't used to train the models |
| Amazon Bedrock: Titan Text Embeddings | Embeds policy chunks, queries, and memory for semantic search | Same account, same IAM, no third-party key to manage |
| Aurora PostgreSQL Serverless v2 + pgvector | Transaction data for Agent B's peer-cohort stats, policy vectors, working/episodic memory, users, and human-in-the-loop checkpoints | One relational + vector store that scales capacity up and down with load |
| Amazon EventBridge / SQS (optional) | Fan-out of decision and incident events to audit sinks or review queues | New consumers can subscribe without changing the pipeline |
Govern: security and safety
| Service | Role in this system | Why this choice |
|---|---|---|
| Amazon Bedrock Guardrails | Input and output safety: denied topics, PII filtering, prompt-attack detection | Configurable, managed policy applied the same way to every model call |
| Amazon Cognito | User sign-in and JWT issuance with viewer / analyst / admin groups | A real identity provider (MFA, password policies, hosted UI) in place of a hand-rolled token service |
| AWS Secrets Manager | Database credentials and any remaining API keys | Secrets stay out of container images and deploy scripts, and support automatic rotation |
| SSM Parameter Store | Non-secret config such as score thresholds, model IDs, and endpoint names | Tune behaviour without rebuilding the image |
| AWS IAM | One least-privilege role per component | The app can invoke this endpoint and these models, and nothing else |
Operate: observability and audit
| Service | Role in this system | Why this choice |
|---|---|---|
| Amazon CloudWatch | Logs, metrics, dashboards, and alarms (latency, error rate, HITL queue depth, guardrail blocks) | One place to see whether the system is healthy |
| AWS X-Ray | Distributed traces across API → agents → SageMaker → Bedrock → Aurora | Shows exactly where a slow request spent its time |
| Bedrock model-invocation logging | Records prompts and responses | An audit trail for every explanation a reviewer saw |
| AWS CloudTrail | Records API calls across the account | Who changed what, and when |
| SageMaker Model Monitor | Watches live inputs and predictions for drift | Alerts you before a decaying model quietly lets fraud through |
How one request uses these services
- An analyst signs in through Cognito and submits a transaction from the UI.
- App Runner receives the request, validates the token, and loads config from SSM and secrets from Secrets Manager.
- Input guardrails screen the request, with Bedrock Guardrails handling prompt-attack and PII checks.
- Agent A calls the SageMaker endpoint. In parallel, Agent B queries Aurora for peer-cohort statistics.
- If the combined score clears the threshold, Agent C embeds a query with Titan, retrieves passages from pgvector, and asks Claude for a cited rationale.
- If the case is uncertain, the workflow checkpoints to Aurora and waits for a human decision.
- Output guardrails verify the rationale. The decision is returned, and events flow to EventBridge.
- Every hop is visible in CloudWatch and X-Ray, and every model call is in the Bedrock invocation logs.
A note on the data: the architecture doesn't depend on any specific dataset. The public transaction data and policy documents used in this POC are listed under Data sources: try it yourself .
Why this architecture matters
A few properties of this design go beyond "it works on AWS."
Expensive reasoning is paid for only when it's needed. Most transactions are ordinary. Agents A and B are cheap, fast, and run on every transaction. The language model and retrieval steps run only for the minority that clear the threshold. LLM cost and latency therefore track the number of suspicious transactions, not total volume, which keeps the economics of putting generative AI on a payments path workable.
Two signals that fail differently. A trained model and a live statistical check don't share blind spots. The model picks up patterns in the anonymized features. The behavioral check catches amounts that are out of line for the merchant category, even as "normal" drifts. Blending them catches more than either alone, and neither depends on the other being up to date.
Explanations are grounded and checkable. Fraud teams, auditors, and regulators don't accept "the model said so." Every rationale in this system cites the policy passages it was built from, and those citations are verified before display. That turns a black-box score into a decision a person can defend.
Humans stay in control where it counts. The system acts on its own only when it's confident. In the uncertain middle band it pauses, keeps its full state, and hands the decision to a person. Because that pause is a durable checkpoint, not an in-memory wait, it survives restarts and scaling events. That property separates a production workflow from a demo.
Each part of the lifecycle moves independently. The model is versioned and deployed through SageMaker. The policies are re-indexed through the RAG pipeline. Thresholds live in Parameter Store, and the application ships through ECR and App Runner. You can retrain the model, add a policy, tune a threshold, or ship new application code without touching the other three.
Data stays in your account, under your controls. Generation, embeddings, safety checks, storage, and identity all run on AWS services governed by IAM and logged by CloudTrail. No transaction data or policy text has to be sent to a third-party AI provider. For financial data, that's often the deciding factor.
A provider seam, not a rewrite. The agent logic, guardrail layering, and decision flow are independent of which model or store sits behind them. The same system runs locally for fast iteration or fully on AWS for production by changing configuration. That gives teams a realistic path from prototype to production.
Further enhancement and hardening
This is a proof-of-concept. Taking it to production means hardening what's there and extending it where real traffic will push on it. The steps below are grouped by theme and roughly ordered by priority.
1. Network and security hardening
- Private networking. Put Aurora and the application in private subnets. Use an App Runner VPC connector (or move to Fargate), and reach Bedrock, SageMaker, Secrets Manager, and S3 through VPC endpoints (AWS PrivateLink) so traffic never crosses the public internet.
- Edge protection. Put AWS WAF in front of the API for rate limiting, bot control, and managed rule sets. Restrict CORS to the known frontend origin.
- Encryption with your own keys. Use AWS KMS customer-managed keys for Aurora, S3, Secrets Manager, and SageMaker artifacts.
- Secrets rotation. Turn on automatic rotation for database credentials in Secrets Manager.
- Tighter identity. Enforce MFA and advanced security features in Cognito, use short token lifetimes, and give each component its own narrowly scoped IAM role.
- Fail closed on high-risk paths. If a safety check is unavailable, the POC prefers availability and lets the request through. For actions like blocking a card or releasing a held payment, switch to fail-closed and route the case to human review.
- Compliance scope. Card data brings PCI DSS into scope. Tokenize card numbers before they reach the agent layer, keep raw PANs out of logs, prompts, and memory, and document data flows for audit.
2. Reliability and scale
- Shared caching. The semantic cache in front of memory recall is per process today. Move it to Amazon ElastiCache so all instances share hits and invalidations.
- Durable, shared workflow state. Make sure human-in-the-loop checkpoints always go to Aurora (or DynamoDB), never to process memory, so any instance can resume any paused case.
- High availability. Run Aurora across multiple Availability Zones with a reader endpoint for Agent B's read-heavy queries. Enable autoscaling on the SageMaker endpoint.
- Resilient model calls. Add timeouts, retries with backoff, and circuit breakers around Bedrock and SageMaker. Use Bedrock cross-Region inference to absorb capacity spikes.
- Long-running reviews. For reviews that take hours, consider AWS Step Functions with task tokens to model the wait explicitly, with timeouts and escalation paths.
- Streaming ingestion. Feed live transactions in through Amazon Kinesis Data Streams or Amazon MSK rather than request/response only.
3. Model quality and the ML lifecycle
- Automated pipelines. Use SageMaker Pipelines to chain data prep, training, evaluation, and registration, with a quality gate that blocks a model worse than the current champion.
- Safe rollouts. Use shadow testing and canary deployments on SageMaker endpoints to compare a challenger model on live traffic before switching.
- Drift and bias monitoring. Use SageMaker Model Monitor for data and prediction drift, and SageMaker Clarify for feature attributions and bias reports to support each decision.
- Close the feedback loop. Every analyst approve/reject is a labelled example. Capture those decisions in S3 and feed them into retraining.
- Stronger models and features. Try gradient-boosted trees, velocity features (transactions per card per hour), device and merchant risk, and graph features (shared devices or accounts, for example with Amazon Neptune).
- Calibrate thresholds to cost. Tune the auto-approve and block thresholds against the real cost of a missed fraud versus a false decline, not a fixed number.
4. RAG and agent quality
- Better retrieval. Add hybrid search (keyword + vector) and a reranking step to improve the top few passages. Filter on metadata such as policy version and effective date so retired policies are never cited.
- Managed RAG option. Evaluate Amazon Bedrock Knowledge Bases for managed ingestion, chunking, and sync from S3.
- Evaluation as a habit. Build a test set of transactions with expected decisions and reference rationales. Score groundedness, correctness, and citation accuracy on every prompt or model change, for example with Amazon Bedrock evaluations.
- Prompt and model governance. Version prompts with Bedrock Prompt Management, pin model versions, and record which prompt and model produced each rationale.
- Agent runtime options. For teams that want more managed orchestration, Amazon Bedrock AgentCore or Bedrock Agents can host the agent layer while keeping the same tools and guardrails.
5. Operations and delivery
- Infrastructure as code. Define every resource in AWS CDK or Terraform so environments are reproducible and reviewable.
- CI/CD. Build, test, scan, and deploy through CodePipeline or GitHub Actions, with separate dev / staging / prod accounts under AWS Organizations.
- Dashboards and alarms. Track decision latency (p50/p95/p99), auto-approve vs review vs block rates, HITL queue age, guardrail block rate, Bedrock token spend, and endpoint errors.
- Cost controls. Set AWS Budgets alerts, per-environment Bedrock usage limits, and right-size the endpoint and Aurora capacity ranges.
- Load and chaos testing. Load-test at expected peak traffic, and use AWS Fault Injection Service to confirm the system degrades safely when Bedrock, SageMaker, or the database slows down.
A suggested roadmap
| Phase | Focus | Outcome |
|---|---|---|
| 1. Harden | Private networking, KMS, Secrets rotation, WAF, Cognito MFA, fail-closed on high-risk actions, IaC | Safe to expose to internal users with real data |
| 2. Scale | ElastiCache, multi-AZ Aurora, endpoint autoscaling, retries/circuit breakers, streaming ingestion | Handles production traffic and survives component failures |
| 3. Improve | SageMaker Pipelines, Model Monitor, analyst-feedback retraining, RAG evaluation, reranking | Accuracy and explanation quality improve continuously instead of decaying |
Closing thoughts
A well-designed agentic system doesn't need a rewrite to become AWS-native. It needs a thin provider seam. Put the model behind a SageMaker endpoint, point generation and embeddings at Bedrock, keep vectors and state in Aurora, and run identity, secrets, and observability on native services. The agent logic, the guardrails, and the decision flow stay the same.
The result is a fraud-detection POC that's fast on ordinary transactions, careful on uncertain ones, and auditable throughout, which is what a system making decisions about real money needs to be.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article