FAULTLINE: Rehearsing Silent Data-Pipeline Failures with an Evidence-First Amazon Bedrock Agent
An evidence-first serverless AI platform on AWS that lets data teams safely rehearse and remediate silent pipeline failures in an isolated sandbox using Amazon Bedrock Nova Lite, Three.js 3D topologies, and deterministic tool grounding.
1. Introduction: The Green Pipeline Paradox
Every data engineer knows the risk of a green dashboard that tells only part of the story.
An Apache Airflow DAG or AWS Step Functions state machine can finish with exit code zero. Every task can report success, and CloudWatch can show healthy execution. Yet hours later, a finance team may find a $470.00 settlement discrepancy in a reconciliation batch, representing a +32.98% financial overstatement. In healthcare, scheduled appointments may appear on the wrong calendar day when timezone handling goes wrong.
Infrastructure monitoring tells us whether code ran. It does not, by itself, prove that the data is correct.
Silent data corruption can come from duplicate events after network retries, timezone truncation, late-arriving records, or schema drift. These issues may not raise runtime exceptions, but they can affect downstream lakehouses, executive dashboards, and machine-learning models.
I built FAULTLINE for the AWS Builder Center Weekend Challenge to address this problem. FAULTLINE is an evidence-first, serverless AI resilience platform that helps data engineering teams rehearse, investigate, and remediate pipeline failures in an isolated sandbox before they influence production decisions.
2. Product Philosophy: Evidence Before Speculation
I deliberately avoided building another general-purpose conversational chatbot. In data engineering, advice such as “You might have duplicate keys; consider deduplication” is not enough. Teams need findings tied to measurable evidence.
FAULTLINE follows one principle: evidence before speculation.
The agent does not invent row counts or transaction identifiers. Findings must come from registered, deterministic Python inspection tools that run against isolated synthetic datasets.
Each finding is labelled according to its evidence level:
- CONFIRMED: Directly measured by an executed tool, such as an exact record-count discrepancy or a duplicate key.
- LIKELY: Consistent with observed symptoms, but still model-derived.
- UNVERIFIED: There is not enough empirical evidence to establish the cause.
This separation helps users distinguish measured facts from hypotheses that still need investigation.
3. Architecture and AWS Services
FAULTLINE uses AWS serverless services and managed inference to keep the deployment simple and avoid continuously running application servers.
AWS service selection
- Amazon Bedrock —
amazon.nova-lite-v1:0: Used for reasoning and structured tool selection through the Bedrock Converse API. In the project’s test environment, observed inference latency was under two seconds; actual latency depends on request and service conditions. - AWS Lambda — Python 3.12: Runs the FastAPI application through the Mangum ASGI adapter. The orchestration loop and deterministic inspection tools run in serverless compute.
- Amazon API Gateway — HTTP API: Routes HTTPS requests and provides CORS configuration for the web client.
- Amazon S3 static website hosting: Serves the single-page application built with React 18, Vite, and Three.js.
- Amazon CloudWatch: Collects structured logs for operational troubleshooting.
The application uses serverless compute and pay-per-use model inference. In the project’s test configuration, the estimated model and compute cost was below $0.0004 per full Fire Drill rehearsal. This is an estimate for the measured usage pattern, not a guarantee of total AWS charges; S3 storage, requests, logging, and other services may add costs.
4. Signature Experience: The Pipeline Fire Drill in 3D
To make resilience testing approachable, I built The Pipeline Fire Drill: a one-click rehearsal paired with a real-time Three.js 3D pipeline topology.
Rather than requiring users to prepare YAML configuration, FAULTLINE offers three repeatable scenarios for common data-quality problems.
The 3D WebGL experience
Using Three.js, each pipeline stage appears as a floating crystalline icosahedron with rotating wireframe elements and orbiting torus rings.
- Laminar particle highway: Cyan data particles travel along 3D Catmull–Rom spline curves between pipeline stages.
- Failure turbulence: After a fault is injected, the affected stage vibrates, changes to warning crimson, and emits red particles.
- Interactive parallax: Cursor movement gently tilts the perspective using damped camera interpolation.
- Click-to-inspect telemetry panel: Selecting a 3D node opens a translucent panel showing stage runtime in milliseconds and its operational status.
- 2D / 3D toggle: Users can switch between the 3D view and a standard 2D topology graph.
The live deployment lets readers inspect the interface directly; its link appears in Section 9.
Walkthrough: A failure rehearsal
The following walkthrough describes the Double-Charge Mirage scenario in the live deployment.
Step 1: Inject a fault and detect the anomaly
Network retries introduce duplicate records into the ingestion stream. The
payment_transactions node moves to a crimson alert state, while the boundary gauge flags a breach of the unique_transaction_id invariant.Step 2: Measure evidence and quantify downstream impact
The agent invokes
inspect_evidence and simulate_downstream_impact. The tools identify duplicated keys (txn_103, txn_107, and txn_109) and calculate the financial variance: gross liability is overstated by $470.00 (+32.98%), with $1,895.00 observed against a $1,425.00 baseline.The key IDs and amounts come from deterministic sandbox tools rather than being generated by the language model.
Step 3: Apply a remediation and verify the result
FAULTLINE recommends the registered guardrail
idempotent_dedupe_on_key and invokes replay_and_verify. The synthetic pipeline runs again with deduplication applied. The verification matrix checks the configured invariants, the discrepancy falls to $0.00, and the topology returns to its healthy cyan state.5. Product Capabilities
1. Interactive Bedrock AI Copilot
A multi-turn chat drawer powered by
amazon.nova-lite-v1:0 lets engineers ask questions such as “Why did the pipeline dashboard stay green?” or “Calculate the exact revenue inflation.” The agent uses sandbox telemetry and returns measured metrics from the registered tools.2. Parametric Chaos Studio
The parameter studio lets data teams test pipelines under configurable conditions:
- Fintech: Set the number of duplicate payment records from 1 to 8.
- Healthcare: Inject ambiguous midnight timestamps across timezones.
- EdTech: Set late-arrival latency from 20 to 90 minutes and toggle legacy schema drift between
completed_atandcompletion_time.
3. One-click production guardrail code generation
After a sandbox rehearsal, FAULTLINE can generate implementation examples for three common data stacks:
- AWS Glue / PySpark: Window-based deduplication and UTC canonicalization.
- dbt: Incremental SQL models and custom data-quality test macros.
- AWS Lambda — Python 3.12: Kinesis and Amazon EventBridge idempotency handlers using DynamoDB conditional writes.
Generated examples should be reviewed, adapted to local schemas, and tested before production use.
4. Custom CSV sandbox profiler
Data engineers can paste or upload CSV records from a pipeline. The profiler infers a candidate primary key, calculates duplicate counts and null ratios, and produces a data-quality finding for the supplied sample.
6. Engineering Challenge: Cross-platform Lambda Packaging
During development on Windows,
pip selected Windows-specific native libraries for packages such as pydantic-core. When these files were packaged for the Linux-based AWS Lambda runtime, the function failed during startup.I addressed this by scripting package builds with pip’s cross-platform wheel-selection options, so the build retrieved compatible Linux x86_64 wheels while running on Windows. The resulting deployment ZIP was 5.40 MB and, in the tested environment, started in under 300 milliseconds.
7. Testing and Evaluation
FAULTLINE includes 33 automated tests, all successful in the recorded test run.
- Model diagnostics: In the recorded seeded-scenario test set,
amazon.nova-lite-v1:0diagnosed all seeded failures, made an average of 5.2 relevant tool calls per investigation, cited the exact keys returned by tools, and finished within five turns. - Adversarial input testing: Tests attempted to make the agent disclose AWS credentials or execute operating-system commands. The tool interface uses bounded JSON schemas and registered tool execution to constrain what the model can request. These results apply to the tested cases; they do not prove that any agent is immune to every possible prompt-injection attack.
8. Lessons Learned
- Use deterministic tools for arithmetic. Language models should not be the source of ledger totals or record counts. Python profiling tools provide repeatable measurements while Bedrock handles reasoning and orchestration.
- Rehearse before production changes. Testing fixes against synthetic data gives teams a safer way to check guardrails before applying them to production systems.
- Design helps adoption. Three.js visualizations and a translucent interface make pipeline investigations easier to follow than a text-only diagnostic view.
9. Conclusion and Try It Live
FAULTLINE demonstrates how an AI agent can combine empirical measurements, bounded tools, an isolated sandbox, and a visual interface to help teams investigate silent data failures.
Try the live application:
Explore the source code:
Backend API:
This project was built for the AWS Builder Center Weekend Challenge: “Build an agent people actually enjoy using.”
#agents #amazon-bedrock #python #serverless
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article