When Infrastructure Is Green but Users Are Failing: Building SilentFailure on AWS
SilentFailure detects user-impacting failures that traditional infrastructure monitoring can miss by correlating infrastructure health, user journeys, and business outcomes. This article explains how we built and deployed SilentFailure using AWS ECS Fargate, Amazon ECR, Amazon RDS for PostgreSQL, IAM, VPC, Prometheus, and Grafana.
When Infrastructure Is Green but Users Are Failing: Building SilentFailure on AWS
Infrastructure is healthy. Your users are not.
Modern applications have become extremely observable.
We can monitor CPU utilization, memory consumption, latency, error rates, container health, database performance, and service availability.
But there is still a dangerous question that traditional infrastructure monitoring does not always answer:
Are users actually succeeding?
That question led us to build SilentFailure — a user-impact intelligence layer that detects failures that can remain invisible to conventional infrastructure monitoring.
The Problem: A Green Dashboard Can Still Mean a Broken Product
Imagine an e-commerce application during a normal production period.
The infrastructure dashboard shows:
- CPU: 31%
- Memory: 42%
- API error rate: 0.3%
- API latency: 220 ms
Everything looks healthy.
There is no obvious outage.
There is no massive spike in HTTP 5xx errors.
There is no CPU or memory exhaustion.
But then we look at the actual user journey.
Out of 100 checkout attempts:
- Only 29 payments succeed
- Checkout conversion is 29%
- The expected baseline is 70%
- Conversion has dropped by 41 percentage points
- Approximately 71 users are potentially affected
The infrastructure is green.
The business is not.
This is what we call a Silent Failure.
Why Traditional Monitoring Is Not Enough
Traditional observability is extremely valuable. It tells engineering teams what is happening inside their systems.
For example:
CPU → HealthyMemory → HealthyLatency → HealthyErrors → Healthy
But applications are ultimately built for users.
A system can technically respond successfully while still producing a broken user experience.
Consider a payment flow:
Open Checkout ↓Select Payment Method ↓Submit Payment ↓Payment Processing ↓Order Confirmation
If the payment request technically succeeds but users are redirected incorrectly, a frontend state transition fails, a particular browser behaves differently, or a downstream business condition prevents completion, infrastructure metrics may remain normal.
Users experience failure before infrastructure experiences an outage.
That gap is where SilentFailure operates.
Our Approach
SilentFailure correlates three dimensions:
┌─────────────────────────┐│ Infrastructure Signals ││ CPU / Memory / Errors ││ Latency / Availability │└────────────┬────────────┘ │ ▼┌─────────────────────────┐│ User Journey Signals ││ Checkout / Payment ││ Success / Failure │└────────────┬────────────┘ │ ▼┌─────────────────────────┐│ Business Outcomes ││ Conversion / Impact │└────────────┬────────────┘ │ ▼ SilentFailure │ ▼ Impacted Incident
Instead of asking only:
"Is my service up?"
SilentFailure asks:
"Are users successfully completing the journeys that matter?"
The Detection Model
At the core of the demo is a simple but powerful distinction.
Case 1 — Infrastructure failure
Infrastructure 🔴Users 🔴
→ Traditional outage
Case 2 — Silent failure
Infrastructure 🟢Users 🔴
→ SilentFailure 🚨
Case 3 — Healthy system
Infrastructure 🟢Users 🟢
→ Healthy
This gives the engineering team an additional dimension of observability:
user impact.
Building the System
We built the application as a containerized full-stack system.
Technology Stack
Frontend
- React
- TypeScript
- TanStack Start
- Vite
- Tailwind CSS
- Recharts
Backend
- Python
- FastAPI
- SQLAlchemy
- PostgreSQL
- psycopg
- Docker
Observability
- Prometheus
- Grafana
AWS
- Amazon ECS
- AWS Fargate
- Amazon ECR
- Amazon RDS for PostgreSQL
- AWS IAM
- Amazon VPC
- Security Groups
Shipping the Backend to AWS
We wanted the project to go beyond a local demo.
The backend was containerized using Docker and published to Amazon ECR.
The architecture became:
AWS┌───────────────────────────────────────────────┐│ ││ ┌───────────────┐ ││ │ Amazon ECR │ ││ │ Docker Image │ ││ └───────┬───────┘ ││ │ ││ ▼ ││ ┌────────────────────┐ ││ │ ECS + AWS Fargate │ ││ │ FastAPI Backend │ ││ └─────────┬──────────┘ ││ │ ││ ▼ ││ ┌────────────────────┐ ││ │ RDS PostgreSQL │ ││ │ Persistent Data │ ││ └────────────────────┘ ││ ││ VPC + Security Groups ││ │└───────────────────────────────────────────────┘
Amazon ECR
ECR stores the Docker image used by the backend deployment.
This gave us a clean build-and-ship workflow:
Source Code ↓Docker Build ↓Amazon ECR ↓ECS Fargate
Amazon ECS + AWS Fargate
We deployed the backend as a Fargate task.
This allowed us to run the container without managing servers directly.
The FastAPI service exposes the application APIs and processes telemetry events.
Amazon RDS for PostgreSQL
Incident and telemetry data is persisted in PostgreSQL running on Amazon RDS.
This separates application compute from persistent storage and gives the backend a managed database service.
IAM
An ECS task execution role provides the permissions required by the ECS task execution environment.
VPC and Security Groups
Networking was configured so that the ECS backend could communicate with the RDS database while the database itself was not publicly exposed.
The Demo: Finding a Failure That Looks Healthy
We created a reproducible silent-failure scenario.
The application injects telemetry representing a checkout journey where infrastructure remains healthy but user conversion drops dramatically.
The resulting telemetry looks like this:
Signal | Result |
|---|---|
API Error Rate | 0.3% |
Latency | 220 ms |
CPU | 31% |
Memory | 42% |
Checkout Starts | 100 |
Successful Payments | 29 |
Conversion | 29% |
Baseline | 70% |
Conversion Drop | 41 pts |
Potentially Affected Users | 71 |
SilentFailure identifies the mismatch.
Infrastructure 🟢 │ │ ▼User Journey 🔴 │ ▼Business Impact 🔴 │ ▼Silent Failure Detected
The platform creates an incident rather than waiting for an infrastructure alert.
Turning Telemetry Into an Incident
SilentFailure exposes APIs for health, telemetry, summaries, incidents, analysis, and metrics.
For example:
curl -X POST \ http://localhost:8000/api/demo/seed-silent-failure
The service responds with a detected incident:
{ "seeded": true, "detected": true, "incident_id": 1, "reason": "Silent failure detected."}
The summary endpoint then exposes the detected impact:
{ "checkout_starts": 100, "payment_successes": 29, "conversion_rate": 29.0, "baseline_conversion": 70, "conversion_drop": 41.0, "api_error_rate": 0.3, "latency_ms": 220, "cpu_percent": 31, "memory_percent": 42, "total_incidents": 1, "open_incidents": 1}
This is the key difference.
Instead of producing another generic:
"Service is healthy"
we surface:
"Infrastructure is healthy, but the checkout journey is failing."
Observability With Prometheus and Grafana
SilentFailure does not try to replace existing observability systems.
It complements them.
The backend exposes Prometheus-compatible metrics such as:
silentfailure_active_incidentssilentfailure_checkout_conversion_percent
These metrics can be visualized through Grafana.
For our demo, Grafana provides operational visibility into:
- Active silent failures
- Checkout conversion
This creates a layered observability model:
Infrastructure Monitoring +Application Metrics +User Journey Signals ↓ SilentFailure ↓ User Impact
Why This Architecture Matters
The most important design decision was not a particular framework or AWS service.
It was the decision to treat user outcomes as observability signals.
A CPU graph tells us about CPU.
An HTTP error rate tells us about HTTP errors.
But neither necessarily tells us whether a customer successfully completed a purchase.
SilentFailure connects those signals.
That makes it possible to identify failures that otherwise remain hidden until:
- customers complain,
- conversion drops become visible in business analytics,
- support tickets increase,
- or engineers manually investigate the journey.
Engineering Lessons
Building SilentFailure taught us several practical lessons.
1. Infrastructure health is not product health
A service can be technically available while users are unable to complete an important workflow.
Monitoring must therefore include the outcome of the workflow, not just the health of the infrastructure supporting it.
2. Correlation is more useful than another isolated metric
Adding another dashboard panel is not necessarily enough.
The value comes from connecting:
System State +User Behavior +Business Outcome
3. Containers make the deployment path predictable
Docker provided a consistent artifact from local development to AWS deployment.
The deployment flow became:
Code ↓Docker Image ↓ECR ↓ECS Fargate
4. Managed services reduce operational overhead
Using Fargate and RDS allowed us to focus on the detection problem rather than spending the hackathon managing servers and database infrastructure manually.
What We Would Build Next
The current implementation is a working foundation.
The next evolution of SilentFailure would include:
- Real-time event streaming
- Learned user journey baselines
- Automatic anomaly detection
- Session-level impact analysis
- More granular user segmentation
- Cross-service dependency correlation
- Slack/PagerDuty integrations
- Automated root-cause investigation
- Historical incident comparison
- Multi-service journey tracing
The long-term goal is to move from:
"Something is wrong."
to:
"This user journey started degrading at this time, these segments are affected, the infrastructure remained healthy, and these signals are most strongly correlated with the degradation."
The Bigger Idea
Observability has traditionally evolved around infrastructure.
First we monitored servers.
Then containers.
Then services.
Then distributed systems.
The next layer is understanding whether the system is actually delivering the outcome users expect.
That is the idea behind SilentFailure.
Infrastructure │ ▼ Services │ ▼ User Journeys │ ▼ Business Outcomes │ ▼ ┌───────────────┐ │ SilentFailure │ └───────────────┘ │ ▼ Actionable Impact
Because ultimately:
A healthy server is not the same thing as a successful user.
Try the Project
GitHub:
The repository contains the frontend, FastAPI backend, Docker configuration, and project documentation.
Final Takeaway
SilentFailure is built around one simple principle:
Don't just monitor whether your infrastructure is alive. Monitor whether your users are succeeding.
When infrastructure is green but users are red, that gap is where silent failures hide.
And that is exactly where SilentFailure looks.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article