AWS Builder Center

When Infrastructure Is Green but Users Are Failing: Building SilentFailure on AWS

SilentFailure detects user-impacting failures that traditional infrastructure monitoring can miss by correlating infrastructure health, user journeys, and business outcomes. This article explains how we built and deployed SilentFailure using AWS ECS Fargate, Amazon ECR, Amazon RDS for PostgreSQL, IAM, VPC, Prometheus, and Grafana.

When Infrastructure Is Green but Users Are Failing: Building SilentFailure on AWS

Infrastructure is healthy. Your users are not.
Modern applications have become extremely observable.
We can monitor CPU utilization, memory consumption, latency, error rates, container health, database performance, and service availability.
But there is still a dangerous question that traditional infrastructure monitoring does not always answer:
Are users actually succeeding?
That question led us to build SilentFailure — a user-impact intelligence layer that detects failures that can remain invisible to conventional infrastructure monitoring.



The Problem: A Green Dashboard Can Still Mean a Broken Product

Imagine an e-commerce application during a normal production period.
The infrastructure dashboard shows:
  • CPU: 31%
  • Memory: 42%
  • API error rate: 0.3%
  • API latency: 220 ms
Everything looks healthy.
There is no obvious outage.
There is no massive spike in HTTP 5xx errors.
There is no CPU or memory exhaustion.
But then we look at the actual user journey.
Out of 100 checkout attempts:
  • Only 29 payments succeed
  • Checkout conversion is 29%
  • The expected baseline is 70%
  • Conversion has dropped by 41 percentage points
  • Approximately 71 users are potentially affected
The infrastructure is green.
The business is not.
This is what we call a Silent Failure.



Why Traditional Monitoring Is Not Enough

Traditional observability is extremely valuable. It tells engineering teams what is happening inside their systems.
For example:
CPU → Healthy
Memory → Healthy
Latency → Healthy
Errors → Healthy

But applications are ultimately built for users.
A system can technically respond successfully while still producing a broken user experience.
Consider a payment flow:
Open Checkout
↓
Select Payment Method
↓
Submit Payment
↓
Payment Processing
↓
Order Confirmation

If the payment request technically succeeds but users are redirected incorrectly, a frontend state transition fails, a particular browser behaves differently, or a downstream business condition prevents completion, infrastructure metrics may remain normal.
Users experience failure before infrastructure experiences an outage.
That gap is where SilentFailure operates.



Our Approach

SilentFailure correlates three dimensions:
┌─────────────────────────┐
│ Infrastructure Signals │
│ CPU / Memory / Errors │
│ Latency / Availability │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ User Journey Signals │
│ Checkout / Payment │
│ Success / Failure │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Business Outcomes │
│ Conversion / Impact │
└────────────┬────────────┘
│
▼
SilentFailure
│
▼
Impacted Incident

Instead of asking only:
"Is my service up?"
SilentFailure asks:
"Are users successfully completing the journeys that matter?"



The Detection Model

At the core of the demo is a simple but powerful distinction.

Case 1 — Infrastructure failure

Infrastructure 🔴
Users 🔴

→ Traditional outage


Case 2 — Silent failure

Infrastructure 🟢
Users 🔴

→ SilentFailure 🚨


Case 3 — Healthy system

Infrastructure 🟢
Users 🟢

→ Healthy

This gives the engineering team an additional dimension of observability:
user impact.



Building the System

We built the application as a containerized full-stack system.

Technology Stack

Frontend

  • React
  • TypeScript
  • TanStack Start
  • Vite
  • Tailwind CSS
  • Recharts

Backend

  • Python
  • FastAPI
  • SQLAlchemy
  • PostgreSQL
  • psycopg
  • Docker

Observability

  • Prometheus
  • Grafana

AWS

  • Amazon ECS
  • AWS Fargate
  • Amazon ECR
  • Amazon RDS for PostgreSQL
  • AWS IAM
  • Amazon VPC
  • Security Groups



Shipping the Backend to AWS

We wanted the project to go beyond a local demo.
The backend was containerized using Docker and published to Amazon ECR.
The architecture became:
AWS
┌───────────────────────────────────────────────┐
│ │
│ ┌───────────────┐ │
│ │ Amazon ECR │ │
│ │ Docker Image │ │
│ └───────┬───────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────┐ │
│ │ ECS + AWS Fargate │ │
│ │ FastAPI Backend │ │
│ └─────────┬──────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────┐ │
│ │ RDS PostgreSQL │ │
│ │ Persistent Data │ │
│ └────────────────────┘ │
│ │
│ VPC + Security Groups │
│ │
└───────────────────────────────────────────────┘


Amazon ECR

ECR stores the Docker image used by the backend deployment.
This gave us a clean build-and-ship workflow:
Source Code
↓
Docker Build
↓
Amazon ECR
↓
ECS Fargate


Amazon ECS + AWS Fargate

We deployed the backend as a Fargate task.
This allowed us to run the container without managing servers directly.
The FastAPI service exposes the application APIs and processes telemetry events.

Amazon RDS for PostgreSQL

Incident and telemetry data is persisted in PostgreSQL running on Amazon RDS.
This separates application compute from persistent storage and gives the backend a managed database service.

IAM

An ECS task execution role provides the permissions required by the ECS task execution environment.

VPC and Security Groups

Networking was configured so that the ECS backend could communicate with the RDS database while the database itself was not publicly exposed.



The Demo: Finding a Failure That Looks Healthy

We created a reproducible silent-failure scenario.
The application injects telemetry representing a checkout journey where infrastructure remains healthy but user conversion drops dramatically.
The resulting telemetry looks like this:
Signal
Result
API Error Rate
0.3%
Latency
220 ms
CPU
31%
Memory
42%
Checkout Starts
100
Successful Payments
29
Conversion
29%
Baseline
70%
Conversion Drop
41 pts
Potentially Affected Users
71
SilentFailure identifies the mismatch.
Infrastructure
🟢
│
│
▼
User Journey
🔴
│
▼
Business Impact
🔴
│
▼
Silent Failure Detected

The platform creates an incident rather than waiting for an infrastructure alert.



Turning Telemetry Into an Incident

SilentFailure exposes APIs for health, telemetry, summaries, incidents, analysis, and metrics.
For example:
curl -X POST \
http://localhost:8000/api/demo/seed-silent-failure

The service responds with a detected incident:
{
"seeded": true,
"detected": true,
"incident_id": 1,
"reason": "Silent failure detected."
}

The summary endpoint then exposes the detected impact:
{
"checkout_starts": 100,
"payment_successes": 29,
"conversion_rate": 29.0,
"baseline_conversion": 70,
"conversion_drop": 41.0,
"api_error_rate": 0.3,
"latency_ms": 220,
"cpu_percent": 31,
"memory_percent": 42,
"total_incidents": 1,
"open_incidents": 1
}

This is the key difference.
Instead of producing another generic:
"Service is healthy"
we surface:
"Infrastructure is healthy, but the checkout journey is failing."



Observability With Prometheus and Grafana

SilentFailure does not try to replace existing observability systems.
It complements them.
The backend exposes Prometheus-compatible metrics such as:
silentfailure_active_incidents
silentfailure_checkout_conversion_percent

These metrics can be visualized through Grafana.
For our demo, Grafana provides operational visibility into:
  • Active silent failures
  • Checkout conversion
This creates a layered observability model:
Infrastructure Monitoring
+
Application Metrics
+
User Journey Signals
↓
SilentFailure
↓
User Impact




Why This Architecture Matters

The most important design decision was not a particular framework or AWS service.
It was the decision to treat user outcomes as observability signals.
A CPU graph tells us about CPU.
An HTTP error rate tells us about HTTP errors.
But neither necessarily tells us whether a customer successfully completed a purchase.
SilentFailure connects those signals.
That makes it possible to identify failures that otherwise remain hidden until:
  • customers complain,
  • conversion drops become visible in business analytics,
  • support tickets increase,
  • or engineers manually investigate the journey.



Engineering Lessons

Building SilentFailure taught us several practical lessons.

1. Infrastructure health is not product health

A service can be technically available while users are unable to complete an important workflow.
Monitoring must therefore include the outcome of the workflow, not just the health of the infrastructure supporting it.

2. Correlation is more useful than another isolated metric

Adding another dashboard panel is not necessarily enough.
The value comes from connecting:
System State
+
User Behavior
+
Business Outcome


3. Containers make the deployment path predictable

Docker provided a consistent artifact from local development to AWS deployment.
The deployment flow became:
Code
↓
Docker Image
↓
ECR
↓
ECS Fargate


4. Managed services reduce operational overhead

Using Fargate and RDS allowed us to focus on the detection problem rather than spending the hackathon managing servers and database infrastructure manually.



What We Would Build Next

The current implementation is a working foundation.
The next evolution of SilentFailure would include:
  • Real-time event streaming
  • Learned user journey baselines
  • Automatic anomaly detection
  • Session-level impact analysis
  • More granular user segmentation
  • Cross-service dependency correlation
  • Slack/PagerDuty integrations
  • Automated root-cause investigation
  • Historical incident comparison
  • Multi-service journey tracing
The long-term goal is to move from:
"Something is wrong."
to:
"This user journey started degrading at this time, these segments are affected, the infrastructure remained healthy, and these signals are most strongly correlated with the degradation."



The Bigger Idea

Observability has traditionally evolved around infrastructure.
First we monitored servers.
Then containers.
Then services.
Then distributed systems.
The next layer is understanding whether the system is actually delivering the outcome users expect.
That is the idea behind SilentFailure.
Infrastructure
│
▼
Services
│
▼
User Journeys
│
▼
Business Outcomes
│
▼
┌───────────────┐
│ SilentFailure │
└───────────────┘
│
▼
Actionable Impact

Because ultimately:
A healthy server is not the same thing as a successful user.



Try the Project

GitHub:
The repository contains the frontend, FastAPI backend, Docker configuration, and project documentation.



Final Takeaway

SilentFailure is built around one simple principle:

Don't just monitor whether your infrastructure is alive. Monitor whether your users are succeeding.

When infrastructure is green but users are red, that gap is where silent failures hide.
And that is exactly where SilentFailure looks.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article