AWS Builder Center
Notes from Building AEGIS: A Self-Healing Backend on AWS

Notes from Building AEGIS: A Self-Healing Backend on AWS

AEGIS is a self-healing backend on AWS that detects infrastructure issues, identifies root causes, safely executes automated remediation, and verifies recovery through real-time CloudWatch metrics. Built with ECS Fargate, Step Functions, DynamoDB, S3, and CloudWatch, the complete detection-to-verification loop ran in 56 seconds during live testing.

Notes from Building AEGIS: A Self-Healing Backend on AWS

Over the past few weeks I shipped AEGIS — an autonomous system that watches a live ECS Fargate service for infrastructure problems, works out what's causing them, and fixes it without a person in the loop. Detection, root cause analysis, a 5-layer safety gate, Step Functions remediation, and then a CloudWatch re-check to verify the symptom actually cleared — the full loop ran in 56 seconds on our last live test. The stack leaned heavily on DynamoDB, CloudWatch, ECS Fargate, S3, and Step Functions, and each one left a different impression by the time we were done.



The good parts

Let's start with what actually worked well, because there was plenty.
ECS Fargate's deployment story was smoother than I expected. Pushing a fresh image to ECR and firing  update-service --force-new-deployment  rolled our API service (4 tasks) and our worker service ( supervisor.py , 1 task, same image, different command) simultaneously — zero dropped requests. We even deliberately scaled the API down to 2 tasks mid-session to create headroom for a  SCALE_OUT  chaos scenario, and it stabilized back to 4 within seconds as a natural side effect of the test. The ECR-to-ECS wiring is tight: once the image is pushed, the rollout finds it by digest with no extra configuration. We never had to think about image resolution.
S3 might be the least exciting thing I can say anything positive about, but that's exactly the point. We set it up once as our evidence store — raw detection and RCA payloads written as JSON blobs under  incidents/{incident_id}/{timestamp}.json  — and then genuinely never thought about it again for the rest of the project. Every chaos scenario wrote its blob. Every  /incidents/{id}/evidence  call returned the correct object. Presigned-style access kept the API stateless. In a system where two other services demanded constant attention, S3's invisibility was a real gift.
Step Functions ended up being a bigger architectural win than we originally planned. Every remediation action AEGIS takes runs as a named execution against  aegis-recovery-workflow . That means every recovery attempt is queryable by ID, has a timestamped start and end, and surfaces  SUCCEEDED  or  FAILED  without us writing a single custom logging statement. When we needed to debug a rollout mid-session, the execution history was the first place we looked — and it was always there, complete. The boundary between "AEGIS decides what to do" and "Step Functions executes and records it" turned out to be a genuinely clean separation that held up under real testing.
DynamoDB, once we moved all shared state into it, made an entire class of bugs disappear. Earlier in the project,  /chaos/history  was returning different counts depending on which of the 4 API replicas happened to answer the request — each task had its own in-process state with no reconciliation. Moving chaos history into the  AegisIncidents  table (tagged  record_type: "chaos_test" ) fixed it instantly. After that: every replica, same data, every time, zero cache-invalidation logic required. That's exactly what you want infrastructure to be.
The ALB just handled everything — CORS, routing across 4 Fargate tasks, health checks, and  idle_timeout=200s  (raised once to stop long-running Step Functions calls getting cut off mid-flight) — and didn't flinch through multiple full service redeployments during testing. For something that could easily have been a source of friction, it wasn't.



The friction

Now the parts that slowed us down.
CloudWatch was the least forgiving service we touched.  get_metric_statistics  threw a  botocore.exceptions.ParamValidationError  inside  detector.py 's  recheck_cleared()  function during our test suite, and the error message pointed nowhere useful — it turned out to be dimension formatting, not the metric name or the time window we assumed at first. We spent time looking in the wrong places. Beyond that: any transient CloudWatch hiccup (a throttle, a blip) just propagates straight up as a raw exception. There's no soft failure, no retry hint baked in. Every other service we used — DynamoDB, S3, ECS — failed more predictably. CloudWatch was the only one that ended up with  try/except  wrapped around every single call by hand, falling back to  checked: False  rather than letting a momentary metric-read failure crash a recovery that already worked at the infrastructure level.
DynamoDB's  limit=100  default bit me hard and silently.  get_incidents()  caps at 100 items unless you override it, and once our  AegisIncidents  table grew past that from chaos testing, the  /status  endpoint just froze at  "total": 100  — forever, with no warning, no truncation flag surfaced anywhere in the response. For a while it looked exactly like a write bug. New records were being created fine; I just couldn't see them through the read cap.  LastEvaluatedKey  is in the response for exactly this reason, but nothing about the default behavior nudges you toward it. For a table that's literally designed to grow — it's an incident store — this should not be a silent default.
Knowing when an ECS rollout actually finished was more work than it should be. After  --force-new-deployment ,  describe-services  shows  PRIMARY  and  ACTIVE  deployments overlapping for a while — the old tasks don't vanish immediately, the new ones haven't all come up yet, and there's no single field that unambiguously means "the rollout is complete, everything is on the new image." We wrote our own poll loop that compared deployment counts until exactly one remained. It worked, but  rollout_state: COMPLETE  is a field that should exist and be readable without building infrastructure around it.
Step Functions' audit trail has a usability gap. All that auditability comes at a cost: if you just want to know "what was the terminal state and why did the execution end," you have to walk the full event history array — each transition is its own timestamped, typed event object. Great for deep debugging. Less great when you're trying to surface a one-line exit reason in a dashboard or incident record. A top-level  last_state  or  exit_summary  field would remove a non-trivial amount of parsing code.



The thing I keep coming back to

The detail I'm most glad we built — and that AWS's infrastructure made genuinely possible — is the verification step at the end.
After Step Functions confirms ECS task count converged, AEGIS re-queries CloudWatch for the real current metric reading. Not the infrastructure state: the actual CPU or memory value that triggered the incident in the first place. Only if that reading has cleared does the incident get marked  RESOLVED . In our live  cpu-spike  test, the response came back with  symptom.current_value: 0.68  — a real CloudWatch read, 0.68% CPU, well below the 80% threshold — before auto-resolving.
That number is the difference between "ECS says tasks are running" and "the service is actually healthy." Most autonomous remediation demos stop at the first signal. We built to the second one. CloudWatch made it possible — despite being the service that gave us the most trouble getting there.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article