
Notes from Building AEGIS: A Self-Healing Backend on AWS
AEGIS is a self-healing backend on AWS that detects infrastructure issues, identifies root causes, safely executes automated remediation, and verifies recovery through real-time CloudWatch metrics. Built with ECS Fargate, Step Functions, DynamoDB, S3, and CloudWatch, the complete detection-to-verification loop ran in 56 seconds during live testing.
Notes from Building AEGIS: A Self-Healing Backend on AWS
Over the past few weeks I shipped AEGIS — an autonomous system that watches a live ECS Fargate service for infrastructure problems, works out what's causing them, and fixes it without a person in the loop. Detection, root cause analysis, a 5-layer safety gate, Step Functions remediation, and then a CloudWatch re-check to verify the symptom actually cleared — the full loop ran in 56 seconds on our last live test. The stack leaned heavily on DynamoDB, CloudWatch, ECS Fargate, S3, and Step Functions, and each one left a different impression by the time we were done.
The good parts
Let's start with what actually worked well, because there was plenty.
ECS Fargate's deployment story was smoother than I expected. Pushing a fresh image to ECR and firing
update-service --force-new-deployment rolled our API service (4 tasks) and our worker service ( supervisor.py , 1 task, same image, different command) simultaneously — zero dropped requests. We even deliberately scaled the API down to 2 tasks mid-session to create headroom for a SCALE_OUT chaos scenario, and it stabilized back to 4 within seconds as a natural side effect of the test. The ECR-to-ECS wiring is tight: once the image is pushed, the rollout finds it by digest with no extra configuration. We never had to think about image resolution.S3 might be the least exciting thing I can say anything positive about, but that's exactly the point. We set it up once as our evidence store — raw detection and RCA payloads written as JSON blobs under
incidents/{incident_id}/{timestamp}.json — and then genuinely never thought about it again for the rest of the project. Every chaos scenario wrote its blob. Every /incidents/{id}/evidence call returned the correct object. Presigned-style access kept the API stateless. In a system where two other services demanded constant attention, S3's invisibility was a real gift.Step Functions ended up being a bigger architectural win than we originally planned. Every remediation action AEGIS takes runs as a named execution against
aegis-recovery-workflow . That means every recovery attempt is queryable by ID, has a timestamped start and end, and surfaces SUCCEEDED or FAILED without us writing a single custom logging statement. When we needed to debug a rollout mid-session, the execution history was the first place we looked — and it was always there, complete. The boundary between "AEGIS decides what to do" and "Step Functions executes and records it" turned out to be a genuinely clean separation that held up under real testing.DynamoDB, once we moved all shared state into it, made an entire class of bugs disappear. Earlier in the project,
/chaos/history was returning different counts depending on which of the 4 API replicas happened to answer the request — each task had its own in-process state with no reconciliation. Moving chaos history into the AegisIncidents table (tagged record_type: "chaos_test" ) fixed it instantly. After that: every replica, same data, every time, zero cache-invalidation logic required. That's exactly what you want infrastructure to be.The ALB just handled everything — CORS, routing across 4 Fargate tasks, health checks, and
idle_timeout=200s (raised once to stop long-running Step Functions calls getting cut off mid-flight) — and didn't flinch through multiple full service redeployments during testing. For something that could easily have been a source of friction, it wasn't.
The friction
Now the parts that slowed us down.
CloudWatch was the least forgiving service we touched.
get_metric_statistics threw a botocore.exceptions.ParamValidationError inside detector.py 's recheck_cleared() function during our test suite, and the error message pointed nowhere useful — it turned out to be dimension formatting, not the metric name or the time window we assumed at first. We spent time looking in the wrong places. Beyond that: any transient CloudWatch hiccup (a throttle, a blip) just propagates straight up as a raw exception. There's no soft failure, no retry hint baked in. Every other service we used — DynamoDB, S3, ECS — failed more predictably. CloudWatch was the only one that ended up with try/except wrapped around every single call by hand, falling back to checked: False rather than letting a momentary metric-read failure crash a recovery that already worked at the infrastructure level.DynamoDB's
limit=100 default bit me hard and silently. get_incidents() caps at 100 items unless you override it, and once our AegisIncidents table grew past that from chaos testing, the /status endpoint just froze at "total": 100 — forever, with no warning, no truncation flag surfaced anywhere in the response. For a while it looked exactly like a write bug. New records were being created fine; I just couldn't see them through the read cap. LastEvaluatedKey is in the response for exactly this reason, but nothing about the default behavior nudges you toward it. For a table that's literally designed to grow — it's an incident store — this should not be a silent default.Knowing when an ECS rollout actually finished was more work than it should be. After
--force-new-deployment , describe-services shows PRIMARY and ACTIVE deployments overlapping for a while — the old tasks don't vanish immediately, the new ones haven't all come up yet, and there's no single field that unambiguously means "the rollout is complete, everything is on the new image." We wrote our own poll loop that compared deployment counts until exactly one remained. It worked, but rollout_state: COMPLETE is a field that should exist and be readable without building infrastructure around it.Step Functions' audit trail has a usability gap. All that auditability comes at a cost: if you just want to know "what was the terminal state and why did the execution end," you have to walk the full event history array — each transition is its own timestamped, typed event object. Great for deep debugging. Less great when you're trying to surface a one-line exit reason in a dashboard or incident record. A top-level
last_state or exit_summary field would remove a non-trivial amount of parsing code.
The thing I keep coming back to
The detail I'm most glad we built — and that AWS's infrastructure made genuinely possible — is the verification step at the end.
After Step Functions confirms ECS task count converged, AEGIS re-queries CloudWatch for the real current metric reading. Not the infrastructure state: the actual CPU or memory value that triggered the incident in the first place. Only if that reading has cleared does the incident get marked
RESOLVED . In our live cpu-spike test, the response came back with symptom.current_value: 0.68 — a real CloudWatch read, 0.68% CPU, well below the 80% threshold — before auto-resolving.That number is the difference between "ECS says tasks are running" and "the service is actually healthy." Most autonomous remediation demos stop at the first signal. We built to the second one. CloudWatch made it possible — despite being the service that gave us the most trouble getting there.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article