
AEGIS: Autonomous Engine for Guarding Integrated Systems — Building an Autonomous Self-Healing System on AWS
AEGIS (Autonomous Engine for Guarding Integrated Systems) is an AWS-based autonomous self-healing infrastructure system that detects anomalies, correlates signals, identifies root causes, and safely executes remediation without human intervention. Built with CloudWatch, ECS Fargate, DynamoDB, S3, Step Functions, ECR, and ALB, AEGIS creates a closed-loop pipeline: Detect → Diagnose → Validate → Remediate → Verify → Audit.
Building AEGIS: An Autonomous Self-Healing System on AWS — What Worked and What Didn't
We built AEGIS — Autonomous Engine for Guarding Infrastructure & Systems — an autonomous self-healing infrastructure system designed to detect failures, reason about their likely causes, safely execute remediation, and verify that the system actually recovered.
Unlike a traditional monitoring system that simply raises an alert, AEGIS closes the entire operational loop:
Detect → Diagnose → Safety Gate → Remediate → Verify → Resolve
The system watches a live Amazon ECS Fargate environment and detects anomalies such as CPU spikes, task crashes, memory pressure, elevated error rates, and database connectivity failures. It correlates these signals, determines a likely root cause with a confidence score, and passes the proposed remediation through a strict five-layer safety gate.
Only when every safety condition is satisfied does AEGIS autonomously execute the recovery workflow through AWS Step Functions.
Then comes the part we consider most important: AEGIS does not assume that a successful infrastructure operation means the problem is fixed. It re-queries the actual Amazon CloudWatch metric that triggered the incident and verifies that the symptom has genuinely cleared.
No human clicks a button.
In our latest live test, the complete detect-to-resolution loop executed in approximately 56 seconds against a live deployment.
AWS provided the infrastructure foundation for the entire system. But building AEGIS also exposed several places where the cloud services worked exceptionally well — and several places where they created unexpected engineering friction.
This is the honest breakdown.
What AEGIS Actually Does
At a high level, AEGIS operates as a closed-loop autonomous operations system.
┌──────────────────────┐ │ Live AWS Services │ │ ECS / ALB / CloudWatch│ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ Detection │ │ Anomaly Detection │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ Signal Correlation │ │ + Root Cause Analysis│ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ Safety Gate │ │ 5 Validation Layers │ └──────────┬───────────┘ │ Approved Action │ ▼ ┌──────────────────────┐ │ Step Functions │ │ Autonomous Recovery │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ CloudWatch Recheck │ │ Did the symptom clear│ │ for real? │ └──────────┬───────────┘ │ ┌─────┴─────┐ ▼ ▼ RESOLVED FAILED
The distinction between remediation success and system recovery is fundamental.
For example, restarting an ECS task successfully proves that ECS performed the requested operation. It does not prove that the CPU spike, memory pressure, or error rate disappeared.
AEGIS therefore treats infrastructure state and application health as two separate signals.
Where Things Got Frustrating
Building autonomous infrastructure automation means dealing with every edge case that normally gets hidden behind manual operational workflows.
Several AWS services exposed exactly those edge cases.
1. DynamoDB's Silent 100-Item Cap Almost Fooled Us
One of our earliest debugging sessions involved an apparently broken incident store.
Our
IncidentStore.get_incidents() method defaulted to:limit = 100
Once our
AegisIncidents table grew beyond 100 records during chaos testing, the /status endpoint continued reporting:"total": 100
No exception was thrown.
The writes were working.
The table contained more than 100 records.
But the API appeared permanently stuck at 100.
The problem was entirely on the read side.
DynamoDB's pagination model uses
LastEvaluatedKey to indicate that additional records are available. The mechanism itself works, but the initial implementation made it easy to mistake the returned page size for the actual number of records.For an incident store that is explicitly designed to grow, this became an important lesson:
A bounded read is not the same thing as a bounded dataset.
We eventually handled pagination explicitly rather than treating the first response as the complete dataset.
The interesting part was not the fix itself. It was how long we initially spent investigating writes, replicas, and API state when the underlying problem was simply pagination.
2. CloudWatch Errors Pointed Nowhere Useful
CloudWatch was probably the AWS service that demanded the most defensive coding.
During testing,
get_metric_statistics threw a:botocore.exceptions.ParamValidationError
inside our
detector.py recheck_cleared() function.Initially, we investigated the metric name and time window.
The actual issue was related to dimension formatting.
That distinction matters in an autonomous system because the remediation workflow depends on CloudWatch for both detection and verification.
A failure in the verification stage can create a particularly awkward situation:
Remediation succeeded ↓CloudWatch verification fails ↓System cannot determine whether recovery succeeded
We therefore wrapped CloudWatch calls with explicit exception handling and made verification fail safely:
checked: False
rather than allowing a transient CloudWatch problem to crash an otherwise successful recovery workflow.
This became a general design principle for AEGIS:
Infrastructure telemetry failure should not automatically become infrastructure failure.
CloudWatch also introduced another important requirement: treating telemetry as an external dependency rather than assuming that every metric request will succeed.
Throttling, transient service errors, malformed dimensions, and unexpected responses all need to be considered when telemetry is part of an autonomous control loop.
3. ECS Rollouts Have No Clean "Settled" Signal
Deploying a new ECS task definition was straightforward.
Knowing exactly when the deployment was completely settled was less straightforward.
After triggering:
update-service --force-new-deployment
ECS can temporarily report overlapping deployments.
We could see both:
PRIMARYACTIVE
during the transition.
There wasn't a single application-level field that gave us the exact semantic we wanted:
"Every desired task is now running the new version and the old deployment is completely gone."
So AEGIS implemented its own polling logic.
We repeatedly called
describe-services and compared deployment state until the deployment converged to a single active deployment.It worked.
But autonomous remediation requires deterministic state transitions, and deployment orchestration often needs an abstraction slightly higher than the raw infrastructure API.
For AEGIS, that meant creating our own notion of:
DEPLOYING ↓CONVERGING ↓STABLE
rather than assuming that the presence of a
PRIMARY deployment meant the rollout was finished.
4. Step Functions Execution History Is Excellent — But Verbose
Step Functions became one of the strongest parts of the architecture, but its execution history also introduced an interesting trade-off.
Every transition is represented as an individual event.
For full auditability, this is excellent.
For answering a simple question such as:
"Why did this remediation finish?"
it means walking through the execution history and identifying the relevant terminal event.
The history gives us exactly what happened, when it happened, and which state transitioned.
That is extremely valuable for debugging autonomous systems.
But when building a dashboard or incident API, we wanted a concise representation such as:
status: FAILEDlast_state: VerifyRecoveryexit_reason: MetricNotCleared
Instead, our application had to derive that summary from the underlying execution history.
This was not a blocker, but it reinforced an important distinction:
Audit data and operational summaries serve different purposes.
We needed both.
Where AWS Delivered
Despite the challenges, several AWS services performed exceptionally well in the architecture.
ECS Fargate Rolling Deployments Were the Standout
Our API and worker services ran on ECS Fargate.
During testing, we pushed a new image to ECR and triggered:
update-service --force-new-deployment
on both services.
The deployments rolled over without dropped requests.
We also deliberately changed the desired task count during chaos testing, including reducing the service from four tasks to two to simulate scale-related scenarios.
The combination of:
ECR → ECS → Fargate → ALB
gave us a clean deployment pipeline without requiring us to manually manage servers.
The service could replace tasks, maintain desired capacity, and continue receiving traffic through the load balancer.
For an autonomous remediation platform, this matters significantly.
AEGIS doesn't need to SSH into machines or execute shell commands on individual servers.
It can express remediation at the infrastructure abstraction level:
Change desired state ↓AWS orchestrates the transition ↓Service converges
That makes the remediation workflow much safer and easier to reason about.
DynamoDB Became Our Source of Truth
Ironically, the same service that caused our pagination problem also solved another major architectural problem.
Earlier in development, our
/chaos/history endpoint maintained state inside individual ECS tasks.With four replicas, this meant the response depended on which replica handled the request.
One task might report:
12 chaos tests
while another reported:
9 chaos tests
There was no shared source of truth.
Moving the history into the
AegisIncidents DynamoDB table solved the problem.We tagged these records with:
record_type = "chaos_test"
After the migration, every replica read from the same persistent datastore.
No cache invalidation.
No cross-replica reconciliation.
No synchronization mechanism.
Just a centralized source of truth.
This is exactly what infrastructure should feel like when it is working correctly:
invisible.
S3 Quietly Did Its Job
S3 became our evidence store.
Each chaos scenario generated a detection payload that was persisted as a JSON object:
incidents/ {incident_id}/ {timestamp}.json
The API could later retrieve the corresponding evidence through:
/incidents/{id}/evidence
This gave us an important separation:
DynamoDB ↓Incident metadata and state
S3 ↓Detailed evidence and artifacts
The API itself remained stateless.
S3 did not require much operational attention after the initial implementation.
And that was a good thing.
In a system where CloudWatch and ECS required significant defensive logic, having object storage that simply persisted and retrieved our evidence was extremely valuable.
Step Functions Gave Us an Audit Trail
Every AEGIS remediation runs through a named Step Functions workflow:
aegis-recovery-workflow
That gave us something extremely valuable for autonomous systems:
traceability.
Every remediation has:
- An execution ID
- Start timestamp
- End timestamp
- Individual state transitions
- Success/failure status
- Execution history
We did not need to build a separate custom workflow logging system to answer basic questions about what the remediation engine did.
The architectural separation became very clean:
AEGIS │ │ decides what should happen ▼Step Functions │ │ executes the workflow ▼AWS │ │ records execution ▼Audit History
When something went wrong during testing, the Step Functions execution history was often the first place we looked.
And it was always there.
For an autonomous system, that auditability is not just convenient.
It is part of the safety model.
The ALB Just Handled Everything
The Application Load Balancer sat in front of our Fargate tasks and handled:
- Traffic routing
- Health checks
- CORS-related integration
- Traffic distribution across tasks
- Connection behavior during deployments
We also tuned the idle timeout to approximately:
200 seconds
for longer-running chaos operations involving Step Functions.
The important part was that repeated redeployments did not require us to manually reconfigure the load balancer.
Tasks could disappear.
New tasks could appear.
The ALB continued routing traffic to healthy targets.
That is precisely the kind of infrastructure abstraction that makes autonomous remediation practical.
The Part We're Most Proud Of: Verification
The most technically important part of AEGIS is not that it can restart a service.
It is that AEGIS verifies whether the remediation actually worked.
A conventional remediation workflow might look like:
CPU spike detected ↓Restart ECS task ↓ECS task running ↓SUCCESS
AEGIS does something different.
CPU spike detected ↓Diagnose root cause ↓Safety gate ↓Restart / scale / remediate ↓Wait for infrastructure convergence ↓Query CloudWatch again ↓Is CPU actually below threshold? │ ┌──┴──┐ │ │ YES NO │ │RESOLVED FAILED / ESCALATE
This distinction is crucial.
Infrastructure state is not the same thing as application health.
An ECS task can be:
RUNNING
while the application is still unhealthy.
A deployment can be:
STABLE
while the original symptom remains.
A remediation operation can be:
SUCCEEDED
while the incident itself is still unresolved.
AEGIS therefore performs a counterfactual check against the original symptom.
A Real Example
During one of our live tests, AEGIS detected a:
cpu-spike
incident.
The incident passed through the complete pipeline:
Detection ↓Correlation ↓Root Cause Analysis ↓Safety Gate ↓Step Functions ↓ECS Remediation ↓CloudWatch Verification ↓RESOLVED
The final verification returned:
symptom.current_value: 0.68
That value came from a real CloudWatch metric read after remediation.
The important thing wasn't simply that ECS reported healthy tasks.
It was that the metric that originally triggered the incident had actually cleared.
That is the difference between:
"The recovery command succeeded."
and:
"The system recovered."
For an autonomous self-healing system, that distinction is fundamental.
What We Learned
Building AEGIS changed how we think about autonomous infrastructure.
The difficult part isn't simply giving an AI agent permission to execute AWS APIs.
The difficult part is building everything around that capability.
An autonomous system needs:
Observability
It must know what is happening.
Reasoning
It must determine what is likely causing the problem.
Safety
It must decide whether an action is safe to execute.
Execution
It needs reliable infrastructure primitives capable of carrying out the action.
Verification
It must independently determine whether the remediation actually worked.
Auditability
It must be possible to reconstruct what happened afterward.
Our final architecture reflects those requirements:
┌─────────────────┐ │ CloudWatch │ │ Telemetry │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ AEGIS │ │ Detection │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Root Cause + │ │ Confidence │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ 5-Layer Safety │ │ Gate │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Step Functions │ │ Execution │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ ECS Fargate │ │ Remediation │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ CloudWatch │ │ Re-verification │ └────────┬────────┘ │ ┌─────┴─────┐ ▼ ▼ RESOLVED ESCALATE
Final Takeaway
AWS gave us most of the primitives required to build AEGIS without managing the underlying servers ourselves.
ECS Fargate provided resilient workload orchestration.
DynamoDB provided shared persistent state.
S3 provided durable evidence storage.
Step Functions provided workflow execution and auditability.
CloudWatch provided the telemetry required to detect and verify failures.
ECR provided the image delivery pipeline.
ALB provided stable traffic routing across changing infrastructure.
But the experience also reinforced something important:
Cloud infrastructure APIs are not automatically autonomous-system APIs.
An autonomous control loop needs additional engineering around pagination, retries, validation, deployment convergence, state interpretation, failure handling, and verification.
The biggest lesson from AEGIS was therefore not that AWS can self-heal infrastructure.
It was that reliable self-healing requires a closed feedback loop.
Detect the problem.
Understand the problem.
Decide whether action is safe.
Execute the action.
Measure the system again.
Only then decide whether the incident is actually resolved.
That final feedback step is what turns infrastructure automation into a genuine self-healing system.
#BharatBuilds #WeMakeDevs #AWS #CloudComputing #DevOps #AIOps #SelfHealingSystems #CloudNative #InfrastructureAutomation #AI #GenerativeAI #AutonomousSystems #AWSCloud
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article