AWS Builder Center

From AI Recommendation to Bounded EKS Remediation: Validating a Risk-Gated Control Plane with Fault Injection

This article shows how to turn AI-generated remediation recommendations into controlled, auditable actions on Amazon EKS. It covers structured remediation contracts, a ten-stage decision workflow, policy-based authorization, scoped Kubernetes permissions, fault-injection validation, independent recovery checks, investigation cost limits, and clear boundaries for actions that should remain manual.

Series: Engineering Safe AI-Assisted Remediation on Amazon EKS (2 articles)

  1. 2
    From AI Recommendation to Bounded EKS Remediation: Validating a Risk-Gated Control Plane with Fault Injection This article
DevOps Cloud Platform & Site Reliability Engineer | AWS Certified Solutions Architect – Professional



Turning AI-assisted incident investigation into controlled, auditable, and reversible production action on Amazon EKS

An AI operations agent can identify a likely cause of an EKS incident long before anyone should give it permission to change the cluster.
That separation matters.
AWS DevOps Agent can investigate incidents, correlate operational context, identify root causes, and produce mitigation plans. It can also correlate related incidents to reduce duplicate investigative work. None of that means an application team has to treat the resulting mitigation as an authorization decision.
Consider a recommendation that looks harmless:
Rollback payments-api to the previous Deployment revision.
Before execution, several questions remain unanswered.
Is the previous revision still compatible with the current database schema? Is the incident actually correlated with the latest rollout? Is the proposed change limited to one namespace? Is another engineer already modifying the Deployment? Does the execution identity have permission to change anything beyond this workload?
The difficult part of AI-assisted remediation is not generating the recommendation. It is building a control plane that can say yes, not yet, or absolutely not for reasons that do not depend on the agent agreeing with them.
The reference architecture below uses a structured remediation contract, a ten-stage workflow, policy-based authorization, narrow EKS permissions, controlled execution, post-change verification, and an explicit investigation budget. The implementation examples are intentionally bounded. The objective is not to automate every incident.
It is to make a small set of understood remediation actions safe enough to automate.

Start with a remediation contract

An AI-generated paragraph should never be executable input.
The investigation layer should first normalize its recommendation into a schema that the control plane understands.
{
"incidentId": "INC-4812",
"cluster": "prod-eks",
"namespace": "payments",
"workload": "payments-api",
"action": "RollbackDeployment",
"targetRevision": "41",
"evidence": {
"failureStartedAfterDeployment": true,
"previousReplicaSetHealthy": true,
"statefulMigrationDetected": false
}
}
There is intentionally no  command  field.
The agent chooses from a small vocabulary of operations. It does not invent:
kubectl ...
aws ...
curl ...
That distinction prevents the remediation interface from becoming a disguised remote shell.
For the reference implementation, I would initially allow only three automatic action classes:
RestartPods
RollbackDeployment
ScaleDeploymentWithinBounds
And I would reject at least these high-impact classes from autonomous execution:
ModifyIAM
DeletePersistentStorage
ModifyDatabaseSchema
ClusterWideConfigurationChange
The list is deliberately conservative. It can grow later, but every new operation expands the failure surface of the executor.

The ten stages should be observable

A remediation workflow becomes difficult to reason about when investigation, authorization, execution, and validation happen inside one function.
I would make the lifecycle explicit:
1. Proposal intake
2. Schema validation
3. Live-state collection
4. Evidence validation
5. Risk classification
6. Policy authorization
7. Human approval if required
8. Controlled execution
9. Independent verification
10. Close, rollback, or escalate
These are not cosmetic workflow states.
Each one closes a different failure mode.
Proposal intake gives the remediation a unique incident and execution identity.
Schema validation ensures the requested operation belongs to the supported vocabulary.
Live-state collection matters because evidence becomes stale. If the agent investigated revision 42 and revision 43 has since been deployed, a rollback decision based on revision 42 should stop.
Evidence validation checks that the observations actually exist. The policy should consume facts such as Deployment revision, ReplicaSet health, probe failures, current availability, and change-window state rather than a natural-language confidence statement.
Risk classification determines how much infrastructure the proposed action could affect and whether reversal is straightforward.
Policy authorization decides whether that class of action may proceed under the current conditions.
Human approval becomes a workflow state rather than a Slack conversation whose result is difficult to audit.
AWS Step Functions supports workflows that pause for human approval and continue only after receiving the approval response. That makes approval an explicit state transition rather than an informal side channel.
The final three stages are just as important. Execution returning zero is not recovery. The control plane has to verify the application independently and either close the incident or stop automation.

Authorization belongs outside the agent

The policy decision should not be another LLM call.
Amazon Verified Permissions provides an  IsAuthorized  API that evaluates principal, action, resource, entities, and request context against policies in a policy store. Its response is an  Allow  or  Deny  decision and can include the policies that determined the result.
For remediation, the authorization request can carry context that has already been independently collected:
principal
AI-Remediator
action
RollbackDeployment
resource
prod-eks/payments/payments-api
context
environment = production
changeWindowOpen = true
statefulMigrationDetected = false
previousReplicaSetHealthy = true
scope = single-deployment
That is more useful than asking:
How confident are you that this rollback is safe?
A language model can provide confidence. It cannot make an unavailable rollback target suddenly exist.
I would also avoid collapsing all risk into one magic number unless the organization can explain exactly how the score is derived.
The stronger policy is usually categorical:
Known operation?
Required evidence present?
Reversible?
Scope bounded?
Change allowed now?
Executor permitted?
If any mandatory condition fails, authorization stops.
That makes the control plane easier to defend during an incident review because an engineer can reconstruct exactly why a change was or was not permitted.

The executor should be less powerful than the workflow

Even a correct policy does not justify broad EKS administration permissions.
The execution identity for  RollbackDeployment  should have exactly the Kubernetes privileges necessary to inspect and modify the target Deployment.
Amazon EKS access entries associate IAM identities with Kubernetes permissions and can either attach EKS access policies or map an IAM identity into Kubernetes groups for RBAC. That gives the remediation executor a clean way to authenticate without granting  cluster-admin .
A custom RBAC role might be limited to:
rules:
- apiGroups: ["apps"]
resources:
- deployments
- replicasets
verbs:
- get
- list
- patch
- apiGroups: [""]
resources:
- pods
- events
verbs:
- get
- list
Notice what is missing:
Secrets write
PersistentVolume delete
RoleBinding modification
ClusterRole modification
Namespace delete
Node administration
That omission is the safety property.
If policy logic contains a bug, the executor still cannot perform operations outside its Kubernetes authorization boundary.
For controlled execution, Systems Manager can provide another useful separation. Automation runbooks define controlled operational procedures, and Systems Manager Run Command can forward command output and errors into CloudWatch Logs for later investigation and audit. AWS also provides EKS-specific Systems Manager Automation runbooks for operational tasks and troubleshooting.
The agent proposes an operation.
The policy authorizes an operation class.
The runbook defines how that operation is performed.
Those should remain separate responsibilities.

Fault injection should test the policy, not just Kubernetes

A small failure is enough to test the boundary.
Start with an application listening on:
5678
The Deployment has readiness and liveness probes targeting the same port.
Now deploy a faulty revision where the application listens on:
9999
but the probes still target:
5678
The resulting investigation should not need sophisticated AI reasoning.
The evidence is concrete:
new Deployment revision exists
container listener changed
readiness probe still targets old port
liveness probe still targets old port
probe connection fails
new revision becomes unhealthy
previous ReplicaSet remains available
The interesting question is what the remediation layer does with those facts.
Suppose the proposed action is:
RollbackDeployment
Before authorizing it, the workflow rechecks the live state:
kubectl get deployment payments-api -n payments
kubectl get rs -n payments
kubectl get pods -n payments
kubectl get events -n payments --sort-by=.lastTimestamp
A publication or evidence pack should preserve the actual output from these commands if an observed restart count, ReplicaSet count, or Ready replica value is later claimed.
Do not replace missing experimental evidence with plausible numbers.
That distinction is important because the control plane is supposed to be evidence-driven. The documentation of the experiment should be held to the same standard.
The authorization logic can now reason over observed conditions:
Action known? yes
Failure correlated with new revision? yes
Previous revision available? yes
Stateful migration detected? no
Scope limited to one Deployment? yes
Change window allows remediation? yes
That is a defensible automatic rollback candidate for a stateless service.
Change one fact:
stateful migration detected = true
and the same Kubernetes symptoms should no longer produce the same decision.
The workflow should require human approval or escalate because the previous application version may no longer be compatible with current state.
The value of the model is not recognizing a broken probe.
It is recognizing that identical symptoms can require different authorization decisions.

Verify recovery independently

Suppose the rollback command completes successfully.
That proves only that Kubernetes accepted the change.
It does not prove the customer path recovered.
Post-change verification should use evidence independent of the executor.
CloudWatch composite alarms can combine several underlying alarms using Boolean logic and can reduce alert noise by representing a broader application-health condition.
For example:
ALARM(ProductionUnhealthy) =
ALARM(High5xx)
OR ALARM(P99LatencyExceeded)
OR ALARM(ReadyReplicaDeficit)
After rollback, the workflow waits through a stabilization interval and checks both Kubernetes and application telemetry:
Deployment rollout complete?
Expected replicas Ready?
Restart rate stable?
Application health alarm OK?
Critical transaction succeeding?
If Kubernetes is healthy but the CloudWatch service alarm remains in  ALARM , automation stops.
It should not immediately ask the AI agent for a more aggressive remediation.
That would create a dangerous feedback loop:
action fails
‚Üì
agent proposes stronger action
‚Üì
action fails
‚Üì
agent proposes stronger action
A bounded remediation system needs a defined point where uncertainty returns control to an engineer.

Investigation cost is another boundary

AWS DevOps Agent investigations are currently priced at $0.0083 per active agent-second, which equals approximately $0.498 per active agent-minute. Additional AWS services used during investigations, such as CloudWatch queries, can create their own charges.
A policy can therefore define modeled investigation ceilings.
For example:
Per incident:
15 active agent-minutes
Daily:
240 active agent-minutes
Using the current published rate:
15 √ó $0.498 = $7.47
240 √ó $0.498 = $119.52
Those numbers are modeled maximums, not measured savings and not predictions of average incident cost.
That wording matters.
The purpose of the ceiling is to prevent a noisy alarm, duplicated ticket, or poorly bounded investigation from consuming unlimited agent time.
AWS DevOps Agent already performs incident correlation during triage so related incidents can be linked rather than investigated independently.
The remediation control plane should apply the same principle to execution.
One incident plus one workload plus one action should produce one active remediation identity.

What should remain manual

A mature automation platform is defined partly by what it refuses to automate.
I would keep these outside autonomous execution by default:
IAM modifications
cluster-wide RBAC changes
PersistentVolume deletion
database schema changes
data mutation or repair
cross-cluster destructive operations
unknown remediation commands
changes whose rollback path is unverified
This is not because automation cannot technically perform them.
It is because the evidence needed to establish safety is substantially more complicated than it is for restarting one stateless workload or rolling back one known Deployment.
The boundary can evolve as evidence improves.
It should not expand simply because the agent has become better at explaining why it wants permission.
The architecture works when investigation intelligence and production authority remain intentionally unequal.
AWS DevOps Agent can gather context and produce a mitigation plan. A structured contract converts that plan into something deterministic systems can evaluate. Policy decides whether the operation is permitted. EKS authorization limits the executor. A reviewed runbook performs the change. CloudWatch and Kubernetes independently verify the result.
If any one of those boundaries disagrees with the recommendation, automation stops.
That is not a limitation of AI-assisted remediation.
For production EKS systems, that is the feature that makes it usable.
Add the following section near the end, preferably before “What should remain manual.”

Validation on Amazon EKS

I validated the failure path on an Amazon EKS 1.36 cluster using a two-replica Deployment. The known-good revision listened on port  5678 , with readiness and liveness probes targeting the same port. Both replicas reached Ready state with zero container restarts before fault injection.
I then introduced a faulty revision that moved the application listener to port  9999  while leaving both probes configured for port  5678 . Kubernetes created a Pod for the new ReplicaSet, but the Pod never became Ready. Repeated liveness-probe failures drove its restart count from  1  to  4 .
The failed rollout did not require the known-good replicas to be replaced. The previous ReplicaSet remained at two Ready replicas, and both healthy Pods retained zero restarts throughout the validation window.
EndpointSlice state reflected the same boundary. The known-good endpoints remained  ready=true  and  serving=true , while the faulty revision was marked  ready=false  and  serving=false .
I ran a separate transaction probe against the Kubernetes Service during the failure and rollback window. It completed 120 HTTP requests successfully with zero failures. This provided application-level evidence that the unhealthy revision was excluded from serving traffic while the known-good revision continued handling requests.
After rollback, the Deployment returned to  2/2  Ready replicas and the faulty ReplicaSet scaled to zero.
I also exercised the deterministic authorization layer against three scenarios:
bounded-stateless-rollback: ALLOW
rollback-after-stateful-migration: ESCALATE
iam-change: DENY
The distinction is intentional. The same rollback action that is acceptable for a bounded stateless failure becomes an escalation when the evidence indicates state compatibility may have changed. An IAM modification remains outside the autonomous execution boundary.
The manifests, policy example, architecture model, and sanitized validation results are available in the accompanying GitHub repository:
GitHub:  https://github.com/PradeepKandepaneni/bounded-ai-remediation-eks 
The repository intentionally excludes raw AWS account and environment metadata while preserving the artifacts required to reproduce the technical pattern. The public repo currently contains the manifests, deterministic policy example, architecture, sanitized validation summary, and publication references.

Series: Engineering Safe AI-Assisted Remediation on Amazon EKS (2 articles)

  1. 2
    From AI Recommendation to Bounded EKS Remediation: Validating a Risk-Gated Control Plane with Fault Injection This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article