AWS Builder Center

Building a Production-Safe AI Remediation Firewall for Amazon EKS

Risk-gated AIOps with AWS DevOps Agent, Amazon Verified Permissions, AWS Systems Manager, AWS Step Functions, and Amazon CloudWatch

Series: Engineering Safe AI-Assisted Remediation on Amazon EKS (2 articles)

  1. 1
    Building a Production-Safe AI Remediation Firewall for Amazon EKS This article
DevOps Cloud Platform & Site Reliability Engineer | AWS Certified Solutions Architect – Professional
AI can reduce the time required to investigate incidents, correlate telemetry, and identify probable root causes. But giving an AI system unrestricted production access creates a new reliability risk: the remediation platform itself can become the cause of the next outage.
This article presents an AI Remediation Firewall for workloads running on Amazon Elastic Kubernetes Service (Amazon EKS).
The design follows one principle:
AI proposes. Deterministic policy authorises. Controlled automation executes. Independent telemetry verifies.
The architecture uses:
  • AWS DevOps Agent for investigation and root-cause analysis
  • A structured remediation proposal as the handoff contract
  • AWS Step Functions for workflow control
  • Amazon Verified Permissions for deterministic authorisation
  • AWS Systems Manager Automation for controlled execution
  • Amazon CloudWatch Application Signals and CloudWatch Synthetics for independent verification
  • Kubernetes RBAC and IAM to restrict blast radius
  • Progressive delivery and rollback to contain failed changes
The objective is not unrestricted autonomous operations. It is bounded autonomy.



The production incident

Consider a digital commerce platform running on Amazon EKS.
The application contains:
  • API gateway
  • Checkout service
  • Inventory service
  • Payment service
  • Pricing service
  • Amazon Aurora PostgreSQL
  • Amazon ElastiCache for Redis
  • Amazon SQS
  • Application Load Balancer
  • CloudWatch Application Signals
  • CloudWatch Synthetics
A deployment changes the checkout configuration:
database:
poolSize: 80
connectionTimeoutMilliseconds: 500

retry:
maximumAttempts: 8
initialBackoffMilliseconds: 25

The previous configuration was:
database:
poolSize: 30
connectionTimeoutMilliseconds: 2000

retry:
maximumAttempts: 3
initialBackoffMilliseconds: 250

Within minutes:
  • Checkout p99 latency increases from about 220 ms to more than four seconds
  • HTTP 5xx errors increase
  • Aurora connections approach the configured maximum
  • SQS queue depth grows
  • HPA activity increases
  • CPU usage rises across checkout and payment services
  • Pods remain  Running 
  • Liveness probes continue to pass
This is a difficult incident because infrastructure health looks acceptable at a superficial level.
Restarting pods provides only temporary relief. Increasing replicas creates more database connections and makes the problem worse.
AWS DevOps Agent investigates:
  • CloudWatch metrics and logs
  • Application Signals topology
  • Kubernetes events
  • Deployment history
  • CloudTrail configuration changes
  • Aurora metrics
  • SQS metrics
  • Approved runbooks and previous incidents
It identifies the latest configuration change as the probable trigger and proposes:
Roll back the checkout service to the previous known-good revision.
That recommendation may be correct, but it is not yet authorised.



Why an AI recommendation must be treated as untrusted input

An AI-generated remediation can be technically plausible and still be unsafe because:
  • The root-cause analysis may be incomplete
  • Multiple deployments may be active
  • The target may have changed after the investigation started
  • The rollback revision may contain a security defect
  • The proposed blast radius may be too large
  • The service may already be recovering
  • The action may affect a critical dependency
  • The rollback artefact may no longer exist
  • The recommendation may be based on stale telemetry
The AI system must not decide:
  1. Its own permissions
  1. Whether its evidence is sufficient
  1. Whether the blast radius is acceptable
  1. Whether human approval is required
  1. Whether the remediation succeeded
Those decisions belong outside the AI reasoning boundary.



Reference architecture

CloudWatch SLO alarm or incident alert
|
v
AWS DevOps Agent
|
Investigation and RCA
|
v
Remediation Proposal Adapter
|
v
Amazon EventBridge
|
v
AWS Step Functions
|
+---------+----------+
| |
v v
Live-context collector Schema validator
| |
+---------+----------+
|
v
Deterministic risk engine
|
v
Amazon Verified Permissions
|
+-----------+-----------+
| | |
v v v
Auto-execute Approval Deny
| |
+-----+-----+
|
v
AWS Systems Manager Automation
|
v
Restricted EKS executor
|
v
Canary or bounded rollout
|
v
Application Signals and Synthetics
|
+-----+------+
| |
v v
Promote Roll back

The trust boundaries are:
  1. Investigation boundary AI gathers evidence and proposes an action.
  1. Authorisation boundary Deterministic logic decides whether that action is allowed.
  1. Execution boundary A restricted runbook performs one allowlisted operation.
  1. Verification boundary Independent telemetry decides whether recovery occurred.



1. Keep diagnostic access read-only

The investigation layer should be able to:
  • Read CloudWatch metrics and logs
  • Read Kubernetes events
  • Read deployment metadata
  • Read CloudTrail changes
  • Read service topology
  • Read approved incident history
  • Invoke narrow diagnostic tools
It should not have broad write access to:
  • Kubernetes resources
  • IAM
  • Security groups
  • Databases
  • Secrets
  • CI/CD pipelines
  • DNS
Expose bounded tools such as:
get_kubernetes_events(namespace, workload)
get_deployment_revision(namespace, deployment)
collect_eks_node_diagnostics(instance_id)
query_application_errors(service, start_time, end_time)
compare_configuration(current_revision, previous_revision)

Do not expose:
execute_shell(command)

A generic shell endpoint turns a controlled diagnostic integration into remote code execution.



2. Require a structured remediation proposal

Do not send free-form AI text to an executor.
Use a versioned contract:
{
"schemaVersion": "1.0",
"proposalId": "rp-01JZ8A17R2K5",
"incidentId": "inc-checkout-0017",
"createdAt": "2026-07-28T20:12:31Z",
"expiresAt": "2026-07-28T20:17:31Z",
"target": {
"environment": "production",
"cluster": "commerce-prod",
"region": "us-east-1",
"namespace": "checkout",
"resourceType": "ArgoRollout",
"resourceName": "checkout-api"
},
"action": {
"type": "ROLLBACK_DEPLOYMENT",
"parameters": {
"targetRevision": 41
},
"requestedBlastRadiusPercent": 5
},
"evidence": [
{
"source": "APPLICATION_SIGNALS",
"type": "LATENCY_INCREASE"
},
{
"source": "CLOUDTRAIL",
"type": "RECENT_CHANGE"
},
{
"source": "KUBERNETES",
"type": "CONFIGURATION_CHANGE"
},
{
"source": "AURORA",
"type": "RESOURCE_SATURATION"
}
],
"rollback": {
"available": true,
"knownGoodRevision": 41
},
"agentAssessment": {
"confidencePercent": 92,
"summary": "Revision 42 introduced retry amplification"
}
}

Reject proposals containing:
  • Unknown actions
  • Wildcard targets
  • Free-form commands
  • Missing rollback data
  • Expired timestamps
  • Unregistered resources
  • Unknown revisions
  • Excessive blast radius
  • Insufficient evidence
The proposal contains intent, not executable commands.
Accept:
{
"action": {
"type": "ROLLBACK_DEPLOYMENT",
"parameters": {
"targetRevision": 41
}
}
}

Reject:
{
"command": "kubectl delete pods --all --all-namespaces"
}




3. Separate action risk from incident urgency

A critical incident does not make a dangerous remediation safe.
Calculate two independent values.

Action risk

Action Risk =
Action Severity × 0.25
+ Resource Criticality × 0.20
+ Blast Radius × 0.20
+ Evidence Uncertainty × 0.15
+ Change Collision × 0.10
+ Rollback Weakness × 0.10

Example action severity:
Action
Severity
Restart one stateless pod
10
Scale within approved limits
20
Progressive rollback
30
Modify HPA policy
45
Change database parameters
70
Modify routing
85
Modify IAM
95
Delete data
100
Example resource criticality:
Resource
Criticality
Development service
10
Internal production service
35
Customer-facing API
70
Authentication service
90
Payment platform
100

Incident urgency

Incident Urgency =
SLO Burn Rate × 0.40
+ Customer Impact × 0.35
+ Incident Duration × 0.25

Urgency may shorten approval time or escalate responders. It must not lower the action-risk threshold.



4. Do not trust AI confidence directly

The control plane should calculate validated confidence from evidence quality.
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any

ALLOWLISTED_ACTIONS = {
"RESTART_SINGLE_POD",
"SCALE_WITHIN_LIMIT",
"ROLLBACK_DEPLOYMENT",
}

@dataclass(frozen=True)
class RiskResult:
risk_score: int
validated_confidence: int
route: str
reasons: list[str]

def clamp(value: int) -> int:
return max(0, min(100, value))

def validated_confidence(proposal: dict[str, Any]) -> int:
evidence = proposal.get("evidence", [])
sources = {item.get("source") for item in evidence if item.get("source")}
types = {item.get("type") for item in evidence if item.get("type")}

agent_confidence = clamp(
int(proposal.get("agentAssessment", {}).get("confidencePercent", 0))
)

has_change = "RECENT_CHANGE" in types
has_telemetry = bool(
{"LATENCY_INCREASE", "ERROR_RATE_INCREASE", "RESOURCE_SATURATION"}
& types
)

if len(sources) >= 3 and has_change and has_telemetry:
cap = 95
elif len(sources) >= 2 and has_telemetry:
cap = 75
else:
cap = 50

return min(agent_confidence, cap)

def calculate_action_risk(
proposal: dict[str, Any],
context: dict[str, Any],
) -> RiskResult:
action = proposal["action"]["type"]

if action not in ALLOWLISTED_ACTIONS:
return RiskResult(100, 0, "DENY", ["Action is not allowlisted"])

expires_at = datetime.fromisoformat(
proposal["expiresAt"].replace("Z", "+00:00")
)

if datetime.now(timezone.utc) >= expires_at:
return RiskResult(100, 0, "DENY", ["Proposal expired"])

confidence = validated_confidence(proposal)
uncertainty = 100 - confidence

risk = round(
clamp(context["actionSeverity"]) * 0.25
+ clamp(context["resourceCriticality"]) * 0.20
+ clamp(context["blastRadiusPercent"]) * 0.20
+ clamp(uncertainty) * 0.15
+ (100 if context["concurrentChange"] else 0) * 0.10
+ (0 if context["rollbackAvailable"] else 100) * 0.10
)

reasons = []

if not context["rollbackAvailable"]:
risk = max(risk, 70)
reasons.append("No verified rollback point")

if context["concurrentChange"]:
risk = max(risk, 65)
reasons.append("Concurrent change detected")

if context["blastRadiusPercent"] > 25:
risk = max(risk, 65)
reasons.append("Blast radius exceeds 25 percent")

if confidence < 70:
risk = max(risk, 61)
reasons.append("Evidence confidence below threshold")

if risk <= 25 and confidence >= 90:
route = "AUTO_EXECUTE"
elif risk <= 60:
route = "REQUIRE_APPROVAL"
else:
route = "DENY"

return RiskResult(clamp(risk), confidence, route, reasons)

The weights are organisational policy, not universal values.
The critical properties are:
  • Inputs are bounded
  • Logic is deterministic
  • Algorithms are versioned
  • Decisions are reproducible
  • Every factor is recorded
  • AI cannot change thresholds
  • Failure defaults to denial



5. Refresh production context before authorisation

Do not authorise using only the data captured during investigation.
Immediately before execution, verify:
  • The incident is still active
  • The target revision is still deployed
  • No other production change is running
  • The previous revision still exists
  • The previous revision is security-compliant
  • The workload has not already recovered
  • The target resource still maps to the same owner and criticality
  • The SLO is still burning
  • The proposal has not expired
This prevents time-of-check/time-of-use failures.
A five-minute proposal expiration is a reasonable starting point for autonomous actions.



6. Authorise with Amazon Verified Permissions

Model the remediation controller as an application:
  • Principal: remediation orchestrator
  • Action: auto-execute or request-approval
  • Resource: production workload
  • Context: risk, confidence, blast radius, rollback status, action type
Evaluate  AutoExecute  first. If denied, evaluate  RequestApproval . If both are denied, block the proposal.
Automatic execution policy:
permit (
principal == RemediationControl::Service::"orchestrator",
action == RemediationControl::Action::"AutoExecute",
resource
)
when {
context.riskScore <= 25 &&
context.validatedConfidence >= 90 &&
context.blastRadiusPercent <= 10 &&
context.rollbackAvailable &&
!context.concurrentChange &&
(
context.actionType == "RESTART_SINGLE_POD" ||
context.actionType == "SCALE_WITHIN_LIMIT" ||
context.actionType == "ROLLBACK_DEPLOYMENT"
)
};

Approval policy:
permit (
principal == RemediationControl::Service::"orchestrator",
action == RemediationControl::Action::"RequestApproval",
resource
)
when {
context.riskScore <= 60 &&
context.validatedConfidence >= 70 &&
context.blastRadiusPercent <= 25 &&
context.rollbackAvailable &&
!context.concurrentChange
};

Explicitly forbidden actions:
forbid (
principal,
action,
resource
)
when {
context.actionType == "MODIFY_IAM" ||
context.actionType == "DELETE_DATA" ||
context.actionType == "DISABLE_AUDIT_LOGGING" ||
context.actionType == "OPEN_PUBLIC_NETWORK_ACCESS"
};

The authorisation request can be submitted from Lambda:
import boto3

client = boto3.client("verifiedpermissions")

def is_authorized(
policy_store_id: str,
route_action: str,
workload_id: str,
context: dict,
) -> bool:
response = client.is_authorized(
policyStoreId=policy_store_id,
principal={
"entityType": "RemediationControl::Service",
"entityId": "orchestrator",
},
action={
"actionType": "RemediationControl::Action",
"actionId": route_action,
},
resource={
"entityType": "RemediationControl::Workload",
"entityId": workload_id,
},
context={
"contextMap": {
"riskScore": {"long": context["riskScore"]},
"validatedConfidence": {
"long": context["validatedConfidence"]
},
"blastRadiusPercent": {
"long": context["blastRadiusPercent"]
},
"rollbackAvailable": {
"boolean": context["rollbackAvailable"]
},
"concurrentChange": {
"boolean": context["concurrentChange"]
},
"actionType": {"string": context["actionType"]},
}
},
)

return response["decision"] == "ALLOW"

Store with every decision:
  • Risk-engine version
  • Policy-store ID
  • Policy version
  • Risk score
  • Validated confidence
  • Decision route
  • Determining policy IDs



7. Orchestrate with AWS Step Functions

The state machine should coordinate:
  1. Proposal validation
  1. Expiration check
  1. Live-context collection
  1. Risk calculation
  1. Verified Permissions evaluation
  1. Human approval where required
  1. Runbook execution
  1. Canary verification
  1. Promotion or rollback
  1. Audit completion
A simplified workflow:
{
"StartAt": "ValidateProposal",
"States": {
"ValidateProposal": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Next": "CollectLiveContext"
},
"CollectLiveContext": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Next": "CalculateRisk"
},
"CalculateRisk": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Next": "AuthoriseRoute"
},
"AuthoriseRoute": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Next": "RouteDecision"
},
"RouteDecision": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.decision.route",
"StringEquals": "AUTO_EXECUTE",
"Next": "ExecuteRunbook"
},
{
"Variable": "$.decision.route",
"StringEquals": "REQUIRE_APPROVAL",
"Next": "ApprovalRunbook"
}
],
"Default": "Denied"
},
"ExecuteRunbook": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Next": "VerifyCanary"
},
"ApprovalRunbook": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Next": "VerifyCanary"
},
"VerifyCanary": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Next": "VerificationDecision"
},
"VerificationDecision": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.verification.passed",
"BooleanEquals": true,
"Next": "Promote"
}
],
"Default": "Rollback"
},
"Promote": {
"Type": "Succeed"
},
"Rollback": {
"Type": "Fail"
},
"Denied": {
"Type": "Succeed"
}
}
}

Production implementations also need:
  • Retries with bounded backoff
  • Catch handlers
  • Idempotency
  • Dead-letter handling
  • Timeouts
  • Duplicate suppression
  • Per-workload locking
  • Compensating actions
  • Audit-write failure handling



8. Execute only allowlisted runbooks

Map each action to a versioned Systems Manager runbook.
RUNBOOK_CATALOG = {
"RESTART_SINGLE_POD": {
"document": "AIOps-RestartSinglePod-v3",
"maximumBlastRadius": 10,
},
"SCALE_WITHIN_LIMIT": {
"document": "AIOps-ScaleDeployment-v4",
"maximumBlastRadius": 20,
},
"ROLLBACK_DEPLOYMENT": {
"document": "AIOps-RollbackDeployment-v7",
"maximumBlastRadius": 25,
},
}

The AI selects intent from the catalogue. It does not generate commands.
Example runbook:
description: Roll back an approved EKS workload
schemaVersion: '0.3'

assumeRole: '{{ AutomationAssumeRole }}'

parameters:
AutomationAssumeRole:
type: AWS::IAM::Role::Arn

ExecutorFunctionName:
type: String

ClusterName:
type: String

Namespace:
type: String

WorkloadName:
type: String

TargetRevision:
type: Integer

TrafficPercent:
type: Integer
allowedValues:
- 5
- 10
- 25

ProposalId:
type: String

mainSteps:
- name: ExecuteRestrictedRollback
action: aws:invokeLambdaFunction
timeoutSeconds: 120
maxAttempts: 1
onFailure: Abort
inputs:
FunctionName: '{{ ExecutorFunctionName }}'
InvocationType: RequestResponse
Payload: >
{
"operation": "ROLLBACK_DEPLOYMENT",
"cluster": "{{ ClusterName }}",
"namespace": "{{ Namespace }}",
"workload": "{{ WorkloadName }}",
"targetRevision": {{ TargetRevision }},
"trafficPercent": {{ TrafficPercent }},
"proposalId": "{{ ProposalId }}"
}

For medium-risk actions, add an  aws:approve  step before invoking the child runbook.
High-risk actions such as IAM changes, data deletion, network exposure, or disabling audit controls should remain outside the autonomous remediation catalogue.



9. Restrict EKS access

Use a dedicated IAM role and EKS access entry for the executor.
Do not grant  cluster-admin .
Map the role to a Kubernetes group:
aiops-remediation-executor

Namespace-scoped Role:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: aiops-remediation-executor
namespace: checkout
rules:
- apiGroups:
- apps
- argoproj.io
resources:
- deployments
- rollouts
verbs:
- get
- list
- watch
- patch
- update

- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch

RoleBinding:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: aiops-remediation-executor
namespace: checkout
subjects:
- kind: Group
name: aiops-remediation-executor
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: aiops-remediation-executor
apiGroup: rbac.authorization.k8s.io

The executor must not be able to:
  • Read Secrets
  • Modify ClusterRoles
  • Create privileged pods
  • Execute commands in arbitrary containers
  • Modify other namespaces
  • Delete namespaces
  • Change its own IAM access
  • Modify EKS access entries
For stronger isolation, use separate executor roles for each action category.



10. Limit blast radius with progressive delivery

An approved action should not immediately affect every replica.
Start with:
Initial traffic: 5%
Observation window: 5 minutes
Required successful synthetic runs: 3
Maximum 5xx rate: 1%
Maximum p99 latency: 750 ms
Required business validation success: 100%

Stop promotion when:
  • Error rate rises
  • Latency exceeds the threshold
  • New Kubernetes errors appear
  • Database pressure worsens
  • SLO burn rate increases
  • Business validation fails
  • Another production change begins
Promote in stages:
5% → 25% → 50% → 100%

Each stage requires independent verification.



11. Verify business behaviour, not only infrastructure health

A successful rollout does not prove recovery.
Pods may be healthy and return HTTP 200 while producing an incorrect checkout total.
CloudWatch Synthetics should validate both technical and business invariants.
const https = require("https");
const log = require("SyntheticsLogger");

const API_HOST = process.env.API_HOST;

function sendRequest(options, body) {
return new Promise((resolve, reject) => {
const request = https.request(options, response => {
let data = "";

response.on("data", chunk => {
data += chunk;
});

response.on("end", () => {
resolve({
statusCode: response.statusCode,
body: data
});
});
});

request.on("error", reject);
request.write(body);
request.end();
});
}

exports.handler = async () => {
const body = JSON.stringify({
customerId: "synthetic-customer",
items: [
{
sku: "SYNTHETIC-SKU-001",
quantity: 2
}
]
});

const options = {
hostname: API_HOST,
method: "POST",
path: "/api/v1/checkout/quote",
headers: {
"Content-Type": "application/json",
"Content-Length": Buffer.byteLength(body),
"X-Synthetic-Transaction": "true"
},
timeout: 5000
};

const started = Date.now();
const response = await sendRequest(options, body);
const duration = Date.now() - started;

if (response.statusCode !== 200) {
throw new Error(`Unexpected HTTP ${response.statusCode}`);
}

if (duration > 750) {
throw new Error(`Latency exceeded threshold: ${duration} ms`);
}

const payload = JSON.parse(response.body);

const subtotal = payload.lines.reduce(
(total, line) => total + line.unitPrice * line.quantity,
0
);

const expected = subtotal + payload.tax + payload.shipping;

if (Math.abs(payload.total - expected) > 0.01) {
throw new Error(
`Business invariant failed. Expected ${expected}, received ${payload.total}`
);
}

if (payload.currency !== "USD") {
throw new Error(`Unexpected currency: ${payload.currency}`);
}

log.info(`Synthetic checkout passed in ${duration} ms`);

return {
passed: true,
durationMilliseconds: duration
};
};

Synthetic transactions must not create real payments, shipments, inventory consumption, or customer notifications.
Use:
  • Reserved synthetic identities
  • Test inventory
  • Payment-provider test mode
  • Cleanup workflows
  • Explicit synthetic headers
  • Analytics filters



12. Verification contract

The verification layer should query authoritative telemetry directly.
{
"proposalId": "rp-01JZ8A17R2K5",
"observationWindowSeconds": 300,
"requirements": {
"minimumSuccessfulSyntheticRuns": 3,
"maximumErrorRatePercent": 1,
"maximumP99LatencyMilliseconds": 750,
"maximumDatabaseConnectionUtilisationPercent": 75,
"maximumSloBurnRate": 2,
"minimumHealthyReplicaPercent": 100
}
}

Example result:
{
"passed": true,
"observations": {
"successfulSyntheticRuns": 5,
"errorRatePercent": 0.4,
"p99LatencyMilliseconds": 420,
"databaseConnectionUtilisationPercent": 61,
"sloBurnRate": 0.8,
"healthyReplicaPercent": 100
}
}

Promotion requires every mandatory condition to pass.
Do not average away a critical business failure.



End-to-end execution

Step 1: Detect

Application Signals detects an elevated SLO burn rate and triggers an investigation.

Step 2: Investigate

AWS DevOps Agent correlates telemetry, configuration history, deployment metadata, and service dependencies.

Step 3: Propose

The proposal adapter creates:
Action: ROLLBACK_DEPLOYMENT
Target revision: 41
Initial traffic: 5%
Agent confidence: 92%
Rollback available: Yes


Step 4: Refresh context

The control plane confirms:
  • Revision 42 is still active
  • Revision 41 exists
  • No concurrent deployment is running
  • The incident is still active
  • The workload matches the inventory record

Step 5: Calculate risk

Factor
Value
Action severity
30
Resource criticality
70
Blast radius
5
Evidence uncertainty
8
Change collision
0
Rollback weakness
0
30 × 0.25 = 7.50
70 × 0.20 = 14.00
5 × 0.20 = 1.00
8 × 0.15 = 1.20
--------------------
Risk score = 23.70, rounded to 24

Validated confidence is 92%.

Step 6: Authorise

Verified Permissions allows automatic execution because:
  • Risk is at or below 25
  • Confidence is at least 90
  • Blast radius is at or below 10%
  • Rollback exists
  • No concurrent change is active
  • The action is allowlisted

Step 7: Execute

Systems Manager invokes the restricted rollback executor.
The executor can modify only:
commerce-prod
└── checkout namespace
└── checkout-api


Step 8: Verify

Five percent of traffic moves to revision 41.
CloudWatch Synthetics validates the checkout path. Application Signals validates latency, errors, and SLO burn rate. Database and queue metrics confirm recovery.

Step 9: Promote or roll back

If every gate passes, traffic moves progressively to 100%.
If any gate fails, the workflow restores the prior state and marks the remediation unsuccessful.

Step 10: Prevent recurrence

The incident generates preventive work:
  • Add retry budgets
  • Enforce exponential backoff
  • Validate connection-pool settings
  • Add configuration contract tests
  • Add load tests
  • Add deployment policies
  • Improve SLO coverage
A platform that repeatedly fixes the same incident without creating preventive engineering work is only automating toil.



Failure scenarios to test

1. Retry amplification

Inject excessive retries, minimal backoff, and aggressive database pool settings.
Expected result:
  • AI correlates deployment and database saturation
  • Scaling is not selected as the first response
  • Rollback is proposed
  • Connection pressure decreases after remediation

2. Silent network failure

Inject an invalid route or packet-filtering rule.
Expected result:
  • Pods may remain running
  • Service-to-service calls fail
  • Node diagnostics provide evidence
  • The system does not restart every workload blindly

3. HPA thrashing

Configure an aggressive target with insufficient stabilisation.
Expected result:
  • Oscillation is detected
  • Unbounded scaling is denied
  • HPA changes require approval
  • Verification uses a longer window

4. Partial secret rotation

Rotate credentials for only part of the workload.
Expected result:
  • Pod-level error distribution reveals the split
  • Secret changes remain outside autonomous execution
  • A predefined security runbook is used

5. Phantom success

Return HTTP 200 with an incorrect checkout total.
Expected result:
  • Infrastructure checks pass
  • Business synthetic fails
  • Promotion stops
  • Automatic rollback begins

6. Concurrent deployment

Start another deployment after the proposal is generated.
Expected result:
  • Context refresh detects the collision
  • Automatic execution is denied
  • The proposal becomes stale

7. Missing rollback artefact

Delete the previous image or revision.
Expected result:
  • Rollback weakness increases
  • Risk exceeds the threshold
  • The proposal is denied



Security controls

IAM separation

Use separate roles for:
  • Investigation
  • Proposal ingestion
  • Context collection
  • Risk evaluation
  • Policy evaluation
  • Orchestration
  • Runbook execution
  • EKS modification
  • Verification
  • Audit writing
Do not allow one role to modify the proposal, policy, executor, and audit record.

Immutable evidence

Store decisions and evidence in encrypted Amazon S3.
Consider:
  • Versioning
  • Object Lock where required
  • AWS KMS customer-managed keys
  • Restricted deletion
  • CloudTrail data events
  • Payload checksums

Idempotency

Create an idempotency key:
incidentId + target + actionType + targetRevision

If an identical action is:
  • Running: reject the duplicate
  • Successful: return the previous result
  • Failed: require a new proposal
  • Rolled back: block automatic retry

Concurrency locking

Use DynamoDB conditional writes to lock a workload:
LOCK#commerce-prod#checkout#checkout-api

Only one remediation workflow should modify a workload at a time.

Input integrity

Hash the proposal before authorisation.
Before execution, verify:
hash(authorised proposal) == hash(execution proposal)


Network controls

Where possible:
  • Use private subnets
  • Use a private EKS endpoint
  • Use VPC endpoints
  • Restrict outbound access
  • Restrict security-group egress
  • Avoid general internet access for the executor

Secrets

The proposal must not contain:
  • Passwords
  • Tokens
  • Private keys
  • Database credentials
  • Customer data
Pass references, not values.



Observability for the remediation platform

The remediation control plane needs its own SLOs.
Recommended metrics:
RemediationProposalReceived
RemediationProposalRejected
RemediationProposalExpired
RemediationRiskScore
RemediationAutoExecuted
RemediationApprovalRequested
RemediationDenied
RemediationExecutionSucceeded
RemediationExecutionFailed
RemediationVerificationPassed
RemediationVerificationFailed
RemediationRollbackSucceeded
RemediationRollbackFailed
RemediationDuplicateSuppressed
RemediationConcurrentChangeDetected

Recommended dimensions:
Environment
Cluster
Namespace
Service
ActionType
RiskBand
DecisionRoute
RunbookVersion
PolicyVersion
Outcome

Do not use proposal IDs as metric dimensions. Store them in logs.
Alarm on:
  • Executor failure
  • Rollback failure
  • Workflow timeout
  • Policy-evaluation failure
  • Approval timeout
  • Audit-write failure
  • Unexpected increases in automatic remediation
  • Repeated remediation of the same workload
When the control plane fails internally, deny or pause remediation. Do not bypass the control.



Metrics that prove value

Do not publish invented performance claims.
Measure:
Metric
Definition
Mean time to detect
Incident start to alert
Mean time to investigate
Alert to probable root cause
Mean time to authorise
Proposal to policy decision
Mean time to remediate
Authorised action to recovery
Mean time to verify
Execution to verified recovery
Unsafe-action prevention rate
Unsafe proposals blocked
Automatic-remediation success rate
Auto actions passing verification
False-remediation rate
Executed actions based on incorrect RCA
Rollback success rate
Failed remediations reversed
Change-collision prevention rate
Conflicting changes blocked
Cost per investigation
AI, telemetry, workflow, and execution cost
Repeat-incident rate
Recurrence of the same failure mode
Compare:
  1. Manual incident response
  1. AI-assisted investigation with manual execution
  1. AI-assisted investigation with risk-gated execution
The objective is:
Reduce recovery time without increasing the rate or blast radius of unsafe production changes.



Limitations

This pattern does not eliminate risk.
  • Root-cause analysis may be incomplete
  • Telemetry may be missing
  • Deterministic policies can still be wrong
  • Runbooks can contain defects
  • Human approval can delay recovery
  • Cross-account approval may require additional workflow design
  • Business validation may be difficult to automate safely
  • Break-glass actions still require human ownership
Do not expand autonomous permissions during an outage. Maintain a documented and audited break-glass process.



Production-readiness checklist

Investigation

  • Diagnostic access is read-only
  • MCP tools are narrowly scoped
  • Generic shell access is prohibited
  • Evidence sources are recorded
  • Output uses a versioned schema

Authorisation

  • Actions are allowlisted
  • Risk and urgency are separated
  • Live context is refreshed
  • Proposals expire
  • Default is deny
  • Forbidden actions have explicit deny policies
  • Policy changes use CI/CD
  • AI cannot change its own permissions

Execution

  • Runbooks are versioned
  • Parameters are validated
  • Free-form commands are rejected
  • IAM uses least privilege
  • Kubernetes RBAC is namespace-scoped
  • Secrets are inaccessible
  • Blast radius is bounded
  • Actions are idempotent
  • Concurrent changes are locked

Verification

  • Verification is independent
  • Infrastructure and business checks are included
  • Multiple successful observations are required
  • Failure triggers rollback
  • Rollback is verified
  • Success requires evidence

Audit

  • Proposal is immutable
  • Risk factors are stored
  • Policy and runbook versions are stored
  • Approvals are recorded
  • Execution logs are retained
  • Verification results are retained
  • Denied actions are searchable



Conclusion

AI can accelerate incident investigation, but investigation and production authority are different capabilities.
A system that proposes a change, authorises it, executes it, and judges its own success has collapsed too many trust boundaries.
The safer design separates responsibilities:
  • AWS DevOps Agent investigates
  • A structured proposal captures intent
  • A deterministic risk engine evaluates danger
  • Amazon Verified Permissions authorises
  • AWS Step Functions coordinates
  • Systems Manager executes allowlisted runbooks
  • IAM and Kubernetes RBAC contain access
  • Progressive delivery limits blast radius
  • Application Signals and Synthetics verify recovery
The result is not uncontrolled autonomy. It is bounded autonomy:
  • AI for investigation
  • Policy for authority
  • Runbooks for execution
  • SLOs and business tests for truth
  • Humans for ambiguity and high-risk decisions
AI proposes. Deterministic policy authorises. Controlled automation executes. Independent telemetry verifies.
That is the difference between an impressive AIOps demo and a production-safe reliability platform.



Disclaimer

The architecture, code samples, scoring weights, thresholds, IAM boundaries, Cedar policies, Kubernetes permissions, approval rules, and verification criteria in this article are reference examples.
Adapt and test them against your workload characteristics, security policies, compliance requirements, incident processes, service quotas, recovery objectives, and operational maturity.
Do not enable automatic production remediation until the actions have passed controlled failure injection, rollback testing, security review, and operational readiness exercises.

Series: Engineering Safe AI-Assisted Remediation on Amazon EKS (2 articles)

  1. 1
    Building a Production-Safe AI Remediation Firewall for Amazon EKS This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article