AWS Builder Center

The Authorization Gap in AI Operations: Building a Policy-Enforced EKS Control Plane with Cedar, Step Functions, and Systems Manager

The hardest problem in AI-driven operations is not getting an agent to diagnose an incident. Modern models can correlate logs, metrics, deployment events, traces, Kubernetes state, and historical incidents well enough to produce plausible remediation proposals. The harder question begins one step later: Who decides whether the proposed action is actually allowed to touch production? That distinction matters because reasoning quality and operational authority are different properties.

DevOps Cloud Platform & Site Reliability Engineer | AWS Certified Solutions Architect – Professional



An agent can be highly accurate and still eventually make a bad decision. If that decision carries unrestricted production authority, model accuracy becomes a weak safety boundary. A better architecture assumes that recommendations can be wrong and constrains what happens when they are.
Telemetry + Cluster State
|
v
AI Investigator
|
v
Structured Remediation Proposal
|
v
Deterministic Evidence Collector
|
v
Policy + Risk Evaluation
| | |
v v v
AUTO APPROVAL DENY
| |
| Human Decision
| |
+----> Revalidation
|
v
Bounded Runbook
|
v
EKS Executor
|
v
Independent Verification
The important architectural decision is not which model sits at the top.
It is that the model does not own the bottom half.

Start with an executable contract

An operations agent should never hand an executor a natural-language instruction such as:
Fix the payments service.
There is almost nothing meaningful to authorize in that request.
Instead, require the agent to produce a typed remediation proposal:
{
"action": "RollbackDeployment",
"cluster": "prod-eks",
"namespace": "payments",
"workload": "payments-api",
"observedRevision": 42,
"targetRevision": 41,
"reason": "error_rate_regression",
"executionBounds": {
"timeoutSeconds": 180,
"maxUnavailable": 1
}
}
This document describes intent, not truth.
That distinction is critical.
The model may propose that revision 41 is the rollback target. It should not be trusted to assert that revision 41 is healthy, that no incompatible database migration occurred, or that revision 42 is still running when execution begins.
Those facts need to come from deterministic systems.
A separate evidence collector can enrich the request:
{
"deploymentGeneration": 108,
"currentRevision": 42,
"previousRevisionHealthy": true,
"statefulMigrationDetected": false,
"maintenanceFreeze": false,
"evidenceTimestamp": "2026-08-19T13:01:14Z"
}
This gives us the first hard boundary:
The agent proposes the action. Trusted infrastructure establishes the state under which that action may be considered.
Without that separation, policy enforcement becomes security theater. An agent capable of supplying both the request and the evidence used to authorize that request can effectively authorize itself.

Put authorization outside the agent

Amazon Verified Permissions is useful here because it externalizes authorization decisions into policies written in Cedar. The application asks whether a principal may perform an action against a resource in a particular context, and receives an authorization decision.
Conceptually, a rollback policy could resemble:
permit (
principal == RemediationActor::"eks-remediator",
action == RemediationAction::"RollbackDeployment",
resource is KubernetesWorkload
)
when {
context.previousRevisionHealthy == true &&
context.statefulMigrationDetected == false &&
context.maintenanceFreeze == false &&
context.evidenceAgeSeconds <= 30
};
The exact schema will depend on the implementation, but notice what is absent:
modelConfidence > 0.95
Confidence may be useful for deciding whether the system needs more investigation. It is a poor substitute for operational invariants.
The control plane should ask questions such as:
Is the target revision known?

Is it healthy?

Has persistent state changed?

Is the request still current?

Is this namespace eligible for autonomous remediation?

Does the executor have authority for this resource?

Is a change freeze active?
These are much stronger authorization signals than whether an LLM reports high confidence.
Cedar also supports explicit  forbid  policies and a default-deny evaluation model. A matching forbid overrides permits, and a request without an applicable permit is denied.
That makes certain boundaries straightforward:
forbid (
principal,
action,
resource
)
when {
context.production == true &&
context.modifiesPersistentData == true
};
Some operations should simply never enter an autonomous path.
Deleting persistent data, changing cluster-wide authorization, modifying identity infrastructure, or executing arbitrary shell commands are reasonable examples.

Do not turn Cedar into an incident workflow engine

There is an architectural trap here.
It is tempting to make the policy system return increasingly complicated outcomes:
ALLOW_AUTO
ALLOW_WITH_APPROVAL
ALLOW_IF_SRE
INVESTIGATE_MORE
ESCALATE_SEV1
I would avoid that.
Authorization and operational risk classification are related, but they are not the same responsibility.
Keep the classifier deterministic and separate:
Risk classifier
|
+--> AUTO
|
+--> HUMAN_APPROVAL
|
+--> ESCALATE
Then ask the authorization system whether the identified actor is permitted to execute the requested operation in that path.
This keeps Cedar focused on authorization rather than turning policy expressions into a hidden orchestration language.

Step Functions owns workflow state

Authorization tells us whether an operation may happen.
It does not solve what happens before or after the decision.
This is where AWS Step Functions fits naturally.
A remediation state machine could look like:
CollectEvidence
|
v
ValidateProposal
|
v
ClassifyRisk
/ | \
AUTO APPROVAL DENY
| |
| WaitForHuman
| |
+------+
|
v
RefreshEvidence
|
v
Reauthorize
|
v
ExecuteRunbook
|
v
VerifyRecovery
/ \
SUCCESS FAILED
|
v
ESCALATE
For human approval, Step Functions supports the callback-with-task-token pattern. A workflow can pause until an external process returns the task token using  SendTaskSuccess  or  SendTaskFailure . Human approval is a natural use case for this pattern.
The critical state in that diagram, however, is not  WaitForHuman .
It is:
RefreshEvidence

Human approval expires

Suppose the system proposes:
Rollback payments-api from revision 42 to revision 41
An SRE reviews the evidence and approves the action.
Eight minutes pass before execution.
During those eight minutes, the deployment pipeline releases revision 43.
The original approval is now describing a production state that no longer exists.
Executing the approved rollback blindly would mean modifying revision 43 based on analysis of revision 42.
This is a time-of-check versus time-of-use problem.
The solution is simple conceptually:
Approval authorizes intent. It does not freeze production state.
Immediately before execution, recollect the critical evidence and evaluate authorization again.
The remediation contract should also carry preconditions:
{
"expectedGeneration": 108,
"expectedRevision": 42,
"targetRevision": 41
}
The executor reads the Deployment immediately before mutation.
If it finds:
expected generation: 108
observed generation: 109
execution stops.
Do not ask the model whether generation 109 is "probably okay."
Return the workflow to investigation.
Once the underlying state changes, the authorized operation is no longer necessarily the same operation.

Make execution boring

The execution layer should be deliberately less intelligent than the investigation layer.
AWS Systems Manager Automation provides runbooks consisting of defined parameters and sequential actions. Custom runbooks can execute scripts, invoke AWS APIs, call Lambda functions, and compose other automation actions.
That allows us to expose something like:
Runbook:
EKS-BoundedDeploymentRollback

Inputs:
Cluster
Namespace
Deployment
ExpectedGeneration
ExpectedCurrentRevision
TargetRevision
TimeoutSeconds
Instead of:
execute(command)
That difference is one of the strongest controls in the architecture.
The AI cannot decide to append:
kubectl delete namespace payments
because there is no arbitrary command parameter.
The execution vocabulary should be intentionally small:
RollbackDeployment

RestartSelectedStatelessPods

ScaleDeploymentWithinBounds

CordonNode

RemoveKnownBadPod
Each action gets its own validation, permissions, preconditions, timeout, and verification behavior.
Systems Manager Automation can invoke a purpose-built Lambda executor through  aws:invokeLambdaFunction .
I would prefer this pattern over turning EKS worker nodes into generic administrative hosts using Run Command.
Systems Manager Run Command is designed to execute commands on managed nodes.
Technically, it could be used to run Kubernetes administration commands from a managed host. Architecturally, that creates a generic command channel precisely where we are trying to remove one.
A dedicated executor with a narrow API is easier to reason about.

Least privilege must exist below policy

Policy alone is not enough.
Imagine that a Cedar policy is accidentally changed and begins permitting an operation it should deny.
If the executor has cluster-admin privileges, the policy error immediately becomes a production capability.
The better model is:
Policy boundary
+
IAM boundary
+
EKS authentication boundary
+
Kubernetes RBAC boundary
+
Runbook parameter boundary
Amazon EKS access entries associate IAM principals with Kubernetes access. Permissions can be supplied through EKS access policies or by mapping the principal to Kubernetes groups and implementing permissions with Kubernetes RBAC.
For a tightly constrained remediation executor, custom Kubernetes RBAC is often attractive because the exact verbs and resources can be controlled.
For example:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: payments
name: bounded-remediator
rules:
- apiGroups: ["apps"]
resources:
- deployments
- deployments/scale
- replicasets
verbs:
- get
- list
- watch
- patch
Then bind the Kubernetes group associated with the executor's EKS access entry.
The executor should not receive access to:
Secrets
ClusterRoles
ClusterRoleBindings
PersistentVolumes
CustomResourceDefinitions
Namespaces
ServiceAccounts
unless a specific remediation capability genuinely requires it.
The principle is stronger than least privilege:
Make prohibited actions technically impossible from the remediation identity.

Do not let the executor declare success

There is another subtle boundary after execution.
Suppose Kubernetes accepts:
PATCH deployment/payments-api
and returns success.
Did remediation succeed?
We know only that the API server accepted the mutation.
We do not know whether:
Pods became Ready

Service endpoints recovered

application errors dropped

latency normalized

dependencies remained healthy

the original SLI recovered

customer transactions succeeded
Execution success and recovery success are different states.
So verification should belong to a different component.
Executor
|
| mutation
v
Kubernetes
|
v
Independent Verifier
The verifier should preferably operate with read-only permissions and consume both Kubernetes state and service-level telemetry.
For a rollback, it might evaluate:
Deployment available replicas
ReplicaSet convergence
Pod readiness
EndpointSlice endpoints
HTTP synthetic check
5xx rate
p95 latency
incident-specific SLI
The actual criteria must be workload-specific.
A batch processor cannot be validated like an HTTP API. A Kafka consumer may require lag and processing-rate checks. A stateful service may require consistency or replication signals.
This is why "AI remediation platform" is often too broad a product boundary.
Safe remediation is easier when capabilities are designed around known workload classes and explicit recovery contracts.

What happens when verification fails?

The obvious answer is:
Undo the remediation.
That answer is dangerous.
The remediation may have changed state. A second rollback may compound the failure. The system may no longer satisfy the preconditions under which the first action was approved.
Automatic reversal should therefore be treated as another remediation request, with its own policy and preconditions.
A safer default is often:
Verification failed
|
v
Freeze further automation
|
v
Capture current state
|
v
Escalate to human
Autonomous systems need a reliable way to stop being autonomous.

Design failure behavior before success behavior

The control plane itself will fail.
That needs explicit semantics.
If Verified Permissions cannot return a decision:
DENY
not:
continue because this is an emergency
If the evidence collector cannot determine whether a stateful migration occurred:
UNKNOWN
must not silently become:
FALSE
If human approval times out, block or escalate.
If the runbook receives an unexpected parameter:
ABORT
If production state changes between authorization and execution:
REINVESTIGATE
If verification is inconclusive:
ESCALATE
These decisions may reduce the percentage of incidents handled autonomously.
That is acceptable.
Automation coverage is not the primary reliability metric for an autonomous remediation system.
The more meaningful question is:
How much production authority can we safely delegate while keeping the worst credible failure bounded?

The architectural lesson

The interesting part of AI operations is no longer whether an LLM can call  kubectl .
It can.
The harder engineering problem is building a system where a bad recommendation cannot automatically become an unbounded production change.
That requires separating responsibilities:
AI
Reason about the incident
Propose an action

Evidence collector
Establish current operational truth

Risk classifier
Determine the execution path

Amazon Verified Permissions + Cedar
Authorize the exact operation

AWS Step Functions
Own workflow state and approval boundaries

AWS Systems Manager Automation
Expose bounded operational runbooks

EKS access entries + Kubernetes RBAC
Constrain technical capability

Independent verifier
Determine whether the system recovered
None of those mechanisms makes the AI correct.
That is precisely why they matter.
A mature production architecture assumes that eventually the model will be wrong, the evidence will become stale, a dependency will behave unexpectedly, or a policy will contain a defect.
The architecture should make those failures survivable.
The goal is not an AI agent powerful enough to operate production without humans.
The goal is an operations control plane disciplined enough that increasingly capable agents can participate without inheriting unrestricted production authority.
That is a much more useful definition of autonomous operations.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article