AWS Builder Center

The Authorization Gap in AI Operations: Building a Policy-Enforced EKS Control Plane with Cedar, Step Functions, and Systems Manager

This article designs a policy-enforced control plane for Amazon EKS that separates AI investigation from deterministic authorization, bounded execution, and independent verification using Amazon Verified Permissions with Cedar, AWS Step Functions, AWS Systems Manager Automation, EKS access controls, and Kubernetes RBAC. It also examines stale-state protection, human approval, rollback risk, and why successful execution is not the same as successful remediation.

DevOps Cloud Platform & Site Reliability Engineer | AWS Certified Solutions Architect – Professional



An AI agent correctly diagnosing an incident is not the same thing as being authorised to fix it.
That distinction becomes important the moment an operations agent moves beyond summarising logs and starts proposing changes to a production Kubernetes cluster. A recommendation such as “roll back  payments-api  to the previous revision” may be perfectly reasonable. It may also be unsafe because the previous revision is unhealthy, a database migration has already changed state, the target namespace is under a change freeze, or the evidence used by the agent is several minutes old.
The difficult engineering problem is therefore not generating the remediation. It is deciding whether the remediation is still safe when execution begins.
A useful control plane separates four responsibilities:
Investigation
↓
Recommendation
↓
Deterministic authorisation
↓
Bounded execution
↓
Independent verification

The AI participates in the first two. It should not own the remaining three.
This article develops a reference architecture for enforcing that separation on Amazon EKS using Amazon Verified Permissions and Cedar for policy decisions, AWS Step Functions for workflow orchestration and human approval, AWS Systems Manager Automation for bounded runbooks, and a narrowly privileged Kubernetes executor.
This is a reference architecture, not a claim of production deployment or measured operational improvement.

Start with a remediation contract, not a prompt

The most dangerous interface in an autonomous operations system is an unstructured instruction such as:
Fix payments-api.

There is nothing useful to authorise. The target, action, expected state, failure boundary and verification criteria are implicit.
A production control plane needs a structured remediation request:
{
"action": "RollbackDeployment",
"cluster": "prod-eks",
"namespace": "payments",
"workload": "payments-api",
"currentRevision": 42,
"targetRevision": 41,
"evidence": {
"currentRevisionHealthy": false,
"previousRevisionHealthy": true,
"statefulMigrationDetected": false,
"deploymentGeneration": 108,
"evidenceAgeSeconds": 14
},
"executionBounds": {
"maxUnavailable": 1,
"timeoutSeconds": 180
}
}

The AI may propose this document, but it should not be trusted to populate every field.
Fields describing live cluster state should come from deterministic collectors. If the model can simply declare  statefulMigrationDetected=false  or  previousRevisionHealthy=true , the policy layer is protecting data supplied by the component it is supposed to constrain.
That creates the first architectural rule:
Policy context must come from trusted evidence collection, not solely from model output.
The remediation request therefore becomes a join between an AI proposal and independently collected operational state.

Cedar is the policy decision point, not the workflow engine

Amazon Verified Permissions externalises application authorisation and evaluates policies written in Cedar. Authorization requests are expressed in terms of a principal, action, resource and context, and the service returns an allow or deny decision.
That makes it a good fit for one narrow question:
Is this actor authorised to perform this action against this resource under the supplied operational context?
It should not be asked to orchestrate the incident.
A simplified Cedar policy might look conceptually like this:
permit (
principal == RemediationActor::"eks-control-plane",
action == RemediationAction::"RollbackDeployment",
resource is KubernetesWorkload
)
when {
context.previousRevisionHealthy == true &&
context.statefulMigrationDetected == false &&
context.evidenceAgeSeconds <= 30 &&
context.maintenanceFreeze == false
};

A separate  forbid  policy can protect classes of operations that should never enter the autonomous path, such as identity modification or persistent-data deletion.
Cedar supports permit and forbid policies, and the effective evaluation model is fail-closed: a matching forbid wins, while the absence of an applicable permit produces a deny decision.
Schemas matter here as well. Verified Permissions can validate policy entities and attributes against a schema, which AWS recommends for production applications because it helps catch invalid references and policy mistakes.
But there is an architectural subtlety that is easy to miss.
Verified Permissions returns ALLOW or DENY. It does not natively return:
ALLOW_AUTO
REQUIRE_HUMAN
INVESTIGATE_MORE
ESCALATE

Trying to turn Cedar into a multi-stage incident engine creates unnecessary complexity.
Instead, keep routing separate.
A deterministic classifier can assign the request to one of three paths:
AUTO
APPROVAL
ESCALATE

Verified Permissions then decides whether the actor is authorised for the action on that path.
This separation keeps policy semantics clean.

Step Functions owns time and uncertainty

Once a remediation request survives policy evaluation, orchestration becomes the next problem.
Consider two rollbacks:
Case A
previous revision healthy
no state migration
namespace isolated
fresh evidence
→ bounded automatic execution

Case B
previous revision healthy
database migration detected
customer impact increasing
→ human decision required

Both involve the same Kubernetes operation. Their operational risk is different.
Step Functions is useful here because a workflow can branch between automated execution and a human approval path. Its callback pattern can pause a task until an external system returns a task token, and AWS explicitly documents human approval as a use case for this mechanism.
A state machine might therefore resemble:
CollectCurrentState
↓
ValidateEvidence
↓
ClassifyRisk
↙ ↓ ↘
AUTO APPROVAL ESCALATE
↓ ↓
Authorize WaitForHuman
↓ ↓
└──── Reauthorize
↓
ExecuteRunbook
↓
Verify
↙ ↘
SUCCESS ROLLBACK

Notice the reauthorisation after human approval.
An approval that waited ten minutes may no longer be safe to execute. Deployments move. New incidents occur. Another engineer may already have rolled the service forward.
Approval should therefore authorise intent, not freeze reality.
Before execution, the workflow should recollect critical state and evaluate policy again.

Systems Manager should execute a runbook, not arbitrary shell commands

It is tempting to make the final step something like:
kubectl rollout undo deployment/payments-api

executed on whichever host has cluster credentials.
That works technically. It is also exactly the type of execution path a control plane should eliminate.
AWS Systems Manager Automation provides runbooks composed of explicit parameters and ordered steps. Automation can call AWS APIs, execute scripts, invoke Lambda functions and invoke other Automation runbooks.
That makes the execution interface something closer to:
Runbook:
EKS-BoundedDeploymentRollback

Parameters:
Cluster
Namespace
Deployment
ExpectedCurrentRevision
TargetRevision
ExpectedGeneration
Timeout

rather than:
Command:
<arbitrary shell>

That difference is substantial.
The AI cannot invent a fourth parameter that changes IAM. It cannot append another  kubectl  operation. It cannot silently widen the namespace.
The runbook is the executable contract.
One design detail deserves pushback: Systems Manager Run Command should not become a generic mechanism for sending Kubernetes commands to EKS worker nodes.
Run Command is designed to execute administrative commands on Systems Manager managed nodes. Using worker nodes as privileged Kubernetes management hosts couples application remediation to infrastructure that should itself remain replaceable.
A cleaner pattern is:
Step Functions
↓
Systems Manager Automation
↓
Purpose-built executor
↓
Kubernetes API

The Automation runbook can invoke a Lambda-based executor or another tightly controlled component. Systems Manager Automation explicitly supports invoking Lambda functions from runbook steps.
The important property is not Lambda itself. It is that the executor exposes a small action vocabulary instead of a general command channel.
For example:
RestartSelectedStatelessPods
RollbackDeployment
ScaleDeploymentWithinBounds

No  ExecuteKubectl .
No  RunShell .
No “administrator mode.”

Give the executor less authority than the platform team

The executor still needs permission to modify Kubernetes resources, which creates another boundary.
Amazon EKS access entries associate IAM principals with Kubernetes permissions. They can be combined with EKS access policies or Kubernetes RBAC. AWS also supports mapping an access entry to Kubernetes groups, where  Role  and  RoleBinding  objects can define the permitted Kubernetes operations.
For a remediation executor, broad cluster administration is unnecessary.
A better boundary might be:
IAM role:
eks-remediation-executor

EKS access entry:
eks-remediation-executor

Kubernetes group:
bounded-remediation

Namespace:
payments

Allowed:
get deployments
get replicasets
get pods
patch selected deployments

Not allowed:
secrets
clusterroles
persistentvolumes
custom resource definitions
namespaces

Different applications may justify different permissions, but the principle remains:
The executor should be technically incapable of performing actions that policy says are prohibited.
Policy is one control. IAM and Kubernetes RBAC are another.
If a policy defect accidentally authorises an IAM-changing remediation, the Kubernetes executor should still have no authority to perform it.
That is defence in depth rather than confidence in one policy engine.

Authorization must be based on current state

There is a race hidden between policy evaluation and execution.
Suppose the control plane observes:
payments-api generation = 108
current revision = 42
target revision = 41

The policy authorises rollback.
Before the executor runs, a deployment pipeline releases revision 43.
Blindly applying the original remediation can now reverse a deployment the control plane never analysed.
This is a classic time-of-check versus time-of-use problem.
The execution contract should therefore contain expected state:
expectedGeneration: 108
expectedCurrentRevision: 42

Immediately before mutation, the executor reads the workload again.
If the observed state differs:
expected generation: 108
observed generation: 109

the executor stops.
It should not ask the AI whether the difference “looks safe.”
The workflow returns to investigation or escalation.
This is one of the most important boundaries in automated remediation: an authorised action against stale state is no longer the same action that was authorised.

Successful execution is not successful remediation

Another common architectural mistake is treating a successful API response as evidence of recovery.
The Kubernetes API accepting a Deployment update proves only that Kubernetes accepted the update.
It does not prove that:
  • replacement Pods became Ready,
  • the Service has healthy endpoints,
  • error rate recovered,
  • latency returned to normal,
  • dependencies remained healthy,
  • the original customer impact disappeared.
Verification should therefore be independent of execution.
A read-only verifier can examine:
Deployment desired vs Ready replicas
ReplicaSet convergence
Pod readiness
EndpointSlice readiness
synthetic transaction results
application error rate
latency
incident-specific SLI

The executor should not be allowed to mark its own action successful.
That gives the architecture a second separation of duties:
AI proposes
Policy authorises
Executor changes
Verifier judges

If verification fails, the workflow needs a predefined response.
That may be:
Rollback the remediation

or:
Stop automation and escalate

The second option is often safer. Reversing a failed remediation is itself another production change and should not be assumed safe merely because it restores the previous configuration.

Design for the control plane to fail

The architecture also has failure modes of its own.
If Verified Permissions is unavailable, the remediation path should fail closed rather than bypass policy.
If evidence collection fails, the system should not substitute model confidence for missing state.
If Step Functions waits for approval while the incident changes, the system should revalidate before execution.
If the Systems Manager runbook receives parameters outside its expected bounds, execution should stop.
If the Kubernetes executor's permissions are expanded later, that should be treated as a security-sensitive change to the remediation boundary.
And if independent verification cannot determine whether the service recovered, ambiguity should result in escalation rather than optimistic success.
These behaviours reduce autonomy, which is exactly the point.
The goal is not to maximise how many incidents an agent can remediate.
The goal is to maximise how many incidents can be remediated without silently increasing production risk.

The boundary that matters

A mature AI operations architecture does not ask:
Can the model execute Kubernetes commands?
That is easy.
It asks:
Under what independently verified conditions can this exact action be authorised, executed within a constrained capability set, and proven to have improved the system?
Amazon Verified Permissions provides an external policy decision point using Cedar. Step Functions provides orchestration and a practical human approval boundary. Systems Manager Automation provides explicit, bounded runbooks rather than unrestricted operational commands. Amazon EKS access entries and Kubernetes RBAC can constrain the executor to the permissions it actually needs.
None of those controls makes an AI recommendation correct.
Together, however, they change what happens when it is wrong.
The agent can investigate aggressively. It can propose quickly. It can reason across more evidence than a human may inspect manually.
But production authority remains somewhere else.
That is the architectural boundary worth preserving.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article