From Automation to Safe Autonomy: What 9 Kiro × AWS Experiments Taught Me

Across 10 Kiro × AWS experiments, I explored the gap between what an agent can do and what we can safely delegate. This post summarizes the key lessons, the evidence gaps I found, and why the next step is to design for Maximum Safe Autonomy rather than Maximum Automation.

nobu_13

Senior Director

Published Oct 16, 2026

Last modified Oct 16, 2026

From Automation to Safe Autonomy: What 10 Kiro × AWS Experiments Taught Me

Over the past several months, I completed a 10-part experiment series around:

Kiro × AWS × Agentic Engineering

The original question was simple:

How much of AWS software development can we automate with Kiro?

I explored that question across CloudFormation, AWS design standards, validation, IAM review, Serverless development, CI/CD repair, architecture review, security review, and FinOps.

The tenth article brought all of those experiments together.

By then, however, the question had changed.

It was no longer:

How much can we automate?

It became:

How much can we safely delegate to an agent?

That shift is the most important outcome of the series.

Five lessons I would share with the community

1. Detection and judgment are different capabilities

Finding more issues does not automatically mean an agent becomes better at judging:

severity
priority
risk
whether action should be taken

In both the Security and FinOps experiments, better detection did not consistently improve calibration.

For me, this means agent evaluation should separate:

Detection → Decision → Calibration → Execution

instead of treating them as one capability.

2. A correct fix is not automatically a safe fix

In the Security experiment, remediation correctness improved while remediation safety did not.

That distinction matters for agentic development.

An agent may be able to generate a technically reasonable change without having enough evidence to decide whether that change should actually be applied.

So I now evaluate these as different questions:

Can the agent produce the change?

and:

Should the agent be allowed to execute the change?

3. Permission can limit autonomy even when reasoning succeeds

The CI/CD experiments showed another boundary.

The agent could sometimes diagnose the failure correctly and attempt a repair, but still fail to complete recovery because its permission boundary prevented the required action.

This suggests that autonomy depends not only on reasoning capability, but also on:

allowed actions
blast radius
rollback permissions
recovery paths

Reasoning success alone is not enough for operational autonomy.

4. Passing a validator is not the same as runtime correctness

Across the series I used tools such as:

cfn-lint
pytest
security rules
deterministic evaluators

These were valuable, but they validate different things.

A static or deterministic validation pass should not automatically be interpreted as:

“This is safe to release.”

Runtime verification and recovery evidence remain separate concerns.

5. Knowing when not to act is also an agent capability

This was one of the biggest changes in my thinking.

In the FinOps experiment, safer behavior sometimes meant:

HUMAN_REVIEW
DO_NOT_RECOMMEND

rather than automatically applying an optimization.

The guardrailed condition reduced unsafe automatic recommendations, but it also reduced the overall automation rate.

That is not necessarily a failure.

For high-context decisions, an agent that knows when to stop may be more useful than an agent that always acts.

What happened when I combined the nine experiments?

For the final synthesis, I mapped the evidence from the first nine experiments into an equal-weight benchmark:

11 engineering phases × 5 capabilities = 55 cells

The five capabilities were:

Plan
Act
Validate
Recover
Escalate

The result was:

Metric	Result
Evidence Coverage	18 / 55
Direct Capability Evidence	14 / 55
Observed Agentic (L3+)	0 / 55
Safe Agentic (L3+)	0 / 55
Human-Gated (Approve / Block)	12 / 55

These numbers require careful interpretation.

18/55 does not mean that 32.7% of AWS engineering is automated.

The benchmark is an analysis structure, not a representation of engineering workload or business impact.

Likewise:

0/55 for L3+ does not mean that Kiro is “not agentic.”

It means that the experiments did not provide enough evidence to confirm the stricter conditions I used for Governed Agentic behavior, such as repeatability, independent validation, and safe behavior under uncertainty or failure.

The biggest gap I found was not simply agent capability.

It was the evidence required to justify greater autonomy.

Capability and safe delegation should be evaluated separately

The final synthesis separated two questions.

Observed capability

What did the agent actually demonstrate?

Recommended operating level

Given the available evidence and operational controls, how much autonomy should we recommend?

To make the second decision, I found that at least these areas repeatedly mattered:

Boundary
Context & Evidence
Validation
Permission & Blast Radius
Recovery
Risk & Reversibility

I also evaluated the human boundary separately:

AUTO
NOTIFY
APPROVE
BLOCK

One lesson from this work is that adding a human approval step should not automatically be treated as a failure of agentic engineering.

Sometimes it is the correct operating design.

From Maximum Automation to Maximum Safe Autonomy

At the beginning of the series, I was mostly asking:

How much can Kiro automate?

After ten articles, I think a more useful engineering question is:

What evidence and controls are required before we delegate the next action to the agent?

That leads to a different optimization target.

Not:

Maximum Automation

but:

Maximum Safe Autonomy

The goal is not to move every workflow toward full autonomy.

The goal is to find the appropriate autonomy level for the task, risk, evidence, and operational environment.

What comes next: AAEM

The next step is a project I am calling:

AAEM — Agentic Autonomy & Evidence Model for AWS Engineering

AAEM is not an AWS framework, and it is not a completed model today.

I also do not plan to simply rename the evaluation structure used in the final synthesis and call it AAEM.

The 55-cell benchmark, capability levels, control conditions, and human gates are useful starting evidence, but they still need to be challenged.

The next phase starts from the problem itself:

What problem should AAEM solve?
Who should use it?
What should the unit of assessment be?
Which concepts belong in the core model?
How should evidence quality affect autonomy decisions?
Which controls should limit autonomy?
How should risk and reversibility change the recommendation?
Where should human approval remain by design?
What evidence should be required before increasing autonomy?

I also plan to compare these ideas with existing AWS and agent-governance approaches before fixing the structure.

Only after that will I define an AAEM v0.1 and validate whether it can explain both the previous experiments and new cases.

The goal is not to create another large checklist.

The goal is to answer a practical engineering question:

For this task, with this evidence and these controls, how much should we safely delegate to the agent — and what evidence do we need next?

The 10-part Kiro × AWS × Agentic Engineering series

This Builder Center post is a condensed summary of the full experiment series.

The ten experiments answered some questions, but they also exposed a much larger one:

How do we move from “the agent can do this” to “we have enough evidence to safely let the agent do this”?

That is the question I want to explore next with AAEM.

# kiro # devops # developer-tools # generative-ai # ai-dlc

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article