What 60 Kiro CI/CD Runs Taught Me About Safe Agentic Repair on AWS
A 60-run Kiro CI repair experiment exploring the value of Guardrails beyond success rate. It examines regression risk, permission failures, recovery paths, and practical design principles for safer Agentic CI/CD.
AI agents can already help us write code. The harder question is whether they can safely repair failing CI pipelines before human review.
I tested this with Kiro across 60 CI repair runs.
The goal was simple:
Can Kiro detect a CI failure, repair the problem, validate the fix, and stop before human approval?
I compared two configurations:
- Pure Kiro — minimal repair instructions
- Guardrailed Kiro — additional Steering, Hooks, and a repair-loop policy
The interesting result was not the success-rate difference.
It was how the agent failed.
Experiment design
I created six failure scenarios covering common AWS development problems:
- CloudFormation syntax/configuration error
- Python application logic regression
- DynamoDB encryption misconfiguration
- Excessive IAM permissions
- Environment-variable contract mismatch
- Application-to-IAM permission mismatch
Each scenario was executed five times with both configurations.
That produced:
- 6 scenarios
- 2 configurations
- 5 runs per configuration
- 60 valid runs
Each run used a fresh workspace.
The validation flow was:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Clean base
↓
Inject failure
↓
Run deterministic validation
↓
Give failure information to Kiro
↓
Agent repairs files
↓
Run validation again
↓
Compare against ground truth
↓
Human gate
The agent never decided whether its own repair was successful.
That decision was always made by deterministic validators.
Results
| Metric | Pure | Guardrailed |
|---|---|---|
| Autonomous Resolution | 25/30 (83.3%) | 27/30 (90.0%) |
| Final Validation Pass | 25/30 | 27/30 |
| Ground Truth Match | 25/30 | 27/30 |
| Regression | 1/30 | 0/30 |
| Average runtime | 57.8s | 67.9s |
At first glance, Guardrailed Kiro looks better.
But the observed 6.7 percentage-point difference was not statistically significant.
Fisher's exact test produced:
1
p = 0.7065
So this experiment does not prove that Guardrails improve repair success rate.
What they changed more clearly was failure behavior.
The most important failure
One Pure Kiro run received a simple CloudFormation error.
The expected fix was effectively one property change.
Kiro correctly diagnosed the problem.
But during the repair,
fs_write replaced the entire CloudFormation template with a single line.The result:
- 81 changed lines
- original validation still failed
- two additional validations regressed
- recovery attempts were later blocked by permission restrictions
This was not primarily a reasoning failure.
The agent understood the problem.
The failure came from the combination of:
- tool usage
- write strategy
- permission boundaries
- recovery limitations
That changed how I think about Agentic CI/CD.
Guardrails reduced tail risk
Successful runs were almost identical in edit size:
- Pure: 2.40 changed lines on average
- Guardrailed: 2.37
So Guardrails did not meaningfully reduce the typical successful edit.
The difference appeared in the tail.
Maximum observed change size:
- Pure: 81 lines
- Guardrailed: 5 lines
Regression:
- Pure: 1
- Guardrailed: 0
This suggests a more useful way to think about Guardrails:
Guardrails may be less about making an agent smarter and more about controlling how badly it can fail.
Permission design matters as much as prompting
Permission rejection occurred frequently in both configurations.
Final counts were:
- Pure: 148 rejected tool calls
- Guardrailed: 161
Seven of the eight failed runs involved permission blocking.
Interestingly, failed runs did not simply have more rejections than successful runs.
The more important question was:
Which operation was rejected?
A rejected optional check may be harmless.
A rejected read, write, or recovery operation can stop the entire repair flow.
For Agentic CI/CD, permission design therefore needs to include not only the normal repair path, but also the recovery path.
Five design principles
The experiment led to five practical rules.
1. Never let the agent grade its own repair
Use external deterministic validators.
Examples include:
- pytest
- cfn-lint
- cfn-guard
- contract tests
- policy checks
The agent proposes the repair. The validator decides whether it worked.
2. Keep ground truth outside the agent workspace
Otherwise the agent may accidentally solve the benchmark instead of the engineering problem.
3. Use fresh workspaces
Every run should start from the same known state.
This prevents hidden state from contaminating later runs.
4. Optimize for failure containment, not only success rate
A 90% repair rate is less useful if the remaining 10% can destroy unrelated files.
Track:
- regression
- maximum change size
- unrelated file changes
- recovery behavior
5. Design recovery permissions explicitly
A safe agent needs a path to recover from its own mistakes.
If rollback or inspection commands are blocked, the safety boundary itself can turn a recoverable mistake into a failed run.
Human review still matters
Human intervention during the automated runs was zero.
That does not mean humans are unnecessary.
The architecture I would use is:
1
2
3
4
5
6
7
8
CI Failure
↓
Agent Repair
↓
Deterministic Validation
↓
PASS → Human Review
FAIL → Retry or Escalation
I would not allow the agent to autonomously approve production deployment.
The useful boundary is earlier:
Let the agent do the repetitive diagnosis and repair work before human review.
Conclusion
Kiro successfully resolved most of the injected CI failures in this experiment.
But the most useful lesson was not the difference between 83.3% and 90.0%.
It was this:
Safe Agentic CI/CD depends on more than model capability.
It also depends on:
- deterministic validation
- minimal-change policies
- permission boundaries
- recovery paths
- regression detection
- human approval
The agent needs to know how to fix the problem.
The system also needs to decide how much damage the agent is allowed to cause when it is wrong.
That is where Guardrails become valuable.
The experiment code, scenario definitions, aggregated results, and evidence are available here:
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article