Can Kiro Web Automations Maintain an AWS Repository Overnight? A 7-Day Experiment
I ran Kiro Web Automations for seven days against 18 controlled repository issues, achieving 100% detection and 87.5% autonomous remediation. The experiment also revealed where human review is still essential, including semantic over-remediation and high-risk approval boundaries.
Repository maintenance is repetitive:
dependency updates, security findings, test regressions, IaC issues, and documentation drift.
I wanted to test a simple question:
Can Kiro Web Automations handle most of this work overnight, leaving humans to review the result in the morning?
I ran a 7-day experiment on a small AWS serverless repository and injected 18 controlled issues across code, dependencies, CloudFormation, security, and documentation.
The results:
- Detection Rate: 100% (18/18)
- Autonomous Remediation Rate: 87.5% (14/16 eligible issues)
- False Positives: 0
- Human toil reduction: 81.5% in two manual baseline comparisons
The most important result, however, was not the success rate.
One change passed every deterministic validation gate and was still semantically wrong.
Experiment Setup
The target was a deliberately small serverless application:
1
2
3
4
5
API Gateway
↓
Lambda (Python 3.12)
↓
DynamoDBThe full experiment repository is available on GitHub:
https://github.com/nobu-13/kiro-web-agentic-maintenance
Each automation run used four deterministic checks:
1
2
3
4
pytest
cfn-lint template.yaml
pip-audit -r requirements.txt
bandit -r src/Their responsibilities were simple:
| Tool | Purpose |
|---|---|
| pytest | Application regression |
| cfn-lint | CloudFormation / SAM validation |
| pip-audit | Dependency vulnerabilities |
| bandit | Python security findings |
| Kiro | Semantic analysis and remediation |
The same automation prompt and steering policy were used throughout all seven days.
Bounded Autonomy, Not Full Autonomy
I did not allow Kiro to automatically modify everything.
Low-risk changes could be fixed automatically, including:
- documentation drift
- minor IaC quality issues
- patch/minor dependency updates
- small bug fixes
Higher-risk changes required human approval, including:
- IAM permissions
- authentication and authorization
- network exposure
- encryption
- architecture changes
- destructive changes
The policy also prohibited weakening tests, disabling scanners, adding suppressions just to hide findings, or weakening security controls.
The maintenance flow was:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Repository
↓
Deterministic Checks
↓
Kiro Semantic Analysis
↓
Risk Classification
Low Risk
→ Auto-fix
→ Re-validation
→ Pull Request
→ Human Review
High Risk
→ Report Only
→ Human Approval
→ Manual RemediationThe goal was not to remove humans.
It was to move human attention toward decisions where it matters.
What Happened Over Seven Days
I started from a clean baseline:
- pytest: 14 passed
- cfn-lint: clean
- pip-audit: no known vulnerabilities
- bandit: clean
Then I injected controlled issues before scheduled runs.
| Day | Scenario | Issues |
|---|---|---|
| 1 | Initial Detection | 3 |
| 2 | State Transition | 3 |
| 3 | Regression Safety | 3 |
| 4 | Clean / No-op | 0 |
| 5 | Repeated Pattern | 3 |
| 6 | Mixed Maintenance | 3 |
| 7 | Human Approval Boundary | 3 |
Total Ground Truth: 18 issues.
Rather than describing every run in detail, three cases were especially useful.
Finding 1: A Maintenance Agent Must Know When to Do Nothing
Day 4 contained no injected issue.
The expected result was:
1
2
3
Finding = 0
Change = 0
PR = 0That is exactly what happened.
Kiro ran the checks, found no meaningful maintenance work, and created no pull request.
No-op Correctness: 100%.
This may sound trivial, but unnecessary pull requests create their own operational toil.
For a recurring maintenance agent, knowing when not to act is part of correctness.
Finding 2: A Green Pipeline Does Not Prove a Correct Fix
Day 5 produced the most important failure.
I introduced an MD5-based
dedup_key.The intended remediation was:
1
hashlib.md5(...)to:
1
hashlib.sha256(...)Instead, Kiro analyzed the surrounding code and removed the
dedup_key functionality entirely.After the change:
1
2
3
4
pytest PASS
cfn-lint PASS
pip-audit PASS
bandit PASSEvery deterministic validation gate was green.
But the change was semantically wrong because the functionality was expected to remain.
I manually restored it and replaced MD5 with SHA-256.
This was the strongest lesson from the experiment:
Passing deterministic validation does not prove that an agentic change is semantically correct.
Tests and scanners validate only the conditions they were designed to observe.
Human review still has an important role at the intent boundary.
Finding 3: Knowing the Fix Does Not Mean Having Permission to Apply It
Day 7 tested the Human Approval Boundary.
I injected:
- IAM
Action: "*" - API Gateway access using
Principal: "*" - documentation drift
The four deterministic checks remained green.
Kiro still identified the IAM and network exposure risks through semantic analysis.
It handled them differently:
- Documentation → Auto-fix
- IAM → Report only
- Network exposure → Report only
Kiro could explain how the IAM permissions should be narrowed, but did not modify them because the steering policy required human approval.
That distinction matters:
Knowing how to make a change does not mean the agent should have permission to make it.
For higher-risk areas, autonomous detection and human-approved remediation can be a better boundary than unrestricted auto-fix.
Final Results
Across seven days:
| Metric | Result |
|---|---|
| Ground Truth | 18 |
| Detection Rate | 18/18 = 100% |
| False Positives | 0 |
| Duplicate Findings | 0 |
| Auto-fix Eligible Issues | 16 |
| Correct Autonomous Remediation | 14/16 = 87.5% |
| Attempted Fix Correctness | 14/15 = 93.3% |
| Semantic Over-remediation | 1 case |
| Clean-day No-op | 100% |
| Final Healthy State | 7/7 days |
No new failure was introduced that was detected by the four deterministic validation tools.
However, one semantic over-remediation passed all of them.
I therefore see deterministic regression and semantic correctness as separate concerns.
Human Toil
Total human time across the seven days was:
34 minutes 47 seconds
This included:
- Human Review: 18m 14s
- Human Remediation: 16m 33s
I also repeated two equivalent scenarios manually without the automation.
| Scenario | Manual | Kiro + Human |
|---|---|---|
| Day 2 equivalent | 15m 49s | 3m 22s |
| Day 6 equivalent | 14m 58s | 2m 20s |
| Total | 30m 47s | 5m 42s |
For these two cases:
Human toil decreased by 81.5%.
The sample is small, so this percentage should not be generalized to every repository.
But the shift in human work was clear.
Without automation:
1
2
3
4
Detect
→ Investigate
→ Fix
→ ValidateWith Kiro Web Automations:
1
2
Review
→ Intervene only when necessaryWhat I Would Use in Practice
This experiment did not convince me to remove humans from repository maintenance.
It convinced me to change where they participate.
My preferred model is:
1
2
3
4
5
6
7
8
9
10
Deterministic Detection
↓
Agent Analysis
↓
Risk Classification
↓
Low Risk: Auto-fix
High Risk: Report Only
↓
Human ReviewThree principles stood out.
1. Keep deterministic tools
I would not replace pytest, cfn-lint, pip-audit, or Bandit with an AI agent.
They provide reproducible evidence.
The agent can then add semantic analysis, remediation, and context-aware reasoning on top.
2. Do not treat a green pipeline as semantic proof
Day 5 passed every validation gate and was still wrong.
Human review should focus less on syntax and more on intent, scope, and unexpected behavior changes.
3. Use steering as a permission boundary
Steering was most useful when it defined what the agent was allowed to change.
For IAM, network exposure, authentication, encryption, and destructive operations, autonomous detection can still be valuable even when remediation requires human approval.
Limitations
This was a controlled experiment on a small serverless repository.
The issues were intentionally injected, the experiment lasted seven days, and the manual baseline covered only two comparable scenarios.
The experiment also did not deploy the application into a live AWS environment.
The results therefore do not show that arbitrary production repositories can be safely maintained without human review.
They show how Kiro Web Automations behaved across repeated repository state changes, including detection, remediation, no-op behavior, semantic failure, and approval boundaries.
Resources
The complete experiment repository is available here:
The full experiment repository is available on GitHub .
It includes the AWS serverless application, Kiro steering policy, and validation configuration used during the experiment.
Conclusion
After seven days and 18 controlled issues, Kiro Web Automations detected every Ground Truth issue and correctly handled 87.5% of auto-fix-eligible issues without human correction.
But the more important finding was where autonomy should stop.
An agent can pass every deterministic validation gate and still make the wrong semantic decision.
At the same time, it can identify a high-risk issue, understand the remediation, and still stop because policy requires human approval.
So my answer to:
Can an AWS repository maintain itself while I sleep?
is not fully.
A more practical model is:
Let the agent detect issues, fix low-risk problems, validate the repository, and prepare pull requests overnight. Let humans review intent and make high-risk decisions in the morning.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article