AWS Builder Center

Builder Center

500 Passing Tests Weren’t Enough: What Building an AI-DLC Simulator with Kiro Taught Me About Human Gates

500 Passing Tests Weren’t Enough: What Building an AI-DLC Simulator with Kiro Taught Me About Human Gates

Even with 500 passing automated tests, the project was not automatically release-ready. This article shares practical lessons from building an AI-DLC Learning Simulator with Kiro, including Human Gates, rework, evidence, blind audits, and evidence-based release decisions.

nobu_13

Senior Director

500 Passing Tests Weren’t Enough: What Building an AI-DLC Simulator with Kiro Taught Me About Human Gates

I built an open-source AI-DLC Learning Simulator with Kiro to make Human Gates, Rework, Evidence, and Release decisions easier to experience.
The first release is intentionally deterministic and does not use a runtime LLM or Amazon Bedrock. The goal was to create a low-cost learning environment for practicing AI-DLC decision points in the browser.
The interesting part is that I also used an AI-DLC workflow to build the simulator itself.
That development process changed how I think about validation in agentic development.

252 tests passed, but the product was not ready

At RC3, the project had:
  • 252 automated tests passing
  • P0: 0
  • P1: 6
  • P2: 8
  • P3: 1
  • Release status: NOT READY
The automated tests were useful, but they did not guarantee that the product semantics were correct.
Several issues required rework, including:
  • Conditional Approval
  • Root Finding vs. downstream manifestation
  • Evidence sufficiency
  • Completion vs. Release
This led to a simple realization:
Test PASS ≠ Release Ready

The product reached 500 tests

After repeated rework, the v1.0.0 release checkpoint reached:
  • 500 automated tests
  • 54 test files
  • TypeScript typecheck: PASS
  • ESLint: PASS
  • Production build: PASS
But I still did not treat that as sufficient release evidence.
I added blind browser-based product audits.

What happened when reviewers disagreed?

Two RC6 reviews reached different conclusions.
One audit raised two P1-level concerns.
Another multi-reviewer audit concluded:
1
2
3
P0: 0
P1: 0
Ready with minor fixes
Instead of deciding by majority vote, I reproduced the disputed behavior in fresh isolated browser environments.
Two areas were checked:
  1. Root Finding Identity
  2. Evidence Semantics
Each was reproduced twice.
The result:
1
2
3
4
5
PASS × 2
PASS × 2

Final judgment:
Not reproduced
No product code was changed during this adjudication.
The disagreement was resolved by additional evidence, not by opinion.

Three lessons from the rework

1. Approval ≠ Risk Resolution

A stage can continue while conditions remain open.
Conditional Approval needs to preserve those conditions downstream instead of turning them into a simple "approved" state.

2. Artifact exists ≠ Evidence sufficient

An evidence artifact can be complete while required execution evidence is still missing.
1
2
3
4
5
Artifact exists
≠
Evidence sufficient
≠
Release ready

3. Completion ≠ Release

Finishing implementation is different from authorizing production release.
Release still requires a separate decision based on evidence and residual risk.

A validation pattern I would reuse

The workflow I would use again is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Generate
↓
Deterministic Validation
↓
Semantic / Product Review
↓
Human Gate
↓
Rework
↓
Independent Validation
↓
Reproduce disputed findings
↓
Evidence-based Release Decision
The most important lesson was this:
AI review should be treated as evidence input, not as the final authority.
A reviewer finding may be important, but it still needs to be reproducible.

AWS architecture

The simulator runs as a low-cost static application:
1
2
3
4
5
6
7
Browser
↓
Amazon CloudFront
↓
Origin Access Control
↓
Private Amazon S3
The infrastructure is defined with AWS CloudFormation.
The application intentionally has:
  • no runtime backend
  • no runtime LLM dependency
  • no AWS credentials in the browser
Kiro was used throughout development and connected to AWS during development and release preparation.

What I learned

Building an AI-DLC learning tool with an AI-DLC workflow made one thing very clear:
Generation ≠ Validation
Test PASS ≠ Release Ready
Approval ≠ Risk Resolution
Finding ≠ Truth
Completion ≠ Release
As coding agents become better at producing artifacts, the human role increasingly shifts toward deciding:
What evidence is sufficient for this work to move forward?
That is the decision model I wanted the simulator to make easier to practice.

Try it

Live Demo:
https://d1s9ig7adbe2qs.cloudfront.net
GitHub:
https://github.com/nobu-13/aidlc-learning-simulator
The project is open source under MIT-0.
# kiro# ai-dlc# developer-tools# generative-ai# devops
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article