AWS Builder Center

Builder Center

Build the yardstick before you rewrite the monolith

Build the yardstick before you rewrite the monolith

Series: Rewriting a PHP monolith to Python with AWS Transform custom + Kiro. Part 1 of 3: building the black-box behavioral test harness that captures the app's current behavior, the yardstick every later step is graded against.

Series: Monolith Modernization - Kiro+AWS Transform (3 articles)

  1. 1
    Build the yardstick before you rewrite the monolith This article
  2. 2
    Transform the monolith, validated against the yardstick every step
  3. 3
    Kiro closes the last mile and finishes the migration

Venugopalan Vasudevan

My last two posts were about turning a monolith into documentation , and then
turning that documentation back into working code . The second one ended on a warning I couldn't stop thinking about: the whole spec-to-rewrite trick only worked because the codebase already had a comprehensive test suite to grade against. Take that away and you're not modernizing, you're guessing.
So this series starts where that warning points: before you rewrite anything, you build the thing that will tell you whether the rewrite worked. A yardstick. A behavioral safety net. The
unglamorous artifact that turns "I think it still works" into "I can prove it still works."
Here's the plan for the series. I'm taking a real open-source monolith, Kanboard, a pure-PHP
Kanban app, and rewriting it from PHP to Python, keeping the exact same API and the exact same database, swapping only the implementation underneath. Two tools do the work, and the split is the whole story: AWS Transform custom performs the bulk cross-codebase rewrite, and it uses this harness as its own validation gate, so it checks its work against the yardstick as it goes, while Kiro, the AI agent I drive in my terminal, builds that harness first and fixes whatever's left at the end.
Three posts:
  1. Map the app with business-rules-extraction, then build the harness (this one). Run AWS
    Transform's managed business-rules-extraction to get the app's domains + full feature inventory, then have Kiro build the harness, using that inventory as the coverage checklist so it captures every feature, not just the happy path.
  2. Point AWS Transform custom at Kanboard: author a PHP → Python transformation skill and run the upgrade, with the harness wired in as Transform's validation command.
  3. Kiro closes the last mile, fixes whatever the transformation left open, and reports the
    reports the final coverage.
This post is part one, and it's the one that transfers to your project whether or not you care
about Kanboard or Python. Everything after this paragraph is a reproducible recipe for using an AI agent to build a rewrite-proof test harness for any app. Steal it.

First, map the app, run business-rules-extraction

Before Kiro writes a single test, point AWS Transform's managedbusiness-rules-extraction at the app. It's a read-only analysis, it never builds, runs, or changes your code, that returns the app's domains, business rules, a reconstructed schema, and the part that matters most here: a complete feature inventory. That inventory is the harness's coverage checklist, the list of everything the tests must exercise.
1
2
3
atx custom def exec \
--transformation-name AWS/business-rules-extraction \
--code-repository-path ./kanboard
For Kanboard it came back with 10 domains and 104 features. (I covered what business-rules-extraction produces in depth in my first post  ,here I'm reusing its output, not re-explaining it.) I hand that feature list to Kiro as the coverage denominator, so "cover every flow" becomes true by construction instead of by hope, and the same output grounds the transformation skill in part two, so you run the analysis once and use it twice.
Full disclosure: I did not do this first the one time it counted, I built the harness blind,
thought it was comprehensive, and only later discovered it covered ~42% of the features. That story, and why running the extraction first is the fix, is in the note at the end.

The problem, and the kind of test that survives it

A rewrite is a promise, the new system behaves like the old one, and it's hard to keep because
the old system's behavior was never written down anywhere except the old system. Worse, your
existing tests can't come with you: PHPUnit can't run against Python, JUnit can't judge Go. They're bolted to the implementation you're deleting.
So you need a yardstick that doesn't care what language the app is written in. Two properties turn a test suite into one:
Characterize, don't verify. You're not proving the app is right, you're proving the rewrite
behaves identically to today's app, quirks and bugs included. Capture the current observable
behavior and freeze it as the ground truth. "What it does today" is the spec.
Test at the boundary, not in the code. The implementation is being replaced, so every assertion has to live outside it, on the wire, not in the classes. That gives three language-neutral layers:
LayerWhat it freezesWhy it survives the rewrite
Data golden masterfull read-back state after a known set of writes"what's in the system" doesn't care what produced it
API request/response pairseach endpoint's response + side effectsa JSON request → JSON response is the same behind PHP or Python
UI behavioral E2Euser-observable outcomes (a task appears; the dashboard loads)asserting behavior by role/label survives a markup rewrite
Swap in REST, gRPC, GraphQL, a CLI, or a queue and nothing changes but where you record the pairs.
Because everything lives at the boundary, one setting, the base URL, points the whole suite at
the old app today or the new one tomorrow. That retargetability is the yardstick, so protect it:
keep the harness network-only, never let it import app code, and add a guard test that fails the
build if it does.
One catch decides everything: coverage is the ceiling. Anything the harness doesn't exercise is
a behavior you can't claim the rewrite preserved. Most legacy apps have no suite like this, so you have to build it before you can safely rewrite anything, and that's a real project of its own, not a quick setup step.

I let the agent do the volume and kept the judgment for myself

Writing a comprehensive characterization suite by hand is exactly the kind of large, patterned,
"translate this existing thing into tests" work an AI agent is good at, and exactly the kind of
work where an agent left alone will confidently build something, just not always the thing you
meant. The trick is the loop.
I used Kiro's Spec mode for that loop. A spec  is a structured way to turn a high-level idea into an implementation plan through three reviewable phases:
requirements → design → tasks, each written to a file you approve before the next one begins.
It's the opposite of freestyle codegen: it drags the consequential decisions into the open before
any code exists, where they're cheap to change, and it's the right tool precisely when the thing
you're building is big and getting it wrong is expensive, which describes a yardstick exactly.
Spec mode also has a built-in correctness step worth knowing about: from your requirements it
generates property-based tests , universal invariants checked across hundreds of generated inputs, not just the handful of examples you'd think to write. That's what pins the harness's own load-bearing logic (how it normalizes and compares responses), and it earned its keep: a generated input caught a normalization bug, Python's str.isdigit() accepts non-ASCII digits that int() then can't parse, that no hand-written example would ever have tried.
1
requirements → design → tasks → execute
The kickoff prompt is deliberately short, a compact seed to start the spec from:
Build a black-box behavioral test harness for <my app> that captures its current behavior and replays it against a future rewrite. Talk to the app only over its API and a browser, never import app code, and target a configurable base URL so the identical suite runs against the old and the new implementation. Cover every feature, not just every endpoint: if a business-rules-extraction (or any feature/domain inventory) is available, use it as the coverage checklist and check every feature against it; features with no read-API, event-driven or background behaviors that persist a result the app never hands back through a query, are covered by asserting their side effects, reading the data store directly where there's no read-API. Three layers: (1) seed a deterministic dataset and snapshot the normalized read-back state; (2) exercise every API method and freeze request/response pairs, success, permissions, and errors, but normalize the responses (strip timestamps and IDs, resolve IDs to stable names) and reference entities symbolically so each frozen pair replays against a freshly-seeded rewrite; (3) drive key user flows in the browser with semantic locators. Baseline everything green on the current app first.
That's intentionally a seed, not a finished brief, spec mode plus your review will surface your
app's own traps, the same way mine surfaced Kanboard's. If you'd rather see the finished shape to aim for, the appendix at the end of this post gives the full, comprehensive brief.
And a spec is a loop, not a one-shot, because I didn't hand it a perfect brief, no one-page brief ever is. The agent built faithfully from what I gave it; the gaps that showed up were gaps in my framing, and review is where you fold in the domain knowledge you didn't think to front-load. Three examples, none of them the agent's fault:
  • Auth. I hadn't mentioned that the app-wide API token bypasses permission checks, so
    permission probes would have passed no matter what. Review added a real, under-privileged user identity.
  • Coverage. I'd asked for "workflows, permissions, errors," not "every method", so review added a task to probe each one or record a written exclusion.
  • Replay. I hadn't thought hard enough about it, that one earns its own section below.
Each traces back to something I could have front-loaded, and none needed me to write code, just to read the app for five minutes and tell the agent what I found. The agent supplies the volume; you supply the judgment, in review.

A real run

Here's what came out the other end. Keep the shape simple, it's three kinds of test, one per layer:
  • A data golden master, one seeded dataset, snapshotted into a single canonical golden.json.
  • 102 API/DB-state request/response probes, one or more per endpoint: reads, multi-step
    workflows, permission denials, error cases, and own-scoped database-state assertions for features with no read-API.
  • 93 UI end-to-end flows across 20 areas, auth, dashboard, board, tasks, projects, columns,
    swimlanes, notifications, analytics, access control, configuration, user & identity, and more, each logs in, acts through semantic locators, and asserts a user-observable outcome (or the own-scoped DB row it writes).
Between them they exercise 159 of the app's 160 API methods and 93 of 103 user-facing UI
journeys, all green against the current app. (The harness also tests itself, more on why that
matters later.)
But the number that actually matters is feature coverage against the business-rules-extraction
inventory: 74% of the 104 documented features (77/104), each asserting a real observable effect. It didn't arrive in one shot, the core of this series: ~42% from the first spec, ~49% after
an API-coverage pass, then 74% once a UI-layer pass drove the event-driven and web-only features the JSON-RPC API can't reach through the browser. Same method, three passes, coverage grown on purpose, against a real denominator. (Now folded into the appendix prompt, so you can ask for it up front.)
Here's the app under test, seeded with that deterministic dataset, the exact state the harness
freezes, and the state the Python rewrite will have to reproduce in a later post:
kanboard
The artifacts it froze are the actual product. A slice of the data golden master:
notice there isn't a raw ID or a timestamp anywhere; everything volatile has been normalized away so the snapshot is stable and diffable:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"name": "Mobile App",
"tasks": [
{
"title": "Beta release",
"column": "Backlog",
"owner": "admin",
"date_due": "2026-09-01",
"priority": "2",
"subtasks": [
{ "title": "Invite testers", "user": "carol", "status": "0" },
{ "title": "Upload build", "user": "admin", "status": "0" }
],
"tags": ["release"]
}
]
}
An API request/response pair for a full task-link lifecycle, the requests name entities
symbolically (never by raw ID), and the frozen response has every volatile value tokenized:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
"request": [
{ "method": "createTaskLink",
"params": { "task_id": "task:Website Redesign/Probe TaskLink A (throwaway)",
"opposite_task_id": "task:Website Redesign/Probe TaskLink B (throwaway)",
"link_id": 1 } },
{ "method": "removeTaskLink", "params": { "task_link_id": "<created task link>" } }
],
"response": {
"count": 1, "removed": true, "updated": true,
"task_link": { "id": "<id:relative#1>", "label": "relates to",
"link_id": "<link:relative#1>", "task_id": "<task:relative#1>" }
}
}
And, because a rewrite that returns success where the original failed is a regression you very
much want to catch, error behavior is frozen too:
1
2
3
4
{
"request": { "method": "createTask", "params": { "project_id": "project:Website Redesign" } },
"response": { "error": { "code": -32602, "message": "Invalid params" } }
}
And a UI flow is the same idea at the browser boundary: log in, act through semantic locators
only (so it survives a markup rewrite), and assert a user-observable outcome, including things
that must disappear:
1
2
3
4
5
6
7
8
9
10
11
12
13
class SearchFilterBoardFlow(UiFlow):
name = journey_id = "board.search-filter"

def run(self, page, base_url):
def body(page, base_url):
page.goto(f"{base_url}/?controller=BoardViewController&action=show&project_id=1")
box = resolve(page.get_by_role("textbox", name="Filter"))
box.fill("assignee:admin")
box.press("Enter")
# observable outcome: the admin-assigned task stays, the alice-assigned one is gone
resolve(page.get_by_role("link", name="Launch checklist"))
resolve_absent(page.get_by_role("link", name="Audit current pages"))
return run_with_login(self.name, base_url, page, body)
Coverage is the one number to take seriously, because the harness is the only oracle, any
method it never touches is a behavior you can't claim the rewrite preserved. So I tracked it and
committed it as an artifact: 159 of 160 API methods exercised, 1 excluded on purpose (it needs an LDAP backend the black-box SUT doesn't provide), 93 of 103 UI journeys covered, and, the measure that matters most, 77 of 104 features (74%), each with a written reason for anything excluded.
1
2
3
4
5
6
7
8
9
10
$ python -m harness.runner
Layer 1: PASS
Layer 2: PASS
Layer 3: PASS

=== API coverage === methods: total=160 exercised=159 excluded=1 not-yet-covered=0
=== UI coverage === journeys: total=103 covered=93 excluded=10 gaps=0
=== Feature coverage === features: total=104 covered=77 partial=7 scoped-out=20 (74%)

exit code: 0
But the transferable lesson isn't the number, it's the mindset: don't chase 100%. Exercise what
you reasonably can, document what you skip and why, and stop. An honest "covers most, here are the gaps" beats a perfect score you can't stand behind.

Verify the tests are correct, not just green

A characterization suite can be green and still be wrong, "green" only means it agrees with the app it was recorded from. The question that actually matters is a different one:
Would every frozen value still match against a different implementation on a different host, not the one it was recorded from?
You can't answer that by running the tests; they pass trivially on the same target. Two reusable
ways to answer it:
  1. A review pass. Point your agent at the frozen artifacts and have it hunt for anything that
    won't survive a retarget:> Scan every committed request/response pair and the golden snapshot for values that would change > if this same app were re-hosted or re-seeded, hostnames and URLs, auto-generated IDs,> timestamps, anything environment- or run-specific. List them.> In my run this surfaced values that had slipped through normalization, timestamps and the app's> own URLs, which embed the host and a raw id, each a false "regression" waiting to fire against the rewrite.
  2. A guard test. Turn that scan into a test that fails the build if such a value ever reappears,
    so "is this actually retargetable?" is answered mechanically on every run, not from memory.
The meta-lesson for any rewrite oracle: green on the original only proves the yardstick
agrees with the original. It says nothing about whether it's implementation-independent, and you can't discover a cross-target leak by testing on a single target. Make it a check you run, or better, a test that runs itself.

Make it trustworthy

Don't treat these as a manual checklist you run at the end, write them into the spec as
requirements, so the agent builds them in and your review just confirms them (they're all in the
appendix brief). Three checks earn the trust:
  • Green on the original first. Run the suite against the current app, the one you recorded the behavior from, and confirm it fully passes before you trust it anywhere else. If it's red
    there, your recording is wrong, not the app.
  • A committed coverage number. Publish what fraction of the contract is actually exercised; anything unexercised is a behavior you can't vouch for.
  • Determinism. Seed, snapshot, snapshot again, byte-identical, or your "differences" are just
    noise.
(Plus the two you've already met: the retargetability guard from the last section, and the
property-based tests the spec generates to pin the harness's own logic.)
And be clear about what it doesn't do:
  • Coverage is method-level, not every branch, a method is called and its response frozen;
    paths behind filters or optional params may go untried.
  • The retarget is unproven until you retarget, it's built stack-agnostic but has only run
    against the original; the real proof is the shadow run against the rewrite, a later post.

What it cost

Building the harness, the yardstick this whole series rests on, cost about 389 Kiro
credits (Kiro Spec mode, running on the Claude Opus 4.8 model).

The bottom line

The recipe, with a concrete example of each step from the Kanboard run:
  1. Pick the boundary your app is contracted at, for Kanboard, its JSON-RPC API plus the
    browser UI.
  2. Enumerate the contract from source into a flat checklist, parsing the API classes yielded
    160 methods; that list is your coverage denominator.
  3. Drive an AI agent through a spec workflow (requirements → design → tasks → execute) and review every phase, my review is what caught the permission-bypassing token and the "cover every method" gap.
  4. Build three layers, a golden-master snapshot, 102 API/DB-state request/response probes,
    and 93 UI flows.
  5. Make it black-box and retargetable, one env var (KANBOARD_BASE_URL) aims the identical suite at PHP today or Python tomorrow, and a guard test fails the build if any app code sneaks in.
  6. Normalize for replay and scan the frozen artifacts for host/ID/timestamp leaks, the
    scan that surfaced 43 of them before they could fire false regressions. Don't skip it.
  7. Prove it green, publish the coverage number, and note the gaps, 159/160 API, 93/103 UI, 77/104 features (74%), green on the current app, gaps documented.
Behavioral spec plus the boundary plus an agent that iterates against a real app gets you a yardstick that outlives the code it was recorded from. And if that yardstick doesn't exist yet, building it is the project.

Try it yourself

Start with the short prompt above to get moving, or jump to the full prompt in the appendix
below, drop in your app, paste it into your agent's spec mode, and review each phase (that's
where your app's traps surface). Then point it at your app and watch it build your yardstick.
Next Part: I point AWS Transform custom at Kanboard to author and run a PHP → Python
transformation skill, with this harness wired in as Transform's own validation command, so
every divergence gets caught as the rewrite happens, not after. Then the final post: Kiro fixes
what's left, the last mile, and reports the final coverage number the harness produces.

Appendix, the prompt

I originally ran this as three specs, the harness, then an API-coverage expansion, then a
UI-coverage recovery pass. The single prompt below folds all three into one, including the
UI-driven data-state technique that recovers the event-driven and web-only features the API can't reach, so asking for comprehensive, feature-level coverage up front means you won't need the follow-up passes. Replace the <PLACEHOLDERS>, paste it into your agent's spec mode, and
review each phase, the domain traps are yours to catch.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
Build a black-box, stack-agnostic behavioral test harness for <APP + VERSION> that captures its
current observable behavior as frozen, committed artifacts and REPLAYS that behavior against a
future re-implementation. The same suite must run UNCHANGED against both the current app and the
rewrite, so talk to the app ONLY over the network boundary (<HTTP API: REST / JSON-RPC / gRPC / ...>
plus a browser for the UI), never import app code, and target a configurable base-URL env var.

Be self-sufficient — discover, don't wait to be told:
- Enumerate the full API contract by reading <WHERE THE API LIVES: route files / an OpenAPI or proto
spec / a controllers dir> into a committed inventory (every endpoint + its params). That inventory
is your API-surface checklist — necessary, but NOT the whole coverage denominator (feature coverage
below is).
- Extract the persistence schema by introspecting the live data store into a committed artifact.
- Read the app's auth / models / validators to learn side effects, error conditions, and how auth
and permissions really work (common trap: an admin/service token BYPASSES permission checks, so
permission tests are meaningless under it — use a real per-user login too).

Cover every FEATURE, not just every endpoint: the harness is the sole oracle AND it later DRIVES the
rewrite, so any feature it doesn't exercise is one the rewrite won't be pushed to build. Best source
for the feature list is a prior analysis pass — run a business-rules / domain extraction over the app
first (e.g. AWS Transform's business-rules-extraction) and feed its feature inventory in as the
checklist; otherwise enumerate the feature set from the app's controllers / models / actions / jobs.
Endpoint + UI coverage is necessary but NOT sufficient — many features have no read-API: event-driven
dispatches, background or scheduled jobs, audit/log writes, rule or automation side effects — anything
that persists a result the app never returns through a query. Cover those by asserting their SIDE
EFFECTS, and where there is no read-API, read the data store directly (READ-ONLY) and freeze the
normalized rows ("test the data, not the objects"). CRITICAL TRAP: many of those side-effects never
fire on the API path at all — an API call can bypass the event/subscriber layer that produces them —
so hitting the endpoint proves nothing. To reach them, DRIVE THE TRIGGERING ACTION THROUGH THE UI
(the real entry point that fires the subscriber), setting any UI-only precondition first (e.g. enable
a setting or subscribe a user through the settings pages), THEN assert the persisted rows it wrote.
So before scoping out any feature as "no API method," TRY THE UI PATH — most features the API can't
reach are drivable through the browser; keep a scope-out only when the control is genuinely
headless-hostile (drag-only / no accessible name) or the feature is a config-fixed external or
CLI-only dependency. "Done" = every feature has a green replayable test or a justified exclusion — not "every
endpoint returned 200."

Three layers:
1. Data golden master — seed a DETERMINISTIC dataset via the API; snapshot the full read-back state;
normalize it (strip volatile fields, resolve IDs to stable names, sort keys); freeze it; diff
future runs against the frozen master.
2. API request/response probes — exercise EVERY method in the contract inventory, or record an
explicit justified exclusion. Cover success, multi-step workflows, permission denials, and error
cases. Freeze each request/response pair with the request in SYMBOLIC form (entities by stable
name, resolved to real IDs at run time) and the response NORMALIZED, so replay against a freshly
seeded rewrite differs only for genuine behavior changes.
3. UI behavioral E2E — COMPREHENSIVE and inventory-driven, not a hand-picked few: enumerate EVERY
user-facing journey into a committed inventory (grouped by area), then cover each with a flow OR
a justified exclusion. Each flow drives a real browser, authenticates, uses SEMANTIC locators
only (accessible role / label / text / placeholder / title / test-id — never CSS / DOM / XPath),
and asserts a user-observable outcome (an item appears, disappears, or a value changes). Wait on
observable conditions — visible on appearance, hidden on removal — NEVER a fixed sleep. Gestures
(click, select, file-choose, drag) are fine as long as elements are LOCATED semantically; if a
gesture or an unnamed widget can't be driven reliably headless, exclude it with a reason rather
than ship a flaky flow. Beyond the visible outcome, where a feature's real effect is a PERSISTED
ROW an event subscriber writes (audit / activity / notification / state-transition), also assert
those rows in the data store (READ-ONLY) right after the UI action — a UI-driven data-state
assertion — since the visible outcome alone doesn't prove the side-effect fired. Scope every
data-state read to entities THIS test created (use a dedicated throwaway user for user-scoped
effects like audit / notifications) so it is order-independent and replay-safe, and normalize the
rows before freezing.

Cross-cutting:
- Replayability guard: add a test that scans every frozen artifact for values that change on
retarget — the app's own host/URLs, raw auto-generated IDs, wall-clock timestamps — and fails the
build if any appear (they leak silently because a same-target run never exposes them).
- Allowed differences: for responses that legitimately vary across implementations (versions,
timestamps, generated catalogs), assert shape/type only, not the exact value.
- Black-box boundary: put source-reading discovery in a SEPARATE authoring package that runs once to
emit committed artifacts; add a static guard test that fails if any run-time module imports it or
references app source.
- One runner that runs all layers, exits non-zero on any failure OR coverage gap, and writes a
FEATURE coverage report (features covered / excluded-with-reason / total — THE primary measure),
plus the API and UI coverage reports, as committed artifacts.

Baseline everything GREEN against the current app before declaring done. Report honest coverage as
FEATURE coverage FIRST (every feature covered / excluded-with-reason / total) — not just API and UI
coverage, which can read ~100% while real feature coverage is far lower (endpoints and screens don't
touch event-driven, background, or side-effect-only behaviors). If a feature has no
test and no justified exclusion, that's a gap that fails the run. Write down the caveats.
A note on how this prompt got here, my honest path. I didn't start with that "cover every
feature" framing. My first harness looked comprehensive, near-100% of the API and a full set of UI journeys, so I trusted it. It wasn't: it only exercised ~42% of the app's actual features, because whole areas (notifications, analytics, audit logs, permission restrictions, automation) have no read-API and I'd only measured endpoints and screens. I found out the hard way, the rewrite the harness drove came back at ~42% too (an agent builds exactly to its gate). What finally gave me the complete picture was AWS Transform's business-rules-extraction, which I'd run later for the rewrite: its domain/feature inventory was precisely the coverage checklist my harness had needed up front. So the lesson baked into the prompt above: run the analysis first and feed its feature inventory in as your coverage denominator, BRE isn't just rewrite-prep, it's the map that makes your safety net actually complete (and it grounds the rewrite skill later, so you run it once and use it twice).
# aws-transform# kiro

Series: Monolith Modernization - Kiro+AWS Transform (3 articles)

  1. 1
    Build the yardstick before you rewrite the monolith This article
  2. 2
    Transform the monolith, validated against the yardstick every step
  3. 3
    Kiro closes the last mile and finishes the migration
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article