How I Used Claude Code to Make an Autonomous Coding Harness Reliable
A weaker autonomous agent doesn't need a benchmark to get good — it needs a stronger agent to drive it against real work, watch it fail, and fix the root cause in the repo. Here's the loop I used, why gates beat prompts, and a worked example.
Agents Training Agents: How I Used Claude Code to Make an Autonomous Coding Harness Reliable
I've been building an autonomous coding agent — a harness running on Amazon Bedrock AgentCore that you trigger from a GitHub issue. You apply a label or
@ -mention it, and it clones the repo, makes a change, verifies it, and opens a pull request. No self-hosted runner, no laptop left on overnight. (I wrote up how that pipeline is wired in @agent in Your GitHub Issues, No Runner Required .)Here's the part I didn't expect to be the interesting part: the harness didn't start out able to do any of that reliably. The first several runs were a parade of confident failures — PRs that didn't compile, runs that died on a transient model error, a job that ingested 59,000 lines of lint output and blew past the context window.
What fixed it wasn't a better model or a benchmark suite. It was driving the weaker agent with a stronger one — Claude Code — in a tight loop: assign real work, watch it fail, fix the root cause in the repo, redeploy, assign again. The trainer has capabilities the trainee doesn't have yet — it can read the whole codebase, run deploys, inspect infrastructure, and, crucially, edit the trainee's own definition (system prompt, state machine, CI). Training is just the stronger agent using those capabilities to close the gaps each real task exposes.
This post is that loop, why it works, and a worked example where you can watch the failure rate walk down to zero.
The loop
Plain text
┌─────────────────────────────────────────────────────────┐│ 1. Assign a REAL task to the trainee agent ││ ││ 2. Observe the full run, not just the verdict ││ (labels, the final comment, the PR, CI, logs) ││ ││ 3. Diagnose the ROOT cause of any failure or low quality ││ (not the symptom — the reason the symptom was possible)││ ││ 4. Fix it in the repo the trainee runs in ││ (system prompt, pipeline, CI gates, tooling, docs) ││ ││ 5. Redeploy so the fix is live for the trainee ││ ││ 6. Assign another task → back to 1 │└─────────────────────────────────────────────────────────┘
The shape is deliberately close to how a senior engineer brings a junior one up to speed — except the "junior" is an agent, and the "feedback" is a permanent change to the environment it runs in, not a comment it might forget.
Why real tasks, not a benchmark
Every failure I found came from giving the harness an actual issue from the repo's backlog, not a synthetic test. Real tasks surface real failure modes in the order they actually bite:
- A transient model error only matters if a real run is long enough to hit it.
- A verification gap only shows up when a task is complex enough to get the code subtly wrong.
- A context overflow only happens on a task whose verification produces huge output.
A benchmark you design up front tests the failures you already imagined. Real tasks test the ones you didn't — which are exactly the ones still costing you. The failure I'd never have written a test for (59k lines of lint output eating the context window) was the one that taught me the most.
The three kinds of fix — and why gates beat prompts
When the trainee failed, the fix landed in one of three places. They're listed in rough order of durability, and the ordering is the whole point.
- A hard gate that makes the failure impossible (best). Example: a
pull_requestCI check that blocks merging code that doesn't type-check. The trainee can't talk its way past a gate.
- Guidance in the trainee's definition (good, but soft). Example: the system prompt telling it to run the type-checker before opening a PR, and never dump huge command output. A prompt reduces the failure rate; it can't guarantee zero.
- A more robust environment (foundational). Example: retrying transient model errors in the orchestration layer, or fixing a broken deploy so the trainee can run at all.
Prefer a gate over a prompt. A prompt asks the agent to behave; a gate makes misbehavior fail loudly and cheaply. This is the single most important thing I learned. Early on I kept reaching for "tell it harder in the system prompt" — and the agent kept ignoring the instruction on the runs where it mattered. An LLM instruction is a probability nudge, not a guarantee. When all you have is a prompt, assume it will sometimes be ignored, and put a gate behind it as soon as you can.
A worked example
The reliability push was one long turn of this loop. In order:
Round | Task | What broke | Root-cause fix | Kind |
|---|---|---|---|---|
1 | Publish config via infra-as-code | Run died on a transient model 424 | Retry the transient exception in the orchestration layer | Environment |
— | (observed) | Raw model tokens leaked into PR comments | Strip control tokens before posting | Environment |
— | (observed) | One-line system prompt gave no engineering discipline | Rewrote it: investigate → branch → verify → PR | Guidance |
2 | Re-assigned round 1's task | Deploy itself failed on fresh branches | Auth pointed at a dead identity pool; fixed it | Environment |
— | (observed) | Harness couldn't run the package manager | It wasn't on PATH after install | Environment |
3 | Re-assigned again | Opened a PR that didn't compile and claimed success | (a) CI gate blocking non-compiling merges; (b) prompt hardened into a blocking "type-check before PR" step | Gate + Guidance |
4 | Fix a noisy-lint issue | Run overflowed the context window (1.16M > 131k tokens) — it ran lint and ingested 59k lines | Prompt rule: never dump large command output; pipe through tail / grep | Guidance |
5 | Add a docs file | Nothing. It ran the type-checker, reported honestly, opened a clean, minimal PR | — | ✅ |
By round 5 the harness produced a correct, verified, minimal PR on its own — the behavior I was training toward.
Round 3 is the one to sit with. The fix there wasn't "tell it harder to verify" — a prompt had already asked, and been ignored on the exact run where it mattered. The durable fix was a gate (CI that blocks a non-compiling merge) with the prompt as a softer second layer. Ask-then-enforce, not ask-and-hope.
There's a nice bit of poetry in round 3, too: the trainer's own work had to pass through the trainee's gate. When I first switched the CI gate on, the gate itself was broken (a missing generated file, wrong Node version), and the first real PR through it caught that. The gate you build to discipline the trainee disciplines you too.
Gotchas I learned the hard way
- The prompt is deployed, not live-edited. Editing the system-prompt file changed nothing until the stack redeployed and the harness resource actually updated. If you don't confirm the deploy touched the harness, you're testing the old prompt and drawing conclusions from a ghost.
-
SUCCEEDED≠ success. My orchestration caught failures into a "post a failure comment" path that itself succeeded — so a caught failure still reported the overall execution asSUCCEEDED. Distinguish the real success path from the caught-failure path explicitly, or your dashboard will lie to you.
- The trainee will confidently claim work it didn't do. It reported "docs updated" when they weren't, and opened a non-compiling PR calling it done. Never trust the final comment — verify the artifact (the diff, the CI result). This is why gates matter: they check the artifact, not the agent's self-report.
- A gate you add can itself be broken. Test a new gate on a trivial PR before trusting it to protect real ones. A false-failing gate is worse than no gate.
- Self-referential test tasks are a trap. Assigning the "noisy lint output" issue caused the context overflow because the task was about noisy output. Don't test an agent by handing it the exact weakness you're unsure about.
- Clean up test artifacts. Runs create branches, PRs, and labels. Close and delete them, or your backlog quietly fills with noise.
The ladder
Here's why "agents training agents" is more than a cute framing. The loop scales up a level.
Once the harness reliably produces good PRs, it can be handed issues that improve its own definition — a well-scoped prompt or pipeline change is just another coding task. And an agent that can reliably improve one agent can be pointed at standing up and hardening other agents. The trainer role I played by hand becomes a role the trainee can play for the next agent down the line. Each rung raises the floor for the next.
The invariant that keeps the ladder safe at every rung is the same lesson from round 3: gates over promises. An agent training another agent should leave behind hard checks — CI, verification steps, honest-reporting requirements — not just better instructions. That way each new agent inherits a floor it cannot fall below, no matter which agent trained it. Better prompts make an agent more likely to behave; gates make the whole ladder trustworthy, which is the only property that lets you stack another rung on top.
Takeaways
- Drive weak agents with strong ones against real work. Real tasks surface the failures a benchmark can't imagine.
- Diagnose the root cause, not the symptom. "It failed on a 424" → why wasn't that retried? Fix the reason the symptom was possible.
- Prefer a gate over a prompt. A prompt asks; a gate enforces. Ask-then-enforce.
- Never trust the self-report — verify the artifact. Agents claim work confidently. Gates check the diff and the CI result, which don't have opinions.
- The floor you build for the trainee is what makes the next rung safe.
Related reading
- @agent in Your GitHub Issues, No Runner Required: Webhooks + Step Functions + Bedrock AgentCore — the harness pipeline the trainee runs on
- Wiring CopilotKit to a Managed Bedrock AgentCore Harness — putting a UI in front of a managed AgentCore harness
- Watching Claude Code Work: Streaming Agent Telemetry to CloudWatch and Rebuilding the Conversation — how I observed the runs I diagnose above
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article