My bug reproduction agent trusts the container, not the model
A build log on PatchPilot, an agent that reproduces GitHub issues in Docker and generates a failing test and a fix diff, and what deterministic container verdicts, untrusted issue text, and fixture curation taught me while building it.
Software Engineer
The most expensive sentence in open source maintenance is "can't reproduce." A maintainer reads an issue, tries, fails, and closes it. The reporter did real work writing it up and it dies there. PatchPilot is my attempt at that gap. You paste a GitHub issue link, and the agent reproduces the bug in Docker, then generates a failing test and a fix diff. Version one covers Python, C++, and JavaScript.
I am writing this mid-build. Phase 3, the reproduction loop, is not at its gate yet, and the live Docker runs are still ahead of me. What follows is what the build has already taught me.
The verdict comes from the container, not the model
The first decision I made, back in Phase 0, was deterministic-only verdicts. The model reads the issue and writes a reproduction script, and that is its whole job in the loop. Whether the bug reproduced is decided by what happened inside the container: the exit code, the output, whether the quoted error appeared. The model never gets to declare success.
Models are agreeable. Give one a broken script and it will find a way to tell you it worked. Anchoring the verdict to something observable removes that failure class, and it makes the verdict debuggable: when a run says reproduced, I can point at the exact output line that earned it.
Issue text is untrusted input
A GitHub issue is written by a stranger, and the obvious failure is letting the issue text steer what runs. We ran a malicious-issue drill in Phase 3, with issue text crafted to act as a command. It passed because the pipeline never lets issue text near the executor. The issue is data the model reads. The commands that run are built by my code from a fixed template. That separation is the whole defense, and it has to hold by construction.
I had learned the same lesson on PRISM, where commit messages feed the intent analysis and a poisoned commit could steer the output. Untrusted text enters a pipeline, and somewhere downstream something treats it as instructions. Fence it where it enters, or you will fence it everywhere else forever.
The fixture suite caught the agent lying to itself
Before trusting any reproduction result, I curated 10 real Python issues as fixtures, each verified twice: the quoted error present at the base commit, and gone at the fix commit. That double check caught a real defect in my own code. The agent was cloning HEAD instead of the base commit, so repro runs passed against code that already contained the fix. Every "reproduced" result before that fix was fiction, and it looked green.
The same work turned up two more defects: boto3 was never pinned or installed, so the Bedrock path would have failed the moment it ran for real, and a torn-read race in the job store, which only shows up when a job is being written and read at the same time. All three came out of fixture curation, not out of staring at the agent code. The fixtures, saved in scripts/fixture_issues.json, now double as the model shootout. The same 10 issues run on Claude Haiku 5.5 and on Qwen3-Coder through Bedrock, and whichever wins becomes the default. Same issues, same verdicts, count the wins.
Cheap checks before expensive ones
Syntax checks run before any container spins up, so a repro script with a syntax error never reaches Docker. It sounds trivial, and it saves real money, because container runs are the slow, expensive step, and most broken scripts are broken in ways a parser catches in milliseconds. The same instinct runs through the whole design. One job at a time, two runs per IP per day, three globally, because an open demo that runs Docker for strangers is a free compute giveaway if you do not cap it first. The ceilings went in during Phase 2, before the reproduction loop existed, and the host is an EC2 box paid for with AWS Activate credits. The bill stays boring because the limits were decided before the first public run.
What I would do differently
Fixtures first. I built the agent, then built the fixtures that proved it wrong, and they should have existed on day one of Phase 3. Without them, every green run is a rumor.
Second, decide the verdict contract before choosing the model. I settled the model question, Bedrock primary with Haiku against Qwen, after the loop existed, and the contract should have come first. The model is interchangeable. The verdict is the product.
Third, one thing I got right: the abuse ceilings went in during Phase 2, before the first public run, so I can leave the demo open without watching it.
The honest state of it
Phase 1 and Phase 2 are approved and live. Phase 3's core is built at 79 of 79 tests, ruff clean, CI green. But the gate needs the live runs: deploy the Phase 3 backend to the EC2 host, run the fixture suite with real Docker on Haiku and then on Qwen, exercise every verdict path, and run the malicious-issue drill on the host. Until those runs exist, this is a build log, not a victory lap. The mistakes are fresh, the fixes are testable, and the next person building one of these gets the warnings first.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article