An AI wrote the security fix. Here's why we still don't let it merge.
Pre-Prod-Orchestrator scans a GitHub diff, asks a model for a patch, verifies it with deterministic checks and tests, and opens a pull request for human review.
Your scanner flags a SQL injection in a pull request.
Now what?
Someone has to find the line, write a fix, check it, run the tests, and open a PR. Most of that is repetitive. The one part that isn't is deciding whether the fix is actually safe.
We built Pre-Prod-Orchestrator to automate the repetitive parts and keep a human on the last step.
It connects code-change detection, remediation, verification, GitHub workflows, and a web control plane into one pipeline. It is still evolving from separate components into a single integrated workflow.
The one rule we built around
AI proposes. Deterministic checks decide. A human merges.
Detection, remediation, verification, and Git operations are separate stages. A generated patch is only a candidate. It becomes a pull request after it passes an exact-match check, a security re-scan, and the repository's tests.
Here is the full flow:
GitHub PR / Push
↓
Change Capture
↓
Security Scanner (DiffParser, RuleEngine, Rule Registry)
↓
Security Finding
↓
AI Remediation (Context, Prompt, Candidate Patch)
↓
Exact Patch Check
↓
Security Re-scan
↓
Repository Tests
↓
GitOps (Branch → Commit → Pull Request)
↓
Human Review
Step 1: A scanner that works without a model
The scanner is a Python static analysis engine.
It reads:
- structured JSON from Pre-Prod Tester
- unified diffs
- Git commit ranges
- repository contents
Every input follows the same path:
Input
↓
DiffParser
↓
RuleRegistry
↓
RuleEngine
↓
RuleHit
↓
Finding
↓
ReportBuilderThe rule registry covers injection, secrets and credentials, cryptography, authentication, access control, deserialization, network security, configuration, information disclosure, API security, logging, and filesystem security.
The scanner can run with no LLM at all:
model = noneLLM enrichment is an optional layer on top of this baseline.
Why it matters: if the model is unavailable, your scanner should still behave the same way every time.
Step 2: Ask the model for edits, not files
Detection gives you a finding. It doesn't give you a safe patch.
So the remediation component extracts a focused section of the vulnerable source and sends only that context to the model. The model returns structured edits instead of a block of rewritten code:
{
"edits": [
{
"file": "src/auth/login.js",
"start_line": 42,
"end_line": 43,
"original": "...",
"replacement": "...",
"explanation": "..."
}
]
}Before applying an edit, the system checks that
original matches the current file contents.If it doesn't match, the edit is rejected.
This catches a common failure of AI patches: the model believes it is editing one version of the file while the repository has already changed.
Step 3: A patch that applies is not a fixed bug
A patch applying successfully does not mean the vulnerability is gone. So the candidate goes through more stages:
AI-generated candidate
↓
Exact source verification
↓
Patch application
↓
Security re-scan
↓
Repository tests
↓
Remediation PRThe re-scan runs against the resulting change and answers two questions:
- Is the original issue still there?
- Did any new HIGH or CRITICAL finding appear?
Only after that does the GitHub automation act. It never modifies the developer's own branch. It creates a dedicated one:
security-remediation/<sha>Then it commits the change and opens a pull request. The merge stays a human decision.
The team gets a reviewable change with evidence of how it was produced and tested, instead of an AI answer sitting in a dashboard.
Step 4: Pre-Prod Tester feeds the pipeline
Pre-Prod Tester is a companion GitHub Actions workflow. It captures what changed between two commits and outputs structured scan data. A simplified payload contains:
Repository
Base commit
Target commit
Changed files
Diff hunks
Changed lines
Workflow informationThe scanner gets a clean description of the change instead of rebuilding the Git diff on its own.
The data is also published as scan evidence, so the frontend can read results without executing untrusted repository code.
Step 5: A control plane you can review in
The Next.js frontend has views for findings, repositories, remediation, validation, pull requests, audit events, reports, settings, and dashboard metrics.
It runs in two modes:
- Demo: runs an interactive security scenario with no external infrastructure.
- Live: shows real scan results and GitHub workflow data.
We kept them strictly separate. A simulated finding should never look like a real one.
What went wrong
The project grew across several components and branches:
Scanner
Orchestrator
Remediation
Platform API
GitHub Actions
Frontend
Pre-Prod TesterSome of them ended up with overlapping implementations. The next challenge was not another feature. It was getting every piece to agree on contracts, states, and responsibilities.
What helped most was a strict schema,
finding.schema.json . Instead of each component inventing its own idea of a vulnerability, all of them now share one normalized format.
What we'd tell you if you're building this
1. Treat model output as a candidate.
A model can propose a fix. Deterministic checks and tests decide if it's safe.
2. Write the shared schema early.
When several people work on different subsystems, one contract saves a lot of integration pain.
3. Treat repository code as untrusted.
The platform receives arbitrary source and diffs. None of it should become instructions for the platform itself.
4. Keep demo data and real data apart.
A convincing dashboard is useless if simulated findings can show up as real results.
5. Keep the evidence.
"The vulnerability was fixed" is not enough. You should be able to trace it:
Finding
↓
Candidate patch
↓
Patch verification
↓
Security re-scan
↓
Tests
↓
Commit
↓
Pull Request
What's next
We're connecting the pieces end to end, so the platform consumes the scanner's real output and findings become first-class objects in the workflow.
The target flow:
Pre-Prod Tester
↓
Platform
↓
Security Scanner
↓
Finding
↓
Remediation
↓
Validation
↓
Re-scan
↓
Evidence
↓
GitHub Pull Request
↓
Human ReviewThe goal is not to remove developers from security review. It is to make the path from "we found a vulnerability" to "here is a verified change, ready for review" repeatable.
Try it
The project is open source. The repo has the scanner, remediation engine, GitHub Actions workflows, contracts, and the frontend control plane.
If you're building security automation with AI, start with the contract and the scanner before you add more model-driven behavior.
Repo:
github.com/dis-craft/Pre-Prod-Orchestrator If you build something similar, tell me in the comments: where do you draw the line between AI-generated fixes and deterministic verification?
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article