Building an AI layer for Penetration Testing Orchestration without breaking the safety model.

Raisins wraps standard pentest tools and organizes their output. It is a human-in-the-loop orchestrator: it blocks active scanning until an authorization is recorded and the target is confirmed in scope, it never autonomously exploits, and it invokes tools with argument lists rather than shell strings.

I wanted an AI layer on top of that: something to suggest next steps, triage findings, draft report prose, and even drive a bounded plan→act loop. The risk was obvious from my research from already existing tools that use MCP servers in the market. An AI that can propose and run actions is exactly the kind of thing that can quietly grow a second, ungated path to a target, fabricate findings, or leak engagement data. The whole point of Raisins is that it does not do those things. So the real design question was not "how do I add AI" but "how do I add AI while keeping every existing safety property intact."

I built this spec-first: requirements, then design, then a task list, executed in waves with checkpoints between them. That structure paid off because the safety properties were written down as requirements before any code existed, and the design consolidated dozens of individual acceptance criteria into a small set of invariants I could actually test.

The waves roughly followed the feature phases:

Each wave reused existing machinery rather than adding parallel implementations, which is what kept the safety surface from growing.

Bedrock Converse + Meta Llama 3.3 70B Instruct, not Claude. I used AWS Bedrock's Converse API as a plain text-generation endpoint, defaulting to Meta Llama 3.3 70B Instruct. The driver is a balance of cost and quality: it is inexpensive relative to frontier models (good mileage against a fixed AWS credit budget) while giving materially better reasoning/instruction-following than the smallest Bedrock models, and it handles legitimate security-testing terminology without the aggressive over-filtering some smaller hosted models apply. The provider seam makes the model a swappable detail, so an operator who wants Nova, Mistral, DeepSeek, Qwen, or another Bedrock model just changes config.

Not Bedrock Agents or AgentCore. Those ship the agent loop, memory, and tool-invocation to a cloud runtime. That directly fights Raisins's local-agent design, where the loop, the memory, and every safety decision must stay on the operator's machine and under the operator's gate. So I deliberately use Bedrock only as a text API. The agent loop is local Python; the safety gate never leaves the host.

Off by default. The AI config resolves to be disabled unless explicitly enabled, and the enabled flag is additive on the engagement document, so pre-AI engagements load unchanged. Nothing imports boto3 and nothing calls Bedrock until AI is both enabled and a feature is invoked. With AI off, the platform behaves byte-for-byte as it did before.

CLI-first chat. Raisins is CLI-first by design, so chat is a REPL, not a new UI surface. It is advise-first: explanations are ungated and record nothing.

Advise, then act, and only act through the gate. The agent can propose actions, but it can only perform them through a single bridge that maps an intent onto the platform's existing operator functions. That is the crux of the safety story.

Three ideas do most of the work.

A single chokepoint. AgentActions.perform is the only path from an AI intent to a real action. It routes through the existing workflow.run_phase / workflow.run_suggestion, which call workflow._guard → SafetyGate.check. So the gate applies by construction. There is no alternate executor: perform never calls subprocess, run_tool, or Popen, and never re-implements the gate. The agent's repertoire is exactly the operator's capabilities. Out-of-scope always wins, and a gate block is recorded before it raises.

No fabrication. AI command suggestions are never emitted as free-form text. Each idea is mapped to a real tool, and the command line comes from that tool's real build_args; an unmappable idea becomes a labeled MANUAL: note that cannot be run. Triage and narrative derive only from recorded state, and triage references are resolved back to findings that actually exist. AI content is always attributed (ai_generated plus the model id) and never stamped as operator-recorded or tool-parsed.

Trusted vs untrusted separation. The persona, grounding rule, ethical binding, and untrusted-content rule live in the Converse system slot. Everything else, tool output, target text, chat input, and prior turns, is framed as labeled data blocks inside user-role messages. Because no untrusted text enters the system slot, the ethical binding cannot be unbound by injected content or history. Redaction of secret-designated content is the last step before anything reaches the provider.

Property-based tests for each invariant. These safety properties are not just asserted in prose. Each of the 18 correctness properties has a Hypothesis test that exercises it across generated engagements, transcripts, budgets, and scripted (sometimes adversarial) provider outputs.

The suite has roughly 1819 passing tests. The property tests are the backbone: disabled-is-inert, mentor output preserved and ordered ahead of AI, no fabrication, every proposed active action hitting the gate as a prepared action (including when the "action" is smuggled in via injected content), no secret value reaching the provider, attribution present and operator/tool provenance absent, recorded state never mutated, the kill-switch honored at every step, trusted/untrusted separation, transcript round-tripping, and bounded deterministic memory.

Around those, a FakeProvider gives deterministic, network-free end-to-end runs and records exactly what was sent so tests can assert what was not sent. A mocked-boto3 test checks the BedrockProvider request shape and response parsing. Static structural guards assert the discipline holds: boto3 only inside provider.py methods, and no subprocess / run_tool / second SafetyGate anywhere under ai/. Finally, the live Bedrock round-trip was smoke-tested with Nova Micro, since model behavior over the wire is not something property tests are meant to cover.

Enabling AI is a real privacy tradeoff. When AI is on, redacted engagement context and chat text go to AWS Bedrock. AWS states it does not train on those inputs, but the data still leaves the machine. Off-by-default exists precisely so an operator can keep the fully-local behavior for sensitive engagements by simply not turning it on. And active scan traffic is attributable from the operator's IP regardless, which is outside the tool's control.

Local models were not viable on the hardware. A fully local model would have avoided the data-leaves-the-host tradeoff, but it was not practical on the weak hardware this was built on. Bedrock plus a cheap model was the workable answer, with the provider seam left in place so a local backend could slot in later.

The gate did the heavy lifting. The most reassuring part of the build was how little new safety code we wrote. By routing everything through the existing gate and existing operator functions, the AI layer inherited the platform's guarantees instead of re-deriving them. Adding capability was mostly a matter of not adding a second path.

I am still building and debugging before I release and share my repo and build sooner or later this month .