We spent four days building a tool that refuses to say "this looks risky"
Pramaan — how we built an evidence-first codebase verifier on one EC2 box, and every wall we hit doing it.
17–20 September 2026. Two engineers, one hackathon.
The sentence that started it
On the evening of the 17th we were arguing about a static analysis report one of us had been handed at work. It had flagged 340 issues. Maybe nine mattered. Nobody on that team could tell you which nine without reading all 340.
That is the actual problem. Not "we don't have enough analysis." We have infinite analysis. The problem is that static analysis produces hypotheses and presents them as conclusions, and a hypothesis you can't check is just anxiety with line numbers.
So we wrote down one rule and built everything behind it:
A risk is never born confirmed. Only runtime evidence can promote it.
We named the project Pramaan — pramāṇa, the Sanskrit term for a valid means of knowledge, the thing that separates a claim from a proof. It felt less pretentious at 11pm than it does in a blog post, but it stuck, and it kept us honest every time we were tempted to ship a heuristic as a verdict.
Why "it looks risky" is not good enough
Every engineer has a story about the bug that was visible in the code for two years and still took production down. A few that shaped what we built:
Cloudflare, 2 July 2019. A single regular expression with catastrophic backtracking, deployed globally, pinned CPU across the fleet and took down a large fraction of the internet for roughly 30 minutes. The regex was right there in the diff. Most static analyzers would rank it as a low-severity style nit — unless something actually measured CPU under that input.
GitLab.com, 31 January 2017. An engineer removed the wrong database directory during an incident, and then discovered that five separate backup mechanisms had all been silently failing. The gap wasn't knowledge. It was verification — nobody had ever executed the restore path.
Roblox, October 2021. A 73-hour outage that traced to a Consul streaming feature interacting badly with BoltDB write contention, under a load pattern that only appeared at their scale. No amount of reading the code finds that. You find it by putting the system under load and watching.
Knight Capital, 1 August 2012. $440M in 45 minutes, because dead code was reactivated on 7 of 8 servers by a partial deploy. The defect was a runtime configuration fact, invisible to any repository-level scan.
The pattern is consistent: the code was readable, and reading it was not enough. The cost of these events isn't in the finding — it's in the not-knowing-which-finding.
That's the thesis. Now the four days.
Day 1 (18 Sep) — clone it, and don't run it
First commit:
added vm + github clone .Simple-sounding. Two decisions in it that we never revisited:
1. Analyzed repositories are untrusted input. Not "probably fine because it's public." Untrusted. That means the analyzer never runs
npm install , never executes a package script, never triggers a postinstall hook, never starts the target app. It parses files with tree-sitter, in-process, and that's it. If you've ever looked at what a malicious postinstall can do — the event-stream incident in 2018 shipped wallet-stealing code to millions of downloads through a transitive dependency — you know why this line matters.2. No
shell=True , ever. Every subprocess is an argv list. Every file read resolves against the clone root to kill path traversal. We only accept GitHub HTTPS URLs.Those two rules cost maybe 40 minutes of ergonomics over four days and removed an entire category of embarrassment.
Day 2 (19 Sep) — the App Map, and the hash that keeps us honest
added repo analyzer and app map visualization , then fixed analyser to generate probe manifest .The App Map is the canonical artifact: routes, handlers, tables, queries, transactions, outbound calls, connection pools, auth guards, frontend pages, forms, existing tests, the graph edges between all of it, and deterministic risk flags — each traced to a file and a line number.
Detectors live one-per-concern in
backend/app/analyzer/detectors/ : routes.py , handlers.py , db_prisma.py , db_sql.py , db_mongoose.py , transactions.py , outbound.py , pools.py , auth.py , frontend.py , branches.py . Adding a framework means adding a file, not touching a pipeline.The determinism argument we had
Here's where we nearly built the wrong thing. The obvious hackathon move is: throw the repo at an LLM, ask what's risky, render the answer beautifully. It demos great. It is also unfalsifiable, unreproducible, and different every time you run it.
We landed somewhere stricter:
- Everything that carries a verdict — stack, routes, handlers, tables, queries, transactions, outbound calls, pools, guards, pages, forms, edges, and all risk flags and their severities — is computed by static analysis and hashed into
metadata.canonical_hash. Analyze the same commit twice, get the same hash. If that hash moves and the commit didn't, we have a bug.
- The LLM may only append to
invariantsanduncertainties— the semantic claims static analysis genuinely cannot express ("orders must not ship before payment settles").
- Every LLM claim is re-checked by a deterministic validator: does the file exist, are the lines in range, is the symbol known, is the node real. Invalid output is retried at most three times with the validator's specific complaint fed back, then recorded as
UNCERTAIN.
- Confidence is derived from evidence quality, never taken from the model. Models are famously well-spoken about things that are false. Asking one to score its own confidence is asking the defendant to pass sentence.
- A failed enrichment can never fail an analysis. Model down?
invariants: [], everything else ships.
And the bit we're most glad we did: secrets never reach the model.
.env and similar files are excluded from the index entirely, .env.example contributes key names only, and every context window is redacted before it is built ( backend/app/probes/redaction.py scrubs AWS access key IDs, bearer tokens, connection strings, private keys).Issue: llama.cpp's --embedding flag quietly broke completions
We wanted embeddings for retrieval. On some llama.cpp builds, starting
llama-server with --embedding changes pooling behaviour and degrades the completion quality that analysis actually depends on.The fix is a design decision, not a workaround: embeddings get their own server and their own base URL (
PRAMAAN_LLM_EMBEDDING_BASE_URL ). Without one, retrieval falls back to lexical BM25 scoring — worse ranking, nothing broken. That fallback-rather-than-fail shape shows up everywhere in Pramaan and it's the single biggest reason the demo doesn't collapse when something is missing.
Day 3 (19–20 Sep) — the part that makes it a proof
This is the half of the project that justifies the name.
The App Map emits a Probe Input Manifest. From it, planners generate tests across four families —
frontend , backend , unit_test , load . The pipeline is one line:manifest -> TestPlan -> GeneratedTest -> TestResult -> Finding -> VerificationReport
Everything runs inside a disposable sandbox container: non-root, every Linux capability dropped,
no-new-privileges , 3 GB / 2 vCPU / 512 PID ceilings, and — importantly — no host path is ever mounted. The repository goes in as a tar stream, so target code cannot reach the host filesystem. The container is destroyed after the run.Running a repository at all requires explicit consent, per repository. Grants bind to the git remote they were given for, they expire (30 days by default), and every grant, use and revocation is appended to an audit table that is never updated or deleted. That last constraint has a specific reason: if a slug gets repointed at a different repository, it must not inherit permission to execute it.
Controlled experiments, not vibes
Here's the mechanism we're proudest of. An experiment states a hypothesis, arranges the conditions needed to test it, runs it, and reports what was observed. The rules, straight from
backend/app/probes/experiments.py :-
CONFIRMEDrequires direct runtime evidence of the suspected behaviour. Never a static pattern. Never latency alone.
- A causal claim requires a control/treatment pair — the same request, the only difference being the injected fault. Anything less is reported as
CORRELATED_ONLY.
- A missing capability is
NOT_GENERATEDwith a machine-readable reason, not a silent pass.
Concretely, for "this outbound call has no timeout": inject a 3000 ms delay into the dependency, replay the identical request against control and treatment arms, and check whether the app waited it out (more than 80% of the injected delay) or bailed. If it waited, that isn't a pattern match — that's the hang, reproduced.
For N+1, instrument the query layer and count identical row-level lookups in a single request. For unbounded queries, seed 5 rows and 50 rows and compare the shape of the scaling relation — a per-row implementation and a joined one cannot be confused at that spread. For idempotency, replay the same write twice; a duplicate on the first retry is the defect.
And then statistics, because one observation proves nothing
A single 3-second request could be a slow dependency or an unlucky GC pause. So every causal claim is backed by repeated trials, an effect size with a bootstrap confidence interval (2000 resamples), and a significance test at α=0.05 — written in plain
math and random so the probe layer carries zero numeric dependencies.The bootstrap seed is fixed. Re-running the same measurements must produce the same interval. We got that wrong once, watched a confidence interval wobble between identical runs, and fixed it in ten minutes — but it would have been a genuinely embarrassing thing for a judge to catch in a tool whose entire pitch is reproducibility.
This is where the AWS Builders' Library did real work for us. Marc Brooker's "Timeouts, retries, and backoff with jitter" and David Yanacek's "Implementing health checks" and "Using load shedding to avoid overload" are the clearest public writing on why these specific failure modes matter and how they actually manifest. We didn't cite them in code; we built risk types out of them. Michael Nygard's Release It! — integration points, cascading failure, circuit breakers — is the other spine.
Issue: 84 blocked tests, one useless error message
Early on, a run where the sandbox couldn't boot produced 84 tests all reporting the same prose string. Useless. The operator's next action is completely different for "you never granted consent" versus "the app crashed on boot" versus "the runner image is the wrong tag."
So
BlockedCode became a closed enum — CONSENT_NOT_GRANTED , SANDBOX_BACKEND_UNAVAILABLE , DEPENDENCY_INSTALL_FAILED , APP_BOOT_FAILED , APP_HEALTH_TIMEOUT , TOOLCHAIN_MISSING , TARGET_UNREACHABLE , CAPABILITY_MISSING , BUDGET_EXCEEDED , SPEC_INVALID , and a few more. Prose stays in the message field; the UI groups on the code. Same information, one-tenth the reading.Issue: restarting the backend silently ate every run
The registry was a module-level dict. Restart the process and the UI showed an empty history next to a full run-artifacts directory on disk. The worst kind of bug — no error, just quiet data loss.
Fixed with plain
sqlite3 , WAL journal mode so a long report write doesn't block the UI reading run history, and forward-only numbered migrations applied once at boot. The VerificationReport is already a versioned pydantic tree, so it's stored as JSON with only the columns worth querying on promoted alongside it — then shredded into test_result , finding , artifact and run_event rows so cross-run questions are a query rather than a scan of every stored report.
Day 4 (20 Sep) — AWS, and the constraint that decided everything
Here's the finding that reorganised our whole deployment:
Pramaan needs a Docker daemon it can talk to. Every run starts a container, installs the target's dependencies, boots it, tests it, destroys it. That single fact collapses the options:
Option | Works | Why |
|---|---|---|
EC2 + Docker | Yes | Backend drives the host daemon over its socket; sandboxes are siblings |
ECS on EC2 | Partly | Daemon exists, but task placement fights container lifecycle |
ECS Fargate | No | No daemon access at all |
Lambda | No | No daemon, 15-minute ceiling, no persistent workspace |
We started down the Fargate path because it's the "right" modern answer and it's what you reach for reflexively. Every test returns
SANDBOX_BACKEND_UNAVAILABLE . There is no flag, no escape hatch, no privileged mode. The app/runner/ecs.py seam exists in our tree as a deliberate stub with a comment explaining why it's empty.The lesson we'd give anyone: identify your one non-negotiable runtime capability before you pick a compute service, not after. Ours was a Docker socket. We found it on day 4. We'd have found it on day 1 if we had asked the question.
So: one EC2
t3.xlarge on Amazon Linux 2023, an Elastic IP, an EBS gp3 volume at /data holding clones, run artifacts and SQLite, three ECR repositories, CloudWatch Logs, an IAM instance role, and SSM Session Manager for access. Caddy terminates TLS and serves both apps from one origin, which deletes CORS as a category of problem.The Docker socket trade, stated plainly
docker-compose.prod.yml mounts /var/run/docker.sock into the backend. Access to that socket is equivalent to root on the instance. We aren't going to pretend otherwise, and we documented it under a heading that says read before deploying.The mitigations are real but partial: sandboxes are non-root with all capabilities dropped, no host path is ever mounted, memory/CPU/PID ceilings bound a runaway install, and the untrusted code runs one layer further in than the backend. The honest guidance is "run Pramaan on an instance you are willing to give it, and don't co-locate it with anything that matters."
The hardened shape — which needs zero code changes, because the backend drives Docker through its CLI — is to leave the socket unmounted and point
DOCKER_HOST=tcp://runner-host:2376 at a dedicated, disposable runner instance over TLS. That's the right answer beyond a demo, and we designed for it even though we didn't ship it.Issue: the UI loaded perfectly and every single API call failed
Classic, and it cost a genuinely stupid amount of time.
NEXT_PUBLIC_API_URL is inlined by Next.js into the client bundle at build time. We built the frontend image locally with localhost:8000 baked in, pushed to ECR, deployed, and got a perfect-looking dashboard where nothing worked.So
push-to-ecr.sh now takes PUBLIC_ORIGIN as a build arg, and the runbook says in a blockquote that getting it wrong means every page loads and every call fails. Moving to a real domain later means rebuilding the frontend image. This is inherent to build-time env inlining — worth knowing before you containerise any Next app.Issue: an 8B model and a 3 GB sandbox on an 8 GB box
An 8B Q4 model wants about 6 GB resident. Each sandbox is capped at 3 GB. A
t3.large (8 GB) cannot host both, and what you get isn't a clean error — it's the OOM killer picking a victim mid-run.t3.xlarge (16 GB) is the floor when the model shares the host. We documented three alternatives honestly, with cost: a separate GPU instance ( g5.xlarge , roughly $730/month on-demand — stop it when idle), Amazon Bedrock behind a translating proxy, or simply turning the model off.That last one deserves emphasis: the verification pipeline is entirely deterministic. With
PRAMAAN_LLM_ENABLED=false you lose LLM-authored scenarios, semantic invariants and graph enrichment. You keep analysis, the App Map, every route/unit/browser/load test, sandbox boot, findings and experiments. The model is not on the critical path, and we think that's the correct place for a model to sit in a tool that claims to produce evidence.Issue: Bedrock isn't OpenAI-compatible
We wanted Bedrock as the no-servers option. Pramaan speaks exactly one dialect: an OpenAI-compatible
/v1 endpoint. Bedrock's native API isn't that, so it needs a small translating proxy (we documented LiteLLM, which is open source and does this in about twelve lines of YAML). Two environment variables, no new provider class, no application code.The trade we wrote down next to it: your source code leaves your account. For a repo already public on GitHub, fine. For anything sensitive, the self-hosted options keep every byte on hardware you control. We'd rather state that than bury it.
Smaller things that cost real time
- A Next.js dynamic route conflict stopped
next devfrom starting entirely — two route segments with different param names at the same level. Fixed in8e705f8, then11511f1to read the slug on the moved page. Next's error is accurate but doesn't name the two offending files.
- Merge conflicts with an AI coding agent. We ran a Copilot agent on a parallel branch (
feat/llm-test-case-generation); it resolved its own conflicts across two commits. Useful for breadth. It does not understand whycanonical_hashmust stay stable, so every change near the determinism boundary got read line by line.
- Consent expiry surprised us in our own testing — grants bind to the remote and expire, so a test that passed yesterday blocked today. Working as designed. We kept it and added it to the troubleshooting table.
- Orphaned sandbox containers. A backend that crashes mid-run never reaches teardown, leaking a container holding memory and a published port. Now reaped by label at startup, with a daily cron pruning dead images.
What we'd tell ourselves on the 17th
- Find your one hard runtime requirement first. Ours was a Docker daemon, and it eliminated three of four compute options. One hour of research on day 1 replaces a day of rework on day 4.
- Decide what the model is allowed to decide, in writing, before you wire it up. "Append-only, validator-checked, never on the critical path, confidence derived from evidence" is one paragraph. It saved us from shipping a beautiful, unfalsifiable demo.
- Every degraded path needs a named, machine-readable reason. Not a message — a code. This is the difference between a demo that survives a missing dependency and one that dies on stage.
- Fix your random seeds on day one. A wobbling confidence interval in a reproducibility tool is a fatal-looking bug that takes ten minutes to prevent.
- Write the security trade-off down while you're making it. The Docker socket section of our docs took twenty minutes and is the thing we'd point at first in any real review.
The state of it
Four days: deterministic static analysis with a canonical hash, evidence-constrained LLM enrichment, a consent-gated sandbox runner, four probe families, control/treatment experiments with bootstrap confidence intervals, template-first fix proposals verified in throwaway git worktrees, an evidence graph with real causal edges, durable SQLite storage, and a full EC2/ECR/CloudWatch deployment with Terraform and a 45-minute runbook.
What it still won't do is tell you a risk is real until it has reproduced it. That was the point.
Pramaan analyses Node/TypeScript, Express, PostgreSQL (Prisma or raw SQL) and React/Next.js. The engine is FastAPI, tree-sitter and SQLite. Deployment is EC2, ECR, EBS, CloudWatch, IAM and SSM. Unsupported stacks return a structured
{"supported": false, "reason": ...} with HTTP 200 rather than failing — another small honesty rule that took five minutes and prevents a whole class of bad demo.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article