Every LLM Call Should Pass Through a Security Layer — Here's the Pattern

If you're building with LLMs in production, here's an uncomfortable truth: right now, your application probably sends whatever the user typed — PII, credentials pasted by accident, internal data — straight to a model provider's API. And whatever the model returns goes straight back to the user, unexamined.

I kept running into this while building agent systems. The standard advice is "be careful with your prompts" or "filter outputs in your app." That doesn't scale: every app reimplements it, every developer remembers (or forgets) differently, and the moment you add a second model provider, you're doing it twice.

The pattern that fixed it for me: put a policy-enforcing gateway between your apps and every model. One `base_url` change, and every request passes through security policy on the way in and on the way out — no matter which model sits behind it.

I built this as an open-source project called Argus. This post is about the pattern, with Argus as the worked example.

## The threat, concretely

Three things keep showing up in real LLM deployments:

1. **Secrets in prompts.** Developers paste AWS keys, tokens, and `password = ...` lines into prompts while debugging. It happens constantly.
2. **PII in both directions.** User data goes in; the model can echo it back out — or worse, into logs, traces, and eval datasets.
3. **Prompt injection.** "Ignore previous instructions" and its hundred variants. If your app acts on model output, injection is a security-boundary problem, not a prompt-engineering problem.

App-level filtering handles the happy path. It fails the moment someone adds a new code path, a new model, or a new developer who didn't get the memo.

## The gateway pattern

The idea is simple: terminate all model traffic at a proxy you control, and enforce policy there.

```
request → auth → INPUT policy → budget → route → model
→ OUTPUT policy → audit → response
```

Your app changes one line — the `base_url`:

```python
client = openai.OpenAI(api_key="<virtual-key>", base_url="http://localhost:4000/v1")
```

Everything else — detectors, policy, audit — lives in the gateway. New model provider? Point the gateway at it; policy doesn't change. New app? Same `base_url` trick.

## Detectors find, policy decides

A design decision that mattered: separate *detection* from *decision*. Detectors scan text and report findings. The policy engine decides what to do — per direction (`in`, `out`, or `both`), with four actions: `allow`, `redact`, `block`, `flag`.

| Detector | Catches | Default |
|---|---|---|
| Secrets | AWS keys, private keys, GitHub/OpenAI tokens, `password = ...` | **block** |
| PII | emails, phones, SSNs, credit cards, IPs | **redact** |
| Prompt injection | instruction overrides, role games, jailbreak patterns | **flag** |
| Blocklist | your own regexes | **block** |

Why `flag` instead of blocking injection outright? Because blocking breaks legitimate use ("ignore the previous email draft and rewrite it" is not an attack). Flagging marks the request — one configuration change away from block — and records it in the audit trail. Security people get visibility; users don't get false-positive outages.

Adding a new detector (say, an ML-based one later) means subclassing `Detector`. Nothing else changes.

## The audit trail is the point

Here's the part most homegrown filters skip: every policy decision gets recorded — timestamp, key, task, detector, action, and a *redacted* snippet. Raw PII and secrets never touch the audit log. That's what makes it usable for compliance review instead of being a second liability.

```
GET /admin/audit?limit=50&key_id=...
```

## It also watches the money

Since every request already flows through the gateway, cost intelligence comes nearly free: requests get scored for difficulty and routed to the cheapest capable tier, virtual keys carry budgets (over budget degrades gracefully to the cheapest tier instead of failing — hard-stop mode if you want it), and an `X-Task-Id` header groups an agentic run's turns into one costed task. Every response tells you what happened:

```json
"gateway": {
"provider": "openai", "tier": "small",
"cost_usd": 0.000004, "degraded": false,
"security": {"redactions": ["pii:EMAIL"], "flags": ["prompt_injection:INSTRUCTION_OVERRIDE"]}
}
```

## Try it

```bash
pip install argus-gateway
export AIGW_MASTER_KEY="sk-admin-secret"
export OPENAI_API_KEY="..."
argus # serves on 127.0.0.1:4000
```

It also fronts self-hosted models — vLLM, Ollama, llama.cpp, anything OpenAI-shaped — behind the same policy layer. That's the whole point: the policy doesn't care which model answers.

Repo: github.com/karmendra8386/argus (MIT). Issues and PRs welcome — the roadmap has streaming, semantic caching, and an ML-based injection detector, and I'd rather build those with people who've hit these problems in production.

## The takeaway

Don't scatter security logic across every app that calls a model. Terminate model traffic at one gateway, enforce policy in both directions, and audit everything. It's a small architectural decision that removes an entire class of "we didn't think about that path" incidents.