Can You Tell What an LLM Is "Thinking About" Without Looking at Its Output?
Using natural language autoencoders to detect specific types of content in LLM processing
We tried using Natural Language Autoencoders to detect when a model is processing patient health records — and it worked perfectly.
The Setup
Earlier this year, Anthropic published work on Natural Language Autoencoders (NLAs) — a technique that translates a model's internal activation vectors into plain English descriptions. You take an activation from somewhere inside the model, feed it to a specialized "verbalizer" network, and out comes a human-readable explanation of what the model is representing at that point in its computation.
The NLA team released pre-trained weights for several open-weight models, including Gemma 3 12B. I wanted to know: can this actually be useful for something practical? Specifically, can NLA tell you whether a model is processing sensitive data — like patient health information — just by reading its internal state?
Why This Matters
Healthcare organizations are increasingly using LLMs to process clinical documents — summarizing notes, extracting codes, drafting referral letters. A persistent concern is: how do you know what data the model is actually handling internally? You can look at the output, sure. But what about cases where you want to verify data handling before any output is generated, or where you need an audit trail of what information the model touched?
If you could read the model's internal state and say "yes, this model is currently processing patient-identifiable health information" — that's a useful guardrail.
The Experiment
I set up a straightforward test with 80 prompts in three categories:
- 30 synthetic clinical notes wrapped in "Summarize the following clinical note:" — containing synthetic patient names, DOBs, MRNs, diagnoses, medications, and provider names
- 30 general medical questions — things like "Explain the mechanism of action of metformin" or "What are the stages of chronic kidney disease?"
- 20 non-medical controls — photosynthesis, sourdough recipes, GPS, etc.
For each prompt, I:
- Ran Gemma 3 12B and captured the internal activation at layer 32 (roughly two-thirds depth)
- Fed that activation to the pre-trained NLA verbalizer to get a plain-English explanation
- Asked an LLM judge (Claude Haiku) whether the explanation indicates the model was processing patient-specific health information
The Results
| Category | LLM Judge: PHI Detected |
|---|---|
| Clinical notes with PHI | 100% (30/30) |
| Medical questions (no PHI) | 0% (0/30) |
| Non-medical control | 0% (0/20) |
What the NLA Explanations Actually Say
This is the interesting part. Here's what the NLA produces for a clinical note about a CHF patient:
"Clinical SOAP note format structures the documentation, establishing a medical presentation narrative with patient history and CHF diagnosis. The phrase 'Treatment plan: Patient admitted for dyspnea, chest xray ordered' signals a clinical documentation summary, expecting a patient encounter update..."
And for a psychiatry visit note:
"Clinical documentation format: structured SOAP/patient record with medication history, presenting a psychiatric visit summary for depression/hypertension."
Compare that to what it says for "Explain how ACE inhibitors work to lower blood pressure":
"Medical/educational Q&A format signals a question about a drug class, establishing a factual explanation of ACE inhibitors and hypertension."
The distinction is clear. The NLA isn't just detecting medical vocabulary — it's identifying what kind of document the model is processing. "Clinical SOAP note with patient history" vs. "educational Q&A about a drug class." That's exactly the distinction that matters for PHI detection.
The Failed Experiment That Led Here
I should mention: this wasn't my first attempt. I originally tried to show that NLA could detect "unverbalized regulatory awareness" — the idea that a model might internally think about Sarbanes-Oxley compliance without mentioning it in its output. That experiment failed completely. Twice.
The failure taught me something important about what NLA captures at this layer depth. Layer 32 of Gemma 3 12B encodes discourse structure — document type, format, audience, intent. It doesn't encode specific domain knowledge like "this triggers SOX Section 302." But discourse structure is exactly what you need to distinguish "processing a patient record" from "answering a medical textbook question."
I had also tried to actually build the NLA models for a different model, but found that it was going to take more compute time and resources (33 days on a g5.48xlarge) than I could budget for.
Technical Details
For anyone wanting to reproduce this:
- Model: Gemma 3 12B Instruct (
google/gemma-3-12b-it) - NLA weights:
kitft/nla-gemma3-12b-L32-av(verbalizer) andkitft/nla-gemma3-12b-L32-ar(reconstructor) from HuggingFace - Serving: SGLang with
--tp 2 --attention-backend triton --disable-radix-cache - Hardware: g5.48xlarge (8x NVIDIA A10G). SGLang uses 2 GPUs for the verbalizer, the base model shards across the remaining 6.
- Reconstruction fidelity: Cosine similarity 0.996 — the round-trip from activation to text to reconstructed activation loses almost nothing.
A few gotchas I hit along the way:
- Gemma 3 IT is a multimodal model; the decoder layers are at
model.model.language_model.layers, notmodel.model.layers - The
fa3attention backend requires Hopper GPUs (H100); usetritonon A10G - NLAClient expects a local directory path, not a HuggingFace repo ID — use
huggingface_hub.snapshot_download()first - Set
CUDA_VISIBLE_DEVICESto exclude the GPUs SGLang is using, or the base model will try to load onto them
Keyword Matching Isn't Enough
One thing worth noting: I also ran keyword matching against the NLA explanations (looking for terms like "patient," "clinical," "diagnosis"). Keywords hit 53% of the medical-without-PHI explanations — because those explanations naturally mention medical terms in an educational context. The LLM judge correctly scored those as 0% PHI because it understands the difference between "the model is processing a patient record" and "the model is explaining a drug mechanism." If you're building something like this, don't stop at keywords.
What This Could Become
This is a proof-of-concept, not a product. But the path from here to something useful is fairly short:
- Real-time PHI guardrail: Extract activations during inference, run NLA, flag if PHI is detected — all before the model generates output
- Data pipeline auditor: Verify that a de-identification pipeline actually removed PHI by checking what the downstream model "sees" in its activations
- Compliance logging: Record activation-level classifications as an audit trail of what data types the model processed
The pre-trained NLA weights work out of the box. No fine-tuning needed. The inference cost is one additional forward pass through the verbalizer (same size as the base model) per activation you want to explain.
Conclusion
NLA gives you a window into what an LLM is processing at the activation level, expressed in plain English. For the specific question "is this model handling patient health information?" — at least on Gemma 3 12B at layer 32 — it answers with 100% accuracy on our test set.
The broader takeaway: NLA is most useful when you care about what kind of content the model is processing (document type, format, data sensitivity) rather than what specific knowledge it's retrieving. That maps well to content classification, data governance, and sensitivity detection — exactly the problems that regulated industries need solved.
Finally, I suggest checking out the excellent Neuronpedia site for further experimentation.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article