Don't Send That to the LLM: A Practical Guide to PII Masking

LLM agents can process logs, documents, and tool outputs that contain sensitive information. In this article, I share my approach to using Microsoft Presidio to detect and mask PII before it reaches an external LLM, along with practical observations, limitations, and ideas for improvement.

Achmad Al Fauzi Dhiaulhaq

AWS Cloud & AI Enthusiast | Learning, Building & Sharing

Published Oct 9, 2026

Last modified Oct 9, 2026

The problem: raw data reaching an external LLM

When I build an agent, data moves through many places. A user message. A Confluence page. A JSON log. A database result. Sooner or later, some of it goes to an external LLM.

That is a privacy and compliance problem if the data has personal or sensitive details. Masking PII (personally identifiable information) before that point helps in two ways:

It keeps sensitive data away from outside servers and LLM training data.
It keeps developer logs clean. For example, a credit card number or a token can end up printed in an exception log.

In my chatbot plan, an intent step decides which tools the agent may use. My notes say masking must come before that step, because the intent model is an external API.

What I wanted to mask

My first notes split the data into two groups:

Personal data: name, address, phone number, bank account number, card number (PAN), PIN, and RRN (a transaction reference number).
Credentials: usernames and passwords for servers and databases. Later I also covered database connection strings, API keys, and host:port values.

The goal is to detect the value and then mask it. Here are the mask styles I wrote down:

Type	Example after masking	Rule
Card	411111******1111	Keep the first 6 and last 4 digits
Email	b***@example.com	Keep the first character and the domain
Phone	*****7890	Keep the last 4 digits
Account or reference number	00********89	Keep the first 2 and last 2 characters
Password, secret	<Redacted>	Replace the whole value

The open-source tool: how Presidio detects and masks

Presidio is an open-source framework from Microsoft. It finds PII, hides it, and anonymizes it. It has four parts:

Analyzer: finds PII in natural language text.
Anonymizer: changes the PII it found (remove, redact, hash, encrypt).
Image Redactor: finds PII in images using OCR.
Structured: finds PII in structured or semi-structured data.

I worked with the Analyzer and the Anonymizer. Both can run as a Python package or as a REST API.

How the Analyzer finds PII

The Analyzer uses recognizers. A recognizer is a small component that looks for one kind of PII. It can use a regex pattern, a rule, a checksum, context words, or an NER model. NER (named entity recognition) is a model that labels words, like "Budi" as a PERSON.

Under it sits an NLP engine, which can be spaCy, Stanza, or Transformers. One model run is shared by all recognizers in the same request.

Each match gets a score. You set a threshold, for example 0.6. Anything below it is ignored. If the threshold is too low, normal words get masked. If it is too high, real PII slips through.

Context words

A pattern alone can be wrong. Take two sentences with the same 12-digit number:

"Your shipping receipt number is 081234567890" (not PII)
"Contact us via WhatsApp at 081234567890" (a phone number)

With regex only, both are flagged as phone numbers. With context words like "WhatsApp" or "contact" nearby, the second score goes up (my notes use 0.4 to 0.85 as an example). The first stays low and falls under the threshold.

One detail I want to remember: spaCy here only gives tokens and lemmas, so Presidio can match the context words. It is not doing semantic understanding like an LLM.

Here is a custom recognizer from my notes:

1
2
3
4
5
6
7
8
9
10
11
12
pin_block_pattern = Pattern(
    name="pin_block_pattern",
    regex=r"\b[A-Fa-f0-9]{6,16}\b",
    score=0.5
)
for lang in ["en", "id"]:
    recognizers.append(PatternRecognizer(
        supported_entity="PIN_BLOCK",
        patterns=[pin_block_pattern],
        context=["pin_block", "pinblock", "encrypted_pin"],
        supported_language=lang
    ))

How the Anonymizer masks

The Anonymizer supports replace, redact, mask, hash, encrypt, and custom operators. There is also a Deanonymize engine that can reverse the change, but only when the operator is reversible. My notes give this advice:

Redact (<REDACTED>) for LLM safety and public logs.
Hash or encrypt for analytics, when the same person should keep the same identifier.

Language

The default model is English. Other languages work through spaCy or Stanza. Presidio's built-in recognizers are tuned for English and US or European formats. For Indonesian data, my notes say I need custom recognizers for NIK (16 digits), NPWP (15 to 16 digits), and local phone numbers (08xx or +628xx), plus Indonesian context words. Another option is to train a custom NER model.

Structured data vs free text: two layers

My first question was: do I even need Presidio's NLP for JSON logs? For data with known keys, the answer in my notes is no.

A field called customer_name always holds a name. A field called auth_code always holds an auth code. The key already tells me more than a model would. My notes say it plainly: don't use NER for something the schema already tells you.

NER and context scoring become useful when the key is gone. Compare these two:

1
2
Structured:  { "card_no": "4512345678901234" }
Free text:   "Customer used card 4512345678901234 for a transaction."

In the second one, nothing says "card number" except the sentence around it. That is where I need a detector. So the setup I want is two layers in one pipeline: a key-based masker for structured fields, and Presidio for free text. Not one or the other.

Built: the key-based masker for JSON logs

It tries three steps in order:

Exact key match. The key is normalized, then checked against a whitelist file.
Fallback 1: value pattern. A regex looks at the value itself.
Fallback 2: key keywords. It checks if the field name contains words like cvv, cvc, pin, password, passwd, secret, token, apikey, privatekey, accesskey, or otp.

The rules are grouped by how much to hide:

1
2
3
4
5
MASK_REDACT_FIELDS   -> full mask: cvv, pin, password, secret_key
MASK_PARTIAL_CARD    -> card_number, pan
MASK_PARTIAL_EMAIL   -> email, recipient_email
MASK_PARTIAL_PHONE   -> phone, msisdn, contact_number
MASK_PARTIAL_GENERIC -> rrn, account_number, reference_number

I also planned which fields to mask for dummy transaction data. Not every field is PII:

Field	Mask?	How
acq_rrn	Yes	Regex, validation, field context
terminal_id	Yes	Regex, field context
auth_code	Yes	Regex, field context
card_no (card data)	Yes	Regex plus Luhn check
merch_location	Depends on policy	Field-aware, or NER if it is free text
txn_amount, currency, response_code	Usually no	Leave as is

Built: Presidio for credentials in free text

For free text, I wrote custom recognizers. They are regex based, with a starting score and context words to raise the confidence:

Recognizer	Entity	What it catches	Score
CredentialKVRecognizer	CREDENTIAL_KV	password=xxx, secret: xxx	0.75
DBConnectionStringRecognizer	DB_CONNECTION_STRING	Database URIs like postgresql://user:pass@host:5432/db	0.85
APIKeyRecognizer	API_KEY	api_key=... (16+ characters) and Bearer <token>	0.80 / 0.85
HostPortRecognizer	HOST_PORT	IP:port	0.70
UsernameRecognizer	USERNAME	username: admin, user=john	0.85

The username recognizer overrides the analyze() method. That way only the value is marked, not the label username:. Email and phone use Presidio's built-in recognizers.

The credential entities are replaced with <Redacted>. They are pure regex, so they do not depend on the language. The spaCy model (en_core_web_sm) is only used for context matching. The score threshold is 0.6.

Masking Confluence-style markdown

Docs from Confluence are markdown with tables and code blocks. Presidio works on plain text, so I needed extra steps.

Built: the service works in three steps.

Segment. Split the markdown by structure, and keep the original.
Restructure. Turn each piece (table, code block, prose) into a plain string, run Presidio, and join the results back into one text.
Replace back. Collect the PII values that were found, and replace them in the original markdown.

I also have a handler for fenced code blocks, so credentials inside code are caught. It is in my config, but it is not active yet.

Problem I noted: removing markdown symbols fixed one problem, the symbols that disturb regex. It did not fix a second problem: the key and the value are in different places.

Here is a table row (I replaced the real value with a placeholder):

1
| Redis Cache | redis_admin | <secret> | 10.20.30.50:6379 |

The word "Password" lives in the header row, not in this row. After stripping the symbols, the header is still not next to the value. So the context word never reaches it. A VPN note has the same problem:

1
2
- Pass:
  <secret>

The key Pass: is on one line and the value is on the next. Stripping markdown does not join the two lines. My notes say the context window would need to reach across lines to catch this. I have not tested that.

The flow, step by step

Here is the whole flow in one place. Each piece of data takes the layer that fits it.

Data comes in. It can be a Confluence page, a JSON log, a user message, or a tool result.
Structured fields go through the key-based masker: exact key match, then value pattern, then key keywords.
Free text goes through Presidio. The Analyzer finds entities and scores them. Anything under the 0.6 threshold is dropped.
The Anonymizer applies the operator: redact, partial mask, hash, or replace.
Masked output comes out. The real values are gone.
Only then is the data embedded into a vector database, passed to the intent step, or sent to an external LLM.

Diagram: the PII masking flow

The order is the main point: mask first, then use the data.

Where PII masking fits in an agent

I am studying a chatbot with these parts: an LLM, an intent step, MCP tools, skills, and a PII masker. My notes list the PII masker as a gate that cleans data before it goes to an external LLM. I see three places where it fits.

1. Before the intent step (design)

The user message is masked first. Only then does it go to the intent router. The reason in my notes: the intent model is an external API, so it must not see raw data.

2. Inside the MCP tool (design)

MCP gives the agent its tools. The model does not know how to connect to a database. The MCP code holds the endpoints, secrets, and .env config. My notes describe this flow when the agent checks failed transactions:

The LLM calls an MCP tool, like get_logs_transaction.
The MCP code gets the raw data. It still has names, emails, and card numbers.
Before returning anything, the MCP code masks it. It either runs custom regex, or sends an HTTP POST to an internal PII masking API.
The clean data goes back to the LLM.
The LLM analyzes it and answers the user.

So the model never sees the raw values. My reading: a skill can say "do not show customer IDs", but that is only an instruction in a prompt. Masking in code is a stronger guarantee.

This is also why my notes say zero credentials in skills. Skills are prompts that guide how the model thinks. Putting an API key or endpoint in a prompt is risky. Secrets belong in the MCP layer.

3. Before embedding, for RAG (built as a service)

My first meeting notes say the masking API should run before embedding. Confluence data is fetched, masked, and only then embedded into the vector database. That is what the markdown service above is for.

Open-source tools and models I looked at

Presidio picks how heavy the detection is. My notes use three levels:

Light: regex and pattern matching only. No NLP model.
Standard: a spaCy model.
Heavy: a model you host yourself, like a local Hugging Face transformer.

Here are the models I compared. These figures come from my reading notes. They are not my own benchmarks.

Model	Type	Hardware	Memory	Speed
spaCy (en_core_web_lg)	CNN with word vectors	CPU	About 500 MB to 1 GB RAM	Very fast
GLiNER (knowledgator edge version)	Compressed transformer	CPU or small GPU	About 1 GB	Fast with a GPU
GLiNER (nvidia/gliner-PII)	Transformer	GPU	2 to 4 GB VRAM	Fast with a GPU
Flair (ner-english-large)	BiLSTM-CRF	GPU or multi-core CPU	About 1 to 2 GB	Medium
Stanza (en)	Deep learning pipeline	GPU or multi-core CPU	About 2 GB	Slow to medium
Hugging Face (OpenMed-PII)	RoBERTa or BERT	GPU	2 to 8 GB VRAM	Fast with a GPU

Two extra notes. Flair often gets better accuracy than spaCy on messy text, but it is slower. The GLiNER nvidia version is tuned for PII in general, and the edge version is a lighter one.

A tip for servers: create the Presidio analyzer once at startup, and reuse it for every request. The model stays in memory, so detection is much faster.

Idea (not tested): my AWS study notes list Guardrails for Amazon Bedrock. It can block topics, filter harmful content, and detect or redact PII, and it works with different foundation models. I have not tried it. I see it as a managed option next to the open-source route.

Custom combinations

No single method covered everything in my notes. Each one has a job:

Approach	Good at	Weak spot
Rule-based (regex, deny list, checksum)	Data with a fixed format, like emails or card numbers	Needs context, or similar numbers get flagged by mistake
NER (ML model)	Things with no fixed format, like names and places	Heavier, and it reads language context
Hybrid (rule plus context words)	Telling the same pattern apart by the words around it	Context only reaches nearby tokens

Here are the combinations I wrote down. The status column says where each one stands.

Combination	What it does	Status
Key-based masker plus Presidio	Structured fields by key, free text by Presidio	Built: both layers exist. Plan: run them together in one pipeline.
Regex plus context words	Raises the score only when a trigger word is near	Built: custom recognizers with context words
Mask, then intent router, then restricted ReAct	PII is removed before the router sees the message	Design from my notes
Masking inside the MCP tool	The LLM only gets clean tool results	Design from my notes
Jev-AI yes/no check before masking	Ask "does this document contain PII?" as a Noul question, and "what category is it?" as a Choice question	Idea, not tested.
Managed Guardrails for Amazon Bedrock	Detect and redact PII without hosting models	Idea, not tested

On the Jev-AI idea: from my other article, Jev showed lower recall in an independent test. So for now I would still mask everything that leaves the system, and use the check only to learn how often it misses PII. That is my own thinking, not a result.

Use cases

These come from the places I wanted to use masking. Only the first two have a service behind them in my notes. The rest are designs.

Confluence pages before embedding. Mask the markdown, then embed it into the vector database for RAG.
Credentials in free-text docs. Database connection strings, API keys, host:port values, and password=... lines. For example, postgresql://user:pass@host:5432/db becomes <Redacted>.
Transaction and card JSON. Mask fields like card_no, acq_rrn, and terminal_id before an LLM reads the log.
MCP tools that read databases or logs. Mask the result inside the tool, so the agent only sees clean data.
User messages before an external intent model. Mask first, then classify.
Developer logs. Stop card numbers or tokens from showing up in exception logs.

Limitations

This is still early work. Here is what I can't say yet.

I have no accuracy numbers. My notes do not include precision or recall, and I did not time the models. The resource table is from reading, not testing.
The threshold is not tuned. My notes use 0.6 and say to calibrate it. I did not record a tuning test.
English first. The defaults are made for English and US or European formats. Indonesian needs custom recognizers (NIK, NPWP, local phone numbers) and local context words. A custom Indonesian NER model is still just an option.
Context is shallow. It matches nearby tokens and lemmas. It does not understand meaning. Across table rows or lines, it does not reach the value (see the markdown section).
Regex only finds formats I wrote down. A credential in a style I did not cover can slip through. Names and addresses in free text need an NER model.
Heavier models need more. The transformer models in my table want a GPU.
The code-block handler is off. Credentials inside fenced code may be missed for now.
Does masking keep enough context? My notes ask this as a question. If the text is masked too much, the intent step or the LLM may not do its job. I did not measure it.
Reversing the mask is untested. My first meeting notes list "decode" as a final step. Presidio's Deanonymize engine only works with reversible operators. I have no record of trying it.

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Don't Send That to the LLM: A Practical Guide to PII Masking

Achmad Al Fauzi Dhiaulhaq

The problem: raw data reaching an external LLM

What I wanted to mask

The open-source tool: how Presidio detects and masks

How the Analyzer finds PII

Context words

How the Anonymizer masks

Language

Structured data vs free text: two layers

Built: the key-based masker for JSON logs

Built: Presidio for credentials in free text

Masking Confluence-style markdown

The flow, step by step

Where PII masking fits in an agent

1. Before the intent step (design)

2. Inside the MCP tool (design)

3. Before embedding, for RAG (built as a service)

Open-source tools and models I looked at

Custom combinations

Use cases

Limitations