RAG Basics: Making an LLM Answer From Your Data
RAG Basics: how to make an LLM answer from your own data using chunking, embeddings, and retrieval.
The problem
An LLM only knows what was in its training data. It doesn't know your documents, your database, or anything that happened last week. Ask it about those and it will often make something up, confidently.
Retrieval-Augmented Generation (RAG) fixes this with a simple idea: before asking the model, find the relevant text yourself and put it in the prompt.
The pipeline in five steps
- Chunk: split your documents into small pieces (a few hundred tokens each). Whole documents are too big and too vague to retrieve well.
- Embed: turn each chunk into a vector (a list of numbers) using an embedding model. Chunks with similar meaning end up close together in vector space.
- Store: save the vectors in a vector index (FAISS, Chroma, pgvector).
- Retrieve: at query time, embed the question the same way and fetch the top-k nearest chunks, usually by cosine similarity.
- Generate: build a prompt like "Answer using only this context: {chunks}. Question: {q}" and send it to the LLM.
1
2
3
4
5
6
7
8
# Pseudocode of the whole idea
chunks = split(docs)
index = build_index(embed(chunks))
def answer(q):
top = index.search(embed(q), k=4)
prompt = f"Context:\n{top}\n\nQuestion: {q}\nAnswer only from the context."
return llm(prompt)
Why it works
Embeddings capture meaning, not keywords. "Refund policy" and "how do I get my money back" land near each other, so retrieval finds the right chunk even when the words differ. The LLM then only has to read and summarize, which it's good at, instead of recalling facts, which it's bad at.
Where it breaks (the part interviewers ask about)
- Bad chunking: chunks that are too big dilute the match, and chunks that are too small lose context. Overlap between chunks helps.
- Bad retrieval: if the right chunk isn't in the top-k, the model can't answer, and no prompt will fix that. Hybrid search (keyword plus vector) and reranking are the usual upgrades.
- Ignored context: the model may still answer from its own memory. Tell it to say "I don't know" when the context lacks the answer.
- No evaluation: "looks right" isn't a metric. Measure retrieval (did the right chunk come back?) separately from generation (was the answer faithful to it?).
One-line summary
RAG = search first, then ask. Most of the quality comes from retrieval, not from the LLM.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article