Building a Financial Document Q&A System with RAG, LangChain, and OpenAI
AI-powered Q&A system for financial documents using RAG. Upload PDFs or TXT files, ask plain-English questions, and get cited answers grounded in your documents — built with LangChain, OpenAI, FAISS, and Streamlit.
Building a Financial Document Q&A System with RAG, LangChain, and OpenAI
Introduction
Financial documents — annual reports, earnings filings, balance sheets — are dense, lengthy, and critical. Finding a specific figure or understanding a risk factor buried in a 200-page PDF takes time and expertise.
What if you could just ask your documents a question and get a precise, cited answer in seconds?
That's exactly what I built: FinanceRAG, an AI-powered Q&A system for financial documents. In this post, I'll walk through the architecture, key design decisions, and how Retrieval-Augmented Generation (RAG) makes it work reliably without hallucination.
The Problem with Vanilla LLMs
Large Language Models like GPT-4 are impressive, but they have a core limitation for this use case: they don't know your private documents.
You could paste a document directly into the prompt, but:
- Most financial reports exceed GPT's context window
- Costs scale with every token you send
- The model has no way to "focus" on the relevant section
The solution is RAG — instead of sending the whole document, you retrieve only the most relevant chunks and send those.
What is RAG?
Retrieval-Augmented Generation (RAG) combines a retrieval system (search) with a generation system (LLM):
1
2
3
Document → Chunks → Embeddings → Vector Store
↓
User Question → Embedding → Similarity Search → Top Chunks → LLM → AnswerThe key insight: semantic search finds chunks that are meaningfully similar to the question, not just keyword matches. "What were earnings?" finds chunks mentioning "net income", "profit", and "revenue" — because they're close in vector space.
Architecture Overview
FinanceRAG is built on three layers:
| Layer | Technology | Role |
|---|---|---|
| Document Processing | LangChain + PyPDF | Load, parse, and chunk documents |
| Vector Search | FAISS + OpenAI Embeddings | Store and retrieve relevant chunks |
| Answer Generation | GPT-3.5-turbo / GPT-4 | Generate grounded answers |
| Interface | Streamlit | Browser-based UI |
Step 1: Loading and Chunking Documents
Raw PDFs are loaded page by page, then split into overlapping chunks:
1
2
3
4
5
6
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, # ~200 words per chunk
chunk_overlap=200, # overlap maintains context at boundaries
separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(documents)Why overlap? A sentence split across two chunks would lose context in both. The 200-character overlap ensures boundaries don't cut off meaning.
Why 1000 characters? It's a balance — small enough to be precise in retrieval, large enough to contain a complete thought. You can tune this in the UI.
Step 2: Creating Embeddings and a Vector Store
Each chunk is converted to a 1536-dimension vector using OpenAI's
text-embedding-3-small model, then stored in a local FAISS index:1
2
3
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = FAISS.from_documents(chunks, embeddings)
vector_store.save_local("faiss_index")FAISS (Facebook AI Similarity Search) is a highly efficient library for nearest-neighbor search. On a typical 50-page report with ~150 chunks, search takes under 10ms.
Saving the index to disk means you only pay the embedding cost once. Subsequent runs load the index for free in milliseconds.
Step 3: The QA Chain with a Finance-Specific Prompt
The retriever fetches the top 3 most relevant chunks, and a custom prompt instructs GPT to answer strictly from those chunks:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
prompt_template = """You are a financial analyst AI. Answer the following question
based ONLY on the provided document excerpts.
If the answer is not in the documents, say: "I don't have this information
in the provided documents."
Always cite which document section you're referring to.
Context from documents:
{context}
Question: {question}
Answer:"""Two important design choices here:
- "Based ONLY on the provided documents" — prevents the LLM from hallucinating figures it learned during pre-training
- Explicit fallback instruction — if the answer isn't there, the model says so instead of guessing
The chain is assembled with LangChain's
RetrievalQA:1
2
3
4
5
6
7
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vector_store.as_retriever(search_kwargs={"k": 3}),
chain_type_kwargs={"prompt": PROMPT},
return_source_documents=True
)return_source_documents=True is key for transparency — the UI displays exactly which document chunks the answer came from, so users can verify the source.Step 4: The Streamlit Interface
The UI is intentionally simple:
- Sidebar — initialize the pipeline, upload files, configure chunk size and model
- Main area — type a question, get an answer with expandable source citations
1
2
3
4
5
6
7
8
result = st.session_state.pipeline.query(question)
st.write(result["answer"])
for source in result["sources"]:
with st.expander(f"📄 Source"):
st.write(source.page_content)
st.json(source.metadata)Keeping the pipeline in
st.session_state means it persists across reruns without re-initializing.Key Design Decisions
1. Local FAISS over a managed vector DB
For a single-user tool processing private financial documents, a local FAISS index is simpler, faster, and free. There's no data leaving your machine after the initial embedding API call.
2. Configurable chunk size
Different documents benefit from different chunk sizes. Dense tables of figures need smaller chunks; narrative sections work better with larger ones. Exposing this as a slider in the UI lets users tune it without touching code.
3. Model selection in the UI
GPT-3.5-turbo handles most factual retrieval questions well and costs ~10x less than GPT-4. Offering both lets users choose the right tradeoff per task.
4. Overlap chunking over hard splits
Hard splits at exactly 1000 characters would break sentences and tables.
RecursiveCharacterTextSplitter tries paragraph boundaries first, then lines, then words — producing more natural chunks.Cost Breakdown
For a typical 50-page annual report (~75,000 tokens):
| Operation | Tokens | Cost |
|---|---|---|
| Embedding (one-time) | ~75,000 | ~$0.002 |
| GPT-3.5 query | ~1,500 | ~$0.002 |
| GPT-4 query | ~1,500 | ~$0.015 |
Processing an entire report costs less than a cent. Repeated queries reuse the saved index at no additional embedding cost.
What I'd Build Next
- Multi-document comparison — ask questions that span multiple filings (e.g., "How did revenue trend from 2021 to 2023?")
- Table-aware chunking — financial tables lose structure when chunked as plain text; specialized parsing would improve accuracy on numerical questions
- Hybrid search — combining semantic search with BM25 keyword search often improves recall for specific figures and names
- AWS deployment — containerize with Docker, host on Amazon ECS, use Amazon S3 for document storage and index persistence
Running It Yourself
1
2
3
4
5
6
7
8
9
git clone https://github.com/its-me-sweta/Finance_Rag.git
cd Finance_Rag
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
# Add your OpenAI key to .env
echo "OPENAI_API_KEY=sk-..." > .env
streamlit run app.pyOpen localhost:8501 in your browser, upload a PDF, and start asking questions.
Conclusion
RAG is one of the most practical patterns in applied AI right now. It takes the reasoning power of LLMs and anchors it to your own data — solving hallucination, context limits, and privacy concerns in one architecture.
For financial documents specifically, the combination of semantic retrieval + a grounded prompt + source citations makes the system genuinely useful and trustworthy.
The full source code is available on GitHub (its-me-sweta/Finance_Rag). Try it on your next earnings report.
Built with LangChain, OpenAI, FAISS, and Streamlit.
Questions or feedback? Drop a comment below.
Questions or feedback? Drop a comment below.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article