What I Learned Building a RAG Campus Bot as an Newbie (85% vs 58%)
In 2022 I built a campus knowledge assistant to help my hostel-mate find exam dates buried across three PDFs. The patterns I figured out by accident, document-aware routing, honest "I don't know" paths, evaluating on real queries, turned out to be the same ones I use today on production agentic systems.
In my third year at NIT Surat(undergrad), my hostel-mate kept asking me what time the database systems lecture was. Not because she didn't have the timetable, but the academic calendar had been revised three times that semester, the holidays list was on a different PDF, the exam dates were on a third one, and the registrar had quietly updated the elective deadline somewhere in a sub-folder on the institute website that you'd only find if you already knew the URL.
She wasn't lazy. The information was just genuinely hard to find. Not her fault when she has a geeky friend though.
That's how the project that became my campus knowledge assistant started. Students had questions, the answers existed in PDFs and notice-board scans somewhere, but the gap between question and answer was painful. I wanted to close that gap.
This was around May 2022. RAG wasn't a buzzword yet. "Agentic" definitely wasn't. ChatGPT had launched a few months earlier and most of my batchmates were still using it to write break-up texts.
What I built, badly, on the first try
My first version was a single OpenAI call with the entire institute handbook stuffed into the prompt. It cost about ₹12 per query and got things wrong constantly because the handbook was 80+ pages of bureaucratic prose and the model couldn't tell which year's calendar I was asking about.
This is when I figured out, the hard way, that you can't just dump documents into an LLM and call it a system.
I rewrote it as retrieval-augmented:
- Embeddings using
sentence-transformers/all-MiniLM-L6-v2 - Chunked the documents at 500 tokens with 50-token overlap
- Stored them in FAISS
- Retrieved top-k=5
- Stuffed the chunks into the prompt with GPT-3.5-turbo (While writing this we use GPT 5 - how fast the time flows!)
Standard RAG, by 2026 standards. Felt revolutionary in 2022. I had to read research papers for cracking this and honestly felt so proud at that time.
It got better. Not great.
The problem was that "when is the DBMS exam" needed the academic calendar, "what are the elective options for sem 5" needed the curriculum doc, and "what's the attendance policy" needed the student handbook. Naive retrieval pulled chunks from whichever document had the closest cosine similarity, which often wasn't the right one. I'd get an answer about attendance when I asked about exams. Today - we use embeddings so casually that the relevance is hard to strike.
The agent part
Three documents kept showing up as the source of most queries: the academic calendar, the syllabus PDF, and the student handbook. So I stopped trying to make one retrieval system serve all of them and split the bot into routes.
A small classifier — logistic regression on features, decided which document the query was about. Retrieval then ran only within that document.
That's the thing I'd later learn was called "tool routing" or "agentic orchestration." At the time I just thought I was writing if-statements.
The classifier was wrong sometimes. So I added a fallback: if the top retrieved chunk had a confidence score below some threshold, the bot would search across all documents and warn the user that the answer might be incomplete. This sounds obvious now. It was not obvious at 2 AM with a 7 AM lab the next day.
The final pipeline had three tools the LLM could invoke:
- Document search within a routed document
- A structured lookup against a small JSON of exam dates and timetables I'd scraped manually
- An "I don't know" path that explicitly said so instead of hallucinating
The third one mattered more than I realised. Most LLM apps in 2022 were happy to confidently make things up. Returning "I don't have this information, you should check with the academic section" was a feature, not a failure.
Numbers
I evaluated against ~120 [VERIFY exact count] queries collected from friends across years and branches. Three of us [VERIFY] judged each response as correct, partially correct, or wrong — blind to which system produced which answer.
| System | Judged Correct |
|---|---|
| Baseline keyword search | 58% |
| RAG + routing + fallback | 85% |
The 27-point gap wasn't from the LLM. It was from the routing. When I ran ablations [VERIFY — did you actually do ablations?] without the document classifier, accuracy dropped to about 71%. The biggest single win came from knowing which document to retrieve from before retrieving.
I also tracked an unanswered-query rate: how often the bot punted with "I don't know" versus making something up. Baseline keyword search punted on ~40% of queries (no matching documents → no answer). The agent system punted on closer to 12%, which I count as a win even though the punt rate wasn't zero.
What I'd do differently now
Looking back from 2026, what I got right by accident:
- Splitting retrieval by document type instead of treating it as one big corpus
- Building an honest "I don't know" path
- Evaluating on real student queries, not synthetic ones
What I got wrong:
- I didn't version the documents. When the academic calendar updated mid-semester, I had to rebuild from scratch.
- My evaluation set was tiny. ~120 queries isn't enough to be confident about anything, but it was what I had.
- I didn't measure latency. The bot took ~6 seconds per query, which I now know is too slow for anything anyone would actually use.
If I were rebuilding this on AWS today, I'd reach for Bedrock Knowledge Bases for the retrieval layer, S3 with versioning enabled for document storage, and Lambda for the orchestration. The classifier-router pattern still holds. The rest gets easier.
The campus bot wasn't AWS-native. It wasn't production. It wasn't even particularly good by today's standards. But it was the project where I figured out that retrieval is mostly a routing problem, that agents are just functions with confidence thresholds, and that telling users "I don't know" is a feature.
Three years later I work on production agentic systems at scale. I use AWS actively from Bedrock to Telephony AWS connect. I am also exploring speech-to-speech models like Nova Sonic that sounds so innovative. The patterns I use are bigger and more polished versions of the same ideas.
You don't need to start with the polished version. Start with the version that helps your hostel-mate find the DBMS exam date.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article