I Built My Own AI Assistant — Then I Realized the LLM Was the Easy Part 🤖
What started as a simple experiment with an LLM quickly became a deeper engineering challenge. In this article, I share my journey of building FRIDAY, a personal AI assistant using LangChain, and the lessons I learned while adding tools, memory, RAG, file analysis, web access, and multi-step agent workflows.
I Tried Building My Own AI Assistant — Then I Realized the LLM Was the Easy Part 🤖
What started as a simple chatbot slowly turned into a much bigger engineering problem: memory, tools, RAG, files, planning, failures, security, and one uncomfortable question — who is actually in control?
It started with a simple question
“What if I built my own AI assistant instead of just using one?”
At first, the idea sounded simple.
Give an LLM a prompt.
Ask it a question.
Get an answer.
Done.
So I started building FRIDAY, my own AI assistant using Python and LangChain.
And within a surprisingly short time, I discovered something:
Calling an LLM is easy. Building a system around an LLM is not.
The moment I wanted FRIDAY to do something beyond conversation, the architecture started changing.
I wanted it to remember previous interactions.
I wanted it to search the web.
I wanted it to understand files.
I wanted it to work with data.
I wanted it to use tools.
I wanted it to reason through multi-step tasks.
And eventually, I realized I wasn't building a chatbot anymore.
I was building a system.
1. The chatbot illusion
A basic AI application can look like this:
1
2
3
4
5
6
7
User
↓
Prompt
↓
LLM
↓
ResponseIt looks powerful.
But ask the system to perform a real-world task and the limitations become obvious.
Imagine asking:
“Analyze this CSV, find the biggest trend, search for additional context, and explain the result.”
A normal chatbot can generate an answer.
But it can't magically:
- open the CSV
- inspect the data
- calculate statistics
- search for external information
- decide which tool to use
- verify the result
- combine everything
- explain what happened
unless we build those capabilities around the model.
That led me to a different mental model.
Instead of:
LLM = AI assistant
I started thinking:
LLM + tools + knowledge + memory + orchestration + safety = AI assistant
And that changed the entire project.
2. The architecture started with one brain
The first component was the LLM.
FRIDAY needs a reasoning engine capable of understanding natural-language requests and deciding what should happen next.
But I quickly learned an important distinction:
The model should not have to know everything. It should know what it can do.
That's where tools entered the architecture.
A simplified version looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
┌──────────────┐
│ User │
└──────┬───────┘
↓
┌──────────────┐
│ FRIDAY │
│ Agent │
└──────┬───────┘
↓
┌────────────────┼────────────────┐
↓ ↓ ↓
Web Tool File Tool Code Tool
↓ ↓ ↓
Internet Files PythonNow the LLM isn't just producing text.
It is deciding:
“What do I need to accomplish this task?”
That is a completely different problem.
3. Then came the tool problem
I created tools for different capabilities.
For example:
1
2
3
4
def add(a: int, b: int) -> int:
"""Add two numbers."""
return a + bAnd:
1
2
3
4
def multiply(a: int, b: int) -> int:
"""Multiply two numbers."""
return a * bThe important part wasn't the functions themselves.
It was the interface between the LLM and the tool.
The model needs to understand:
- what the tool does
- when to use it
- what arguments it requires
- what result it will return
That gave FRIDAY a basic loop:
1
2
3
4
5
6
7
8
9
10
11
Understand
↓
Choose tool
↓
Execute
↓
Observe result
↓
Decide what to do next
↓
RespondAnd suddenly, the application wasn't simply generating answers.
It was taking actions.
4. But tools created a new problem
Here's where things became interesting.
If an AI can use tools, then the question changes from:
“Can the model answer correctly?”
to:
“Can the model choose the correct action?”
Those are not the same problem.
A model might understand a user's request perfectly and still choose the wrong tool.
Or call a tool unnecessarily.
Or provide incomplete arguments.
Or repeat the same action.
Or misunderstand the result.
So I started thinking about agent evaluation differently.
Instead of testing only:
Input → Answer
I needed to test:
Input → Plan → Tool → Result → Next decision → Final answer
That is a much bigger surface area for failure.
5. Then I gave FRIDAY a memory
A useful assistant shouldn't feel like it forgets everything after every message.
But “memory” isn't simply storing every conversation forever.
That creates another problem:
What should the system remember?
Suppose I have a 100-message conversation.
Should all 100 messages be placed into the next prompt?
Probably not.
More context doesn't automatically mean better context.
So memory became an engineering decision:
1
2
3
4
5
6
7
8
9
Conversation
↓
What is important?
↓
What should be retained?
↓
What can be summarized?
↓
What should be retrieved later?This also connects to a broader challenge in RAG and agent systems: context has to be selected carefully rather than simply made larger. Current AWS Builder Center discussions around RAG and agents similarly emphasize retrieval, context, tools, and orchestration as separate architectural concerns.
6. Then I added RAG
And this introduced another realization.
An LLM may be powerful, but my application knows things the model doesn't.
My documents.
My project files.
My notes.
My knowledge base.
So instead of asking the model to remember everything, I introduced Retrieval-Augmented Generation.
The basic flow became:
1
2
3
4
5
6
7
8
9
User Question
↓
Retrieve Relevant Information
↓
Build Context
↓
LLM
↓
AnswerBut RAG created another question:
What happens when retrieval returns the wrong information?
The model can generate a beautiful answer from bad context.
That's one of the reasons I stopped thinking of RAG as:
“Add a vector database and you're done.”
Retrieval quality matters.
Chunking matters.
Embeddings matter.
Metadata matters.
Context selection matters.
And evaluation matters.
AWS's current RAG guidance similarly treats retrieval as a distinct part of the architecture rather than assuming the model alone solves the problem.
7. Files changed the game again
Then I wanted FRIDAY to work with uploaded files.
A user could provide:
- CSV files
- documents
- text
- project files
- datasets
Now the assistant had to understand external information instead of only receiving a text prompt.
For a data-analysis request, the workflow could look like:
1
2
3
4
5
6
7
8
9
10
11
Upload CSV
↓
Inspect structure
↓
Understand columns
↓
Analyze data
↓
Generate findings
↓
Explain findingsBut there was an important design principle here:
Don't ask the LLM to perform work that deterministic code can perform better.
If Python can calculate an average accurately, let Python calculate it.
If SQL can filter a dataset, let SQL do it.
If a program can validate a value, don't ask an LLM to guess.
The LLM should help with reasoning and interpretation, while deterministic tools handle operations that should be exact.
That separation became one of the most important ideas in my project.
8. The agent became more capable — and more dangerous
This was probably the most interesting part of building FRIDAY.
Every new capability made the assistant more useful.
But every capability also created a new failure mode.
Add web access?
→ What if retrieved content is malicious?
Add file access?
→ What if a file contains instructions intended to manipulate the agent?
Add code execution?
→ What can the generated code access?
Add external APIs?
→ What happens if the wrong tool is called?
Add memory?
→ What information should be retained?
Add multiple tools?
→ How do we prevent unnecessary tool loops?
So I started treating capability and control as two sides of the same architecture.
1
2
3
4
5
6
7
More Capability
↓
More Possible Actions
↓
More Possible Failure Modes
↓
More Need for ControlsThis is one reason production agent architecture is significantly more complicated than a simple LLM application. AWS's current Agentic AI guidance also emphasizes areas such as traceability, human oversight, evaluation, and controlled execution.
9. The biggest lesson: an agent is not just an LLM
After working through these pieces, my mental model changed completely.
I no longer see an AI assistant as:
1
LLM + PromptI see it more like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
┌─────────────────┐
│ User │
└────────┬────────┘
↓
┌─────────────────┐
│ Agent / Planner │
└────────┬────────┘
↓
┌────────────┬───────┼────────┬────────────┐
↓ ↓ ↓ ↓ ↓
Memory RAG Tools Files Web
↓ ↓ ↓ ↓ ↓
└────────────┴───────┼────────┴────────────┘
↓
┌─────────────────┐
│ Validation │
│ & Guardrails │
└────────┬────────┘
↓
Final ResponseThe LLM is important.
But it is only one component.
10. The question I started asking next
Once FRIDAY worked locally, another question became unavoidable:
“What happens when this needs to become a real application?”
A local prototype can get away with a lot.
Production systems can't.
Suddenly we have to think about:
- authentication
- authorization
- secrets
- observability
- scaling
- latency
- cost
- data storage
- model selection
- retries
- failure handling
- deployment
- evaluation
- security
And this is where my AWS learning became much more interesting.
Instead of learning AWS services individually, I started asking:
“Which part of my AI system does this service actually solve?”
11. From local prototype to AWS architecture
I started mapping the pieces conceptually.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Users
↓
API / Frontend
↓
Application Layer
↓
┌─────────────┼─────────────┐
↓ ↓ ↓
Storage AI / LLM Tools
↓ ↓ ↓
S3 Bedrock Lambda
↓ ↓ ↓
└─────────────┼─────────────┘
↓
Data / Memory
↓
ObservabilityThe goal isn't to replace every local component with an AWS service just because it exists.
The goal is to understand why each managed component belongs in the architecture.
AWS's current guidance for production generative-AI systems similarly emphasizes decomposing prototypes into modular components and considering performance, cost, control, and observability rather than carrying a monolithic prototype directly into production.
12. The architecture I would build next
If I were taking FRIDAY toward a more production-oriented architecture, I'd think in layers.
Layer 1 — Interaction
The user communicates with FRIDAY through a web or application interface.
Layer 2 — Orchestration
The agent determines what the task requires.
Layer 3 — Knowledge
RAG provides access to relevant information.
Layer 4 — Tools
The assistant can interact with approved external capabilities.
Layer 5 — Deterministic execution
Python, SQL, APIs, and application logic perform operations that need predictable results.
Layer 6 — Storage
Conversation state, documents, metadata, and application data need durable storage.
Layer 7 — Security & observability
Actions need boundaries.
Failures need visibility.
Important executions need traces.
And costs need monitoring.
This is where an AI project starts looking much more like a distributed software system than a chatbot.
13. The part I got wrong at the beginning
When I started, I thought the hardest part would be choosing the right model.
It wasn't.
Then I thought the hardest part would be making the agent intelligent.
It wasn't.
The harder questions were:
What should the model be allowed to do?
What information should it see?
When should it use a tool?
What happens when the tool fails?
How do we know whether the answer was actually correct?
How do we stop one bad decision from becoming a chain of bad actions?
That's when I realized:
Building an AI assistant is less about making the model smarter and more about designing the system around it intelligently.
14. What I would test before calling FRIDAY “reliable”
I don't think an AI assistant should be judged by five impressive demo questions.
I'd test it against failure.
For example:
Basic reasoning
Can it understand straightforward requests?
Tool selection
Does it choose the correct tool?
Tool failure
What happens when the tool returns an error?
RAG
What happens when the answer isn't present in the knowledge base?
Conflicting information
Can it recognize contradictory sources?
Files
Can it safely analyze an uploaded dataset?
Prompt injection
What happens when untrusted content tells the agent to ignore its instructions?
Long conversations
Does memory remain useful or become noise?
Multi-step tasks
Does the agent eventually reach the goal without getting stuck in loops?
Ambiguous requests
Does it ask for clarification instead of confidently guessing?
These tests are much more valuable to me than simply asking:
“Does the chatbot work?”
15. Where FRIDAY is going next
FRIDAY is still a learning project.
That's important to say.
I'm not presenting it as a production-ready autonomous system.
Instead, I'm using it as a laboratory for understanding how modern AI systems actually work.
The next areas I want to explore are:
- stronger RAG pipelines
- better memory strategies
- structured tool interfaces
- agent evaluation
- observability
- security and prompt-injection resistance
- cloud deployment
- cost-aware model selection
- more reliable multi-step planning
- AWS-native architecture
And eventually, I want to understand something even more important:
How much autonomy should an AI agent actually have?
Because technically being able to do something doesn't necessarily mean the system should be allowed to do it automatically.
16. One final thought
When I started FRIDAY, I wanted to build an AI that could do more.
Now I'm equally interested in building an AI that knows when not to act.
That might be the more important engineering problem.
A powerful model can generate text.
A tool-enabled agent can take actions.
But a reliable AI system needs something more:
boundaries.
The future of AI assistants may not simply be about giving models more tools.
It may be about building better systems around those tools — systems that know what information to use, what actions are allowed, when to ask for help, and when to stop.
And that's the part of AI engineering I'm most interested in learning next.
What would you prioritize?
If you were taking a local AI assistant like FRIDAY toward production, what would you solve first:
① Better reasoning
② Better RAG
③ More tools
④ Security & guardrails
⑤ Evaluation & observability
⑥ Cost and scalability
② Better RAG
③ More tools
④ Security & guardrails
⑤ Evaluation & observability
⑥ Cost and scalability
I'd genuinely like to hear how other builders would approach it.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article