How I Built My First RAG-Based Chatbot Using Amazon Bedrock and Knowledge Bases
In this article, I share my experience How a client requirement led me to explore RAG and build a chatbot using Amazon Bedrock, Knowledge Bases, Amazon S3, Titan Text Embeddings, S3 Vectors, and a small LLM Claude 3.7 Haiku.
It started with a simple client requirement:
“We need a chatbot that can answer questions using our own documents.”
Simple enough, right?
At first, I thought about the obvious solution: connect a large language model to the application and let it answer the questions.
But then came the important question:
How will the chatbot know what is inside our documents?
The information we needed wasn't something I could simply expect a foundation model to know. The chatbot had to work with our own PDF-based data.
So I started researching.
I wanted to understand how I could solve this problem using the AWS services I was already working with. During that research, I came across Retrieval-Augmented Generation, or RAG.
That was the point where the whole problem started making sense.
Instead of trying to train a model with our documents, I could build a system where the chatbot would retrieve the relevant information from our documents and then ask the foundation model to generate an answer using that information.
That became the approach I implemented.
And this was my first hands-on experience building a RAG-based chatbot.
The Idea Behind the Solution
Before getting into AWS, it helped me understand one important difference.
A normal interaction with an LLM looks something like this:
1
2
3
4
5
User
↓
LLM
↓
Answer
But that wasn't enough for our requirement.
We needed something closer to:
1
2
3
4
5
6
7
User
↓
Find relevant information
↓
Give that information to the LLM
↓
Generate an answer
That is essentially the idea behind RAG.
The model doesn't need to magically "know" everything about our documents.
Instead, we give it the relevant pieces of information at the time of the question.
Once I understood this, my next question became:
How do I build this architecture on AWS?
Why Amazon Bedrock Caught My Attention
While researching AWS options, I came across Amazon Bedrock Knowledge Bases.
This was particularly interesting because it could handle much of the knowledge retrieval workflow for me.
Instead of manually building every part of the pipeline, I could connect my documents to a Knowledge Base and let AWS handle the processing and retrieval infrastructure.
The architecture I eventually implemented looked like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Client PDF Documents
│
▼
Amazon S3
│
▼
Amazon Bedrock Knowledge Base
│
▼
Default Chunking
│
▼
Amazon Titan Text Embeddings
│
▼
S3 Vectors
│
▼
Relevant Information
│
▼
Claude 3.7 Haiku
│
▼
Chatbot Response
│
▼
Existing Application
Seeing all these services together was probably the most interesting part of the project for me.
Each service had a different job.
And once I understood those jobs, the whole architecture became much easier to reason about.
Step 1: Giving the Chatbot Something to Know
The first thing I needed was the actual knowledge.
In our case, the source data was stored in PDF documents.
I uploaded those documents to Amazon S3.
At this point, nothing particularly magical was happening.
I simply had:
1
2
3
PDFs
↓
Amazon S3
But S3 was now the source from which my Knowledge Base could build the chatbot's knowledge.
This also introduced an important concept for me:
The application's knowledge doesn't have to live inside the model.
The model can remain a foundation model while the application's specific knowledge lives somewhere else.
That separation is one of the things that makes RAG so useful.
Step 2: Creating the Knowledge Base
Next, I created an Amazon Bedrock Knowledge Base and connected it to the S3 data source.
My main configuration looked like this:
| Component | What I Used |
|---|---|
| Source data | PDF documents |
| Storage | Amazon S3 |
| Chunking | Default chunking |
| Embeddings | Amazon Titan Text Embeddings |
| Vector store | S3 Vectors |
| Foundation model | Claude 3.7 Haiku |
At this point, I had the pieces I needed for the ingestion and retrieval pipeline.
But there was still a question:
How does a PDF become something that a machine can search intelligently?
That brings us to embeddings.
Step 3: Turning Text Into Something Searchable
This was one of the concepts I found most interesting while learning RAG:
Embeddings.
Computers don't understand text in the same way humans do.
So the text from our documents needs to be converted into a numerical representation that captures its meaning.
For this project, I used Amazon Titan Text Embeddings.
The simplified process was:
1
2
3
4
5
6
7
8
9
PDF
↓
Text
↓
Chunks
↓
Titan Text Embeddings
↓
Vector Representation
Imagine my document contains information about Amazon S3.
A user might ask:
"Which AWS service is used for object storage?"
The exact words in the question may not appear in the document.
But the meaning can still be related.
Embeddings help the retrieval system identify that semantic relationship.
This was one of those concepts that made much more sense once I saw it as part of a real application rather than just reading about it.
Step 4: Why Do We Need Chunking?
There was another important piece of the puzzle:
We can't treat an entire PDF as one giant block of information.
Imagine having a 100-page document and asking:
"Find information related to this particular topic."
Searching the entire document as one unit wouldn't be very useful.
So the document is broken down into smaller pieces called chunks.
For my first implementation, I used the default chunking strategy provided by the Knowledge Base.
Conceptually:
1
2
3
4
5
6
7
8
9
PDF
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Chunk 1 Chunk 2 Chunk 3
│ │ │
└─────────────┼─────────────┘
▼
Embeddings
I intentionally didn't start by tuning everything.
This was my first RAG implementation, so my goal was to understand the complete pipeline first.
Once I had the basic architecture working, I knew I could come back later and experiment with different chunking strategies.
Step 5: Where Do All Those Vectors Go?
Once the document chunks were converted into embeddings, those vectors needed somewhere to live.
For this implementation, I used S3 Vectors as the vector store.
So the ingestion pipeline became:
1
2
3
4
5
6
7
PDF
↓
Chunks
↓
Titan Text Embeddings
↓
S3 Vectors
This is where the pieces started coming together for me.
I now had:
- The original documents in S3
- Smaller chunks of information
- Vector representations of those chunks
- A vector store that could be used during retrieval
The documents were no longer just files sitting in storage.
They had become a searchable knowledge source.
Step 6: The Knowledge Base Sync
After configuring the Knowledge Base, I synced the data.
This triggered the processing pipeline:
1
2
3
4
5
6
7
8
9
S3 Documents
↓
Knowledge Base
↓
Chunking
↓
Titan Text Embeddings
↓
S3 Vectors
Once the synchronization completed, the Knowledge Base was ready to retrieve relevant information.
This was a satisfying point in the implementation because the original PDFs had now gone through the entire ingestion pipeline.
But I still needed to connect this knowledge to the application.
Step 7: Connecting AWS to the Existing Codebase
This was where the project moved from AWS console configuration to actual application development.
After creating and syncing the Knowledge Base, I received its Knowledge Base ID.
I then used that ID in the existing application codebase so that the application could interact with the Knowledge Base.
The flow became:
1
2
3
4
5
6
7
8
9
10
11
User
↓
Existing Application
↓
Knowledge Base
↓
Retrieve Relevant Information
↓
Foundation Model
↓
Response
This was an important part of the project for me.
I wasn't simply creating a chatbot as an isolated AWS experiment.
I was taking a cloud service, understanding how it worked, and integrating it into an existing application to solve an actual requirement.
That made the project feel much closer to the kind of work we'd encounter in a production environment.
Step 8: Bringing Claude Into the Picture
Now we had the retrieval side.
But we still needed something to turn the retrieved information into a natural conversation.
For that, I used Claude 3.7 Haiku as the foundation model.
The complete request flow was now something like:
1
2
3
4
5
6
7
8
9
10
11
User asks a question
↓
Knowledge Base finds relevant information
↓
Relevant context is retrieved
↓
Claude 3.7 Haiku receives the context
↓
Claude generates the response
↓
User receives the answer
And this was probably the biggest conceptual takeaway from the project:
The LLM isn't the entire application.
It's one component of the application.
The retrieval system finds the knowledge.
The foundation model uses that knowledge to generate the response.
The application connects everything together.
So, Did I Train Claude?
No.
And this distinction is important.
When I initially started researching the requirement, I was thinking about how to make the model understand our data.
But with RAG, I didn't need to retrain the foundation model with the client's PDFs.
Instead, the documents remained external knowledge.
When a question arrived, the system retrieved the relevant information and provided it to the foundation model as context.
So instead of:
1
2
3
4
5
Our Documents
↓
Train Model
↓
New Model
the architecture was:
1
2
3
4
5
6
7
8
9
Our Documents
↓
Knowledge Base
↓
Retrieve Relevant Information
↓
Claude
↓
Answer
Understanding this difference completely changed how I thought about building AI applications.
The Complete Architecture
After putting everything together, my first RAG implementation looked like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
┌─────────────────┐
│ Client PDFs │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Amazon S3 │
└────────┬────────┘
│
▼
┌────────────────────────────┐
│ Bedrock Knowledge Base │
└─────────────┬──────────────┘
│
▼
┌─────────────────┐
│ Default Chunking│
└────────┬────────┘
│
▼
┌────────────────────────┐
│ Titan Text Embeddings │
└────────────┬───────────┘
│
▼
┌────────────┐
│ S3 Vectors │
└─────┬──────┘
│
Retrieval
│
▼
┌──────────────────┐
│ Relevant Context │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Claude 3.7 Haiku │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Chatbot Response │
└──────────────────┘
It looks complicated when all the components are shown together.
But once I understood what each component was responsible for, the architecture became surprisingly intuitive.
What This Project Taught Me
The biggest thing I learned wasn't simply how to create a Bedrock Knowledge Base.
It was learning how to approach an AI requirement as an engineering problem.
I started with:
"The client needs a chatbot that can answer questions from our documents."
I didn't immediately know the architecture.
So I researched.
Then I learned about RAG.
Then I explored how AWS could provide the pieces I needed.
And eventually the problem became:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Client Requirement
↓
Research
↓
RAG
↓
Amazon Bedrock
↓
Knowledge Base
↓
S3 + Embeddings + Vector Store
↓
Foundation Model
↓
Existing Application
That journey was more valuable to me than simply following a tutorial.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article