Building a Document Q&A Application with Amazon Bedrock and Retrieval-Augmented Generation
Learn how Retrieval-Augmented Generation (RAG) can combine your own documents with foundation models on Amazon Bedrock to build more reliable question-answering applications.
Building a Document Q&A Application with Amazon Bedrock and Retrieval-Augmented Generation
Generative AI models can answer questions across a wide range of topics, but they may not have access to an organization's private or recently updated information. Simply providing a larger prompt is not a scalable solution.
Retrieval-Augmented Generation (RAG) addresses this problem by retrieving relevant information from a knowledge source and providing that information to a foundation model before generating an answer.
In this article, we will understand how a document question-answering application can be designed using Amazon Bedrock and a RAG architecture.
What Is RAG?
Retrieval-Augmented Generation combines two operations:
- Retrieval — find relevant information from a knowledge base.
- Generation — provide that information to a foundation model to generate an answer.
Instead of asking a model:
1
User → Foundation Model → Answera RAG application follows this flow:
1
2
3
4
5
6
7
8
9
User Question
↓
Retrieve Relevant Information
↓
Relevant Context
↓
Foundation Model
↓
Generated AnswerThis allows the model to use information that is specific to the application's knowledge source.
Why Use RAG?
Consider an organization that has thousands of internal documents.
Examples include:
- Product documentation
- Technical manuals
- Company policies
- Research papers
- Support documentation
- Internal knowledge bases
A general-purpose foundation model may not contain this private information.
RAG allows the application to retrieve the relevant information at query time instead of retraining the model whenever the underlying documents change.
High-Level Architecture
A typical document Q&A system can be represented as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
┌─────────────────┐
│ User │
└────────┬────────┘
│
▼
┌─────────────────┐
│ User Question │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Retriever │
└────────┬────────┘
│
Relevant Documents
│
▼
┌─────────────────┐
│ Amazon Bedrock │
│ Foundation Model│
└────────┬────────┘
│
▼
┌─────────────────┐
│ Answer │
└─────────────────┘The retrieval layer finds relevant content, and Amazon Bedrock can then use that context when generating the response.
Step 1: Prepare the Documents
The first stage is collecting the documents that the application needs to understand.
For example:
1
2
3
4
5
documents/
├── aws-guide.pdf
├── architecture.pdf
├── security-guide.pdf
└── developer-guide.pdfLarge documents are usually divided into smaller sections or chunks.
For example:
1
2
3
4
5
6
Original Document
↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4Chunking makes it easier to retrieve only the relevant portions of a document.
Step 2: Convert Text into Embeddings
The text can be represented as numerical vectors called embeddings.
Conceptually:
1
2
3
4
5
"Amazon S3 stores objects"
↓
Embedding Model
↓
[]Semantically similar text produces vectors that are close to each other in vector space.
This allows the retrieval system to search for information based on meaning rather than only exact keyword matches.
Step 3: Store the Knowledge
The document chunks and their embeddings need to be stored in a system that supports retrieval.
A RAG system can use a vector-capable knowledge store to find content that is semantically similar to the user's question.
For example:
1
2
3
4
5
6
7
User Question
↓
Create Query Embedding
↓
Vector Search
↓
Top Relevant ChunksThe retrieved chunks become the context for the generation step.
Step 4: Retrieve Relevant Context
Suppose the user asks:
1
How does Amazon S3 provide data durability?The retrieval system searches the knowledge base and might return:
1
2
3
4
5
6
7
8
Chunk 1:
Amazon S3 is designed to provide high durability for stored objects.
Chunk 2:
S3 stores data redundantly across multiple facilities...
Chunk 3:
S3 provides different storage classes for different workloads...Only the most relevant content needs to be passed to the foundation model.
Step 5: Generate the Answer
The retrieved context can then be combined with the user's question.
Conceptually:
1
2
3
4
5
6
7
8
9
10
11
12
13
Context:
[Relevant document chunks]
Question:
How does Amazon S3 provide data durability?
↓
Amazon Bedrock Foundation Model
↓
Generated AnswerThe model can use the supplied context instead of relying entirely on information contained in its original training data.
A Simple RAG Prompt
A simplified prompt could look like:
1
2
3
4
5
6
7
8
9
Use the following context to answer the question.
Context:
{retrieved_context}
Question:
{user_question}
Answer using only the information available in the context.This approach can reduce the chance of generating information that is unrelated to the application's knowledge base.
Why Amazon Bedrock?
Amazon Bedrock provides access to foundation models through managed AWS APIs.
This means developers can build generative AI applications without managing the underlying model infrastructure themselves.
A RAG application can therefore separate responsibilities:
1
2
3
4
5
6
7
Document Storage
↓
Retrieval
↓
Amazon Bedrock
↓
Generated ResponseThis architecture also makes it possible to change models while keeping much of the surrounding application architecture intact.
Improving Retrieval Quality
Retrieval quality directly affects the quality of the generated answer.
Several factors matter:
Chunk Size
Very small chunks may lose important context.
Very large chunks may contain too much irrelevant information.
The appropriate size depends on the document structure and application.
Number of Retrieved Results
Retrieving too few chunks can omit important information.
Retrieving too many can introduce irrelevant context and increase the amount of information sent to the model.
Metadata
Adding metadata such as:
1
2
3
4
5
document_name
department
category
date
versioncan help filter and organize retrieved information.
Document Quality
Poorly structured or outdated documents can produce poor answers even when the retrieval system works correctly.
RAG is not a substitute for maintaining a reliable knowledge base.
Handling Hallucinations
RAG can reduce hallucinations, but it does not eliminate them.
A robust application should:
- Provide relevant context to the model.
- Instruct the model to stay within the supplied context.
- Return source references when appropriate.
- Handle cases where no relevant information is retrieved.
- Evaluate responses using representative questions.
- Monitor the quality of retrieved documents.
For example, if the retrieval system finds no useful information, the application can respond:
1
2
I couldn't find enough information in the knowledge base
to answer this question.This is better than forcing the model to produce an unsupported answer.
Security Considerations
Enterprise RAG applications often process sensitive documents, so security needs to be designed into the architecture.
Important considerations include:
- Use IAM to control access to AWS resources.
- Encrypt data at rest and in transit.
- Apply least-privilege permissions.
- Control which users can access specific documents.
- Avoid exposing sensitive document content unnecessarily.
- Monitor application activity and access patterns.
Authorization should happen before retrieving information. A user should not be able to retrieve documents simply because those documents contain relevant information.
Common Use Cases
RAG can be applied to many applications:
- Internal company assistants
- Technical documentation assistants
- Customer support systems
- Research assistants
- Educational applications
- Legal document search
- Product knowledge assistants
- Enterprise knowledge bases
For example, a university could build a student assistant that retrieves information from:
1
2
3
4
5
6
7
8
9
10
11
12
13
Academic Regulations
+
Course Materials
+
Examination Guidelines
+
College Policies
↓
RAG System
↓
Student Question
↓
AnswerRAG vs Fine-Tuning
RAG and fine-tuning solve different problems.
| RAG | Fine-Tuning |
|---|---|
| Adds external knowledge at query time | Changes model behavior through additional training |
| Useful for frequently changing information | Useful for specialized behavior or style |
| Can provide source context | Does not inherently provide sources |
| Knowledge remains outside the model | Information influences the trained model |
| Often easier to update | Updating requires another training process |
For frequently changing organizational information, RAG is often a more practical starting point than repeatedly fine-tuning a model.
Production Considerations
A proof-of-concept RAG application is relatively straightforward, but production systems require additional engineering.
Important areas include:
- Retrieval evaluation
- Response evaluation
- Access control
- Observability
- Latency optimization
- Cost management
- Document versioning
- Failure handling
- Prompt management
The most important point is that a RAG system is not simply an LLM connected to a vector database. Retrieval quality, document quality, authorization, and evaluation all affect the final system.
Conclusion
Retrieval-Augmented Generation provides a practical architecture for building AI applications that need access to external or private information.
A typical workflow is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Documents
↓
Chunking
↓
Embeddings
↓
Knowledge Store
↓
User Question
↓
Retrieval
↓
Relevant Context
↓
Amazon Bedrock
↓
Generated AnswerAmazon Bedrock provides the foundation for the generation layer, while the retrieval system supplies the application-specific context.
The result is an architecture that can connect generative AI with continuously evolving knowledge sources without requiring the foundation model itself to be retrained every time the documents change.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article