Beyond the Chatbot: Building a Document-Aware AI Assistant with Amazon Bedrock and RAG
Learn how to build a document-aware AI assistant using Retrieval-Augmented Generation (RAG), Amazon Bedrock, Amazon S3, and a vector knowledge base to generate responses grounded in your own data.
Beyond the Chatbot: Building a Document-Aware AI Assistant with Amazon Bedrock and RAG
Introduction
Large language models (LLMs) have changed how we interact with software. They can explain concepts, summarize information, generate code, and answer questions using natural language. However, when an application needs to answer questions about private documents, internal procedures, or frequently updated information, a general-purpose model has an important limitation: it may not have access to the information required to provide an accurate answer.
One solution is Retrieval-Augmented Generation (RAG), an approach that combines information retrieval with generative AI. Instead of relying exclusively on the model's existing knowledge, a RAG application retrieves relevant information from an external knowledge source and provides it as context for generating a response.
In this article, we will explore how to design a document-aware AI assistant using Amazon Bedrock, Amazon S3, and Amazon Bedrock Knowledge Bases. We will examine the architecture, the retrieval workflow, implementation considerations, and the security practices needed to build a more reliable AI application.
1. Why Traditional Chatbots Are Not Enough
Imagine building an assistant that helps students understand university procedures. Users might ask questions such as:
- What documents are required to complete an administrative process?
- What are the deadlines for submitting a report?
- Which steps should I follow to request a particular service?
A general-purpose LLM might provide a plausible answer, but it may not know the institution's specific rules. Even worse, it could generate an answer that sounds correct but contradicts the official documentation.
Providing more instructions in the prompt does not solve every problem. The application still needs a reliable way to access the relevant information and keep it updated.
This is where RAG becomes useful. It separates two responsibilities:
- Retrieval: Find information relevant to the user's question.
- Generation: Use that information to formulate a clear, contextual response.
The model still generates the answer, but the application supplies evidence that the model would not otherwise have.
2. Understanding the RAG Workflow
A RAG system typically has two major workflows: knowledge ingestion and question answering.
Knowledge ingestion
Before users can ask questions, the application needs to prepare its source documents.
A typical ingestion pipeline includes:
- Upload documents to a storage location.
- Extract text from supported file formats.
- Divide the content into smaller chunks.
- Generate vector embeddings that represent the semantic meaning of each chunk.
- Store the content and its vector representation in a searchable knowledge store.
Chunking matters because a document may contain thousands of words, while only a few paragraphs are relevant to a particular question. Smaller, meaningful chunks allow the retrieval system to identify specific information instead of passing entire documents to the model.
Question answering
When a user submits a question, the system performs a different sequence:
- Receive the question.
- Convert it into a representation suitable for semantic search.
- Retrieve relevant document chunks.
- Provide the retrieved context and the question to an LLM.
- Generate a response based on the available evidence.
- Return the answer and, where supported, references to the source material.
The process can be summarized as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
KNOWLEDGE INGESTION
Documents --> Amazon S3 --> Bedrock Knowledge Base
|
v
Chunking and Embeddings
|
v
Vector Data Store
QUESTION ANSWERING
User Question --> Retrieve Relevant Context
|
v
Amazon Bedrock LLM
|
v
Grounded Answer + SourcesThe main advantage is that the model receives information relevant to the current request rather than being expected to know every detail in advance.
3. Designing the AWS Architecture
AWS provides managed services that simplify several parts of a RAG application.
The proposed architecture uses the following components:
- Amazon S3: Stores source documents such as PDFs, manuals, and guides.
- Amazon Bedrock Knowledge Bases: Coordinates document ingestion, retrieval, and integration with supported embedding and vector storage configurations.
- Amazon Bedrock foundation models: Generate natural-language responses using the retrieved context.
- An application backend: Receives user questions, invokes the appropriate APIs, and applies application-specific validation and authorization.
- A web frontend: Provides the conversational interface.
The high-level architecture looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
+-------------------+
| Web Client |
+---------+---------+
|
v
+-------------------+
| Application API |
+---------+---------+
|
v
+-------------------+
| Amazon Bedrock |
| Knowledge Bases |
+---------+---------+
|
Retrieve relevant data
|
v
+-------------------+
| Foundation Model |
| via Amazon Bedrock|
+---------+---------+
|
v
+-------------------+
| Answer and |
| Source References |
+-------------------+
Documents --> Amazon S3 --> Knowledge Base IngestionThe application backend can be implemented with AWS Lambda and Amazon API Gateway, although other hosting options are possible. The important design decision is to keep the frontend separate from privileged AWS operations and to enforce authorization on the server side.
The precise vector store, embedding model, and supported configuration depend on the Knowledge Bases setup selected for the project.
4. Retrieving Relevant Information with Amazon Bedrock
Amazon Bedrock Knowledge Bases provides managed capabilities for connecting foundation models with external data sources. It can reduce the amount of custom ingestion and retrieval infrastructure a developer needs to build.
For example, an application can use the
RetrieveAndGenerate operation to retrieve relevant content and generate a response through a supported foundation model.The following Python example illustrates the application-side interaction using the AWS SDK for Python, also known as Boto3.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import os
import boto3
bedrock_agent_runtime = boto3.client(
"bedrock-agent-runtime",
region_name=os.environ["AWS_REGION"]
)
def ask_knowledge_base(question: str) -> dict:
if not question or not question.strip():
raise ValueError("A question is required")
response = (
bedrock_agent_runtime.retrieve_and_generate(
input={
"text": question.strip()
},
retrieveAndGenerateConfiguration={
"type": "KNOWLEDGE_BASE",
"knowledgeBaseConfiguration": {
"knowledgeBaseId": os.environ[
"KNOWLEDGE_BASE_ID"
],
"modelArn": os.environ[
"BEDROCK_MODEL_ARN"
]
}
}
)
)
return {
"answer": response["output"]["text"],
"citations": response.get("citations", [])
}This example assumes that the Knowledge Base has already been configured, its data source has been ingested, and the selected model is supported for the operation in the chosen AWS Region. The execution role must also have the permissions required for the relevant Bedrock operations.
The application can call this function from a backend endpoint and return the answer to the frontend.
For example, a frontend request might contain:
1
2
3
{
"question": "What documents are required to submit the report?"
}The backend validates the request, invokes the retrieval-and-generation workflow, and returns the result.
A production API should additionally enforce request-size limits, handle AWS service exceptions, apply authentication and authorization, and return consistent error responses. It should also avoid exposing internal exception details to users.
5. Grounding Responses in Source Documents
Retrieving information is only one part of building a useful RAG system. The application must also encourage the model to distinguish between supported facts and missing information.
A useful instruction for the generation step might be:
1
2
3
4
5
6
7
8
9
10
11
Answer the user's question using only the supplied context.
If the context does not contain enough information,
state that the answer cannot be determined from
the available documents.
Do not invent deadlines, requirements, policies,
or facts.
When possible, identify the source supporting
the answer.This instruction establishes an important behavior: the model should be allowed to acknowledge uncertainty instead of filling gaps with plausible-sounding information.
The application should also preserve and present the source references returned by the retrieval workflow. For example, a response could include the document name and the relevant source location when that information is available.
However, citations do not automatically guarantee correctness. A retrieved passage might be outdated, incomplete, or unrelated to the exact question. Developers should evaluate whether the retrieved evidence actually supports the generated response.
For high-impact use cases, the system may need stricter rules for abstaining, additional verification, or human review.
6. Improving Retrieval Quality
A RAG application is only as useful as the information it retrieves.
If the system retrieves irrelevant passages, even a capable foundation model may generate an incomplete or incorrect answer.
Several factors influence retrieval quality.
Document quality: Source files should contain accurate, current, and readable information. Poorly extracted text can introduce errors before the model receives any context.
Chunking strategy: Very small chunks may lose important context, while very large chunks may include too much unrelated information. The appropriate size depends on the document structure and the questions users are expected to ask.
Metadata: Attributes such as document type, department, publication date, and version can help filter or organize information. Metadata is particularly useful when different users should access different collections of documents.
Retrieval configuration: The number of retrieved chunks and the retrieval strategy affect both relevance and the amount of context sent to the model.
Evaluation: Developers should create a representative set of questions and expected evidence, then measure whether the system retrieves the right passages and generates supported answers.
A useful evaluation process separates retrieval quality from generation quality. If the relevant information never reaches the model, changing the prompt alone is unlikely to fix the underlying problem.
For more demanding use cases, developers can evaluate techniques such as hybrid search, reranking, and query rewriting where supported by the chosen architecture.
7. Security and Data Privacy
Connecting an LLM to private documents introduces security considerations that do not disappear simply because the application uses managed AWS services.
The first principle is least privilege. The application's execution role should have only the permissions required to perform its operations. Access to source documents, the Knowledge Base, and foundation models should be scoped appropriately.
Other important practices include:
- Authenticate users before allowing access to protected information.
- Enforce authorization on the backend, not only in the frontend.
- Apply document-level or collection-level access controls where required.
- Avoid exposing AWS credentials in browser code.
- Encrypt data in transit and configure encryption at rest.
- Review logging policies to prevent sensitive content from being unnecessarily recorded.
- Monitor access and application behavior using appropriate AWS logging and security services.
There is also an AI-specific risk: prompt injection. A document might contain instructions that attempt to manipulate the model, such as asking it to ignore previous rules or reveal sensitive information.
Retrieved documents should be treated as untrusted data, not as privileged instructions. Prompts, application logic, and authorization checks must remain responsible for controlling the system's behavior.
A model instruction alone is not a security boundary. If users must not access a particular document, the retrieval and authorization layers must enforce that restriction before the content is provided to the model.
8. Managing Latency and Costs
A RAG request involves several operations, and each can affect the user experience.
The application may incur costs for model inference, embedding generation, vector storage, document processing, and supporting infrastructure. Exact pricing depends on the selected models, AWS Region, configuration, and usage.
To manage costs effectively, developers should:
- Select a model that meets the application's quality requirements without unnecessary complexity.
- Avoid sending excessive document context to the generation model.
- Monitor token usage and request volume.
- Evaluate caching for repeated, appropriate queries.
- Configure budgets and cost alerts.
- Measure end-to-end latency rather than considering model latency alone.
Caching requires special care. If answers depend on user permissions or frequently updated documents, a cached response must not be reused in a way that exposes information to unauthorized users or serves outdated content.
It is also important to measure retrieval quality before optimizing aggressively for speed. A fast answer that uses the wrong document is not a successful result.
9. From Prototype to Production
A basic RAG assistant can demonstrate the value of connecting an LLM to external knowledge, but production systems require additional engi
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article