AWS Builder Center
Building Trustworthy AI Applications with Amazon Bedrock Knowledge Bases

Building Trustworthy AI Applications with Amazon Bedrock Knowledge Bases

Learn how Retrieval-Augmented Generation (RAG) can help AI applications answer questions using your own documents. This practical guide explains the architecture, implementation workflow, Python example, and key considerations for building more reliable generative AI applications with Amazon Bedrock.

CSE ’27 | Azure Fundamentals Certified | AI, Cloud & Cybersecurity | Aspiring SDE

Building Trustworthy AI Applications with Amazon Bedrock Knowledge Bases

Introduction

Generative AI applications can summarize documents, answer questions, and help users find information quickly. However, large language models (LLMs) can sometimes generate answers that sound convincing but are incorrect or unsupported by evidence.
One way to address this challenge is Retrieval-Augmented Generation (RAG). Instead of relying only on a model's learned knowledge, a RAG application retrieves relevant information from a configured data source and uses it as context when generating an answer.
Amazon Bedrock Knowledge Bases provides a managed way to build this type of workflow using your own data.
In this article, we will explore how RAG works, how to design a document question-answering application, and how to test whether its responses are useful and supported by evidence.

1. Understanding Retrieval-Augmented Generation

A traditional LLM generates an answer based primarily on its learned patterns and the prompt it receives. It may not have access to your latest documentation, internal policies, or project-specific information.
RAG adds a retrieval step before answer generation.
A typical workflow looks like this:
  1. Prepare documents: Collect relevant PDF, text, or other supported files.
  1. Create a knowledge base: Connect the documents to Amazon Bedrock Knowledge Bases.
  1. Process the data: Documents are processed into manageable chunks and represented as vector embeddings for semantic retrieval.
  1. Receive a question: A user submits a question to the application.
  1. Retrieve relevant information: The knowledge base searches for content related to the question.
  1. Generate an answer: The retrieved information is supplied as context to a foundation model.
  1. Return evidence: Where supported by the selected workflow, the response includes citations to the source material.
The result is an AI application that can use information from a defined collection of documents rather than relying entirely on general model knowledge.

2. Architecture of a Document Q&A Application

A simple architecture can use the following components:
  • Amazon S3: Stores the source documents.
  • Amazon Bedrock Knowledge Bases: Manages retrieval from the configured data source.
  • Amazon Bedrock foundation model: Generates natural-language responses using retrieved context.
  • Application backend: Sends user questions to the knowledge base and handles the response.
  • Frontend: Displays the answer and its supporting sources to the user.
The high-level flow is:
 User → Frontend → Backend → Amazon Bedrock Knowledge Base → Retrieved Context → Foundation Model → Answer with Sources 
This is a reference architecture, not a claim that the application has already been deployed.

3. Building the Application: Step by Step

Step 1: Prepare your documents

Start with a small collection of documents containing information you want your application to answer questions about.
For example, you could use:
  • Product documentation
  • Technical guides
  • Frequently asked questions
  • Public research papers
  • Sample company policies created for testing
For a first experiment, choose documents with clear headings and reliable content. Avoid uploading confidential or personal information unless you have the necessary authorization and security controls.

Step 2: Store the documents in Amazon S3

Create an S3 bucket in your AWS account and upload the documents.
Keep the source files organized so you can identify where retrieved information originated. Configure access permissions carefully and avoid making the bucket public simply to make the documents accessible to your application.

Step 3: Create an Amazon Bedrock Knowledge Base

Open the Amazon Bedrock console and navigate to Knowledge Bases.
Follow the setup workflow to configure a knowledge base, connect your supported data source, and select the required models and storage options.
The exact configuration depends on the knowledge base type, supported AWS Region, and available model choices.
After connecting the data source, run the ingestion or synchronization process required by your configuration. Wait for processing to complete before testing queries.

Step 4: Test retrieval and answer generation

Once your knowledge base is ready, submit questions that can be answered from the uploaded documents.
For example, if your document describes a software deployment process, ask:
  • What are the prerequisites for deployment?
  • Which configuration values are required?
  • What troubleshooting steps are recommended?
  • Which source section supports the answer?
Amazon Bedrock Knowledge Bases provides the  Retrieve  operation for retrieving relevant source content and  RetrieveAndGenerate  for retrieving information and generating an answer in a combined workflow.
The generated response can include citations to the retrieved source material, helping the application present evidence alongside its answer.

Step 5: Connect the knowledge base to your backend

A Python backend can call the Amazon Bedrock Agent Runtime API through the AWS SDK for Python, commonly known as Boto3.
First, install the SDK:
pip install boto3
Next, configure AWS credentials using an appropriate AWS authentication method. For a deployed application, prefer an IAM role with the minimum permissions required instead of hardcoding access keys.
The following is an illustrative example of querying an existing knowledge base:
import boto3

client = boto3.client(
"bedrock-agent-runtime",
region_name="YOUR_AWS_REGION"
)

def ask_question(question: str) -> dict:
response = client.retrieve_and_generate(
input={"text": question},
retrieveAndGenerateConfiguration={
"type": "KNOWLEDGE_BASE",
"knowledgeBaseConfiguration": {
"knowledgeBaseId": "YOUR_KNOWLEDGE_BASE_ID",
"modelArn": "YOUR_SUPPORTED_MODEL_ARN"
}
}
)

return {
"answer": response.get("output", {}).get("text", ""),
"citations": response.get("citations", [])
}

if __name__ == "__main__":
result = ask_question(
"What are the main steps described in the documentation?"
)

print("Answer:")
print(result["answer"])

print("\nCitations:")
for citation in result["citations"]:
print(citation)
Important: Replace the placeholders with valid values for your AWS environment. The knowledge base must already exist, the selected model must be supported for the operation, and your AWS identity must have the necessary permissions. This example has not been presented as a tested deployment.
For production use, add request validation, error handling, authentication, logging, and appropriate timeout and retry behavior.

4. How Do We Evaluate the Quality of Answers?

Adding RAG does not guarantee that every answer will be correct. Retrieval may miss important passages, and a model may misinterpret or go beyond the evidence it receives.
A useful evaluation process should test both retrieval and generation.

Retrieval quality

Check whether the retrieved documents actually contain the information needed to answer each question.
Useful measurements include:
  • Precision@k: The proportion of the top k retrieved results that are relevant.
  • Recall@k: The proportion of relevant source items found among the top k results.

Answer quality

Evaluate whether the generated answer:
  • Correctly reflects the retrieved evidence.
  • Answers the actual question.
  • Includes appropriate source citations.
  • Avoids introducing unsupported claims.
  • Clearly states when the available evidence is insufficient.
Create a small test set of questions with expected answers and supporting source passages. Include questions whose answers are missing from the documents, as well as questions that require combining information from multiple sections.
Where possible, combine automated evaluation with human review. A fluent answer is not necessarily a correct answer, and a citation is useful only when it genuinely supports the associated claim.

5. Security and Cost Considerations

Before deploying a RAG application, consider the following:
Access control: Apply least-privilege IAM policies and protect source documents from unauthorized access.
Data privacy: Understand how the selected AWS services handle your data, and do not send sensitive information unnecessarily.
Prompt injection: Treat retrieved documents as untrusted input. A document may contain malicious instructions that attempt to influence the model. Design prompts and application controls so retrieved content is treated as evidence rather than trusted instructions.
Cost management: Review the pricing of the services and models you select, including storage, retrieval, model inference, and any associated vector storage or processing.
Monitoring: Record appropriate operational metrics, retrieval failures, latency, and evaluation results without unnecessarily logging sensitive user content.
Failure handling: Decide what the application should do when retrieval returns no useful evidence or the model cannot produce a supported answer.

6. What Are the Limitations of RAG?

RAG improves access to relevant information, but it is not a complete solution to AI hallucinations.
Its effectiveness depends on the quality of the source documents, document processing, retrieval settings, and the model's ability to use the retrieved context.
For example, if the source material is outdated or contradictory, the generated response may also be misleading. Similarly, a relevant passage may be retrieved but interpreted incorrectly.
That is why retrieval, answer generation, citation verification, and evaluation should be treated as separate parts of the system.

Conclusion

Amazon Bedrock Knowledge Bases can simplify the development of document-grounded generative AI applications by providing managed retrieval and answer-generation capabilities.
By combining relevant source documents, appropriate access controls, evidence-based responses, and systematic evaluation, developers can build AI applications that are easier to inspect and improve.
The most important lesson is that trustworthy AI requires more than generating a plausible answer: it requires evidence, evaluation, and a clear understanding of system limitations.

References

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article