Building Context Aware AI Applications with RAG - A Quick Recap
A concise overview of RAG, chunking strategies, re-ranking, autogenerated metadata filters, query reformulation, guardrails (Responsible AI), multimodal workflows, Structured RAG, and GraphRAG
Introduction
Reliable enterprise QA requires minimizing hallucinations through fact grounded responses. RAG achieves this by indexing domain data, intelligent chunking, embeddings, and precise retrieval. Amazon Bedrock Knowledge Bases strengthen this with managed metadata, auto generated filters, evaluations, guardrails, and multimodal, SQL, and graph based retrieval for richer accuracy and context.
Accuracy isn’t an option, it's a requirement
Knowledge Base Setup Workflow (Amazon Bedrock):
The process begins by connecting Python to AWS services such as S3, STS, and Bedrock, followed by retrieving the region and account details required for resource creation. Unique names are then generated to avoid naming conflicts, and an S3 bucket is defined as the primary data source. Next, a Knowledge Base blueprint is created specifying the chunking strategy, data source, and storage configuration after which the documents are uploaded to S3. An ingestion job is started to extract text, chunk the content, generate embeddings, and store them in the vector database. Finally, the Knowledge Base ID is saved, enabling us to query it later using Bedrock’s fully managed
retrieve_and_generate APIs. This setup forms the foundation on which all RAG features and enhancements operate. | Model Used | anthropic.claude-3-sonnet-20240229-v1:0 |
|---|
Knowledge Base Setup Workflow
RAG Basics - The Core Idea
RAG stands for Retrieval Augmented Generation. It builds context aware answers using your own enterprise data.
RAG flow
Documents will be uploaded to S3, ingested into a Bedrock Knowledge Base, chunked, embedded, and indexed in OpenSearch. When a query is submitted, it will be converted into an embedding, the top-k relevant chunks will be retrieved, sent to Claude, and a final grounded answer will be generated.
Key API used :
1
bedrock_runtime.retrieve_and_generate(...)
This API call retrieves the most relevant chunks from a Knowledge Base based on the user query and then automatically passes that context to a foundation model to generate a grounded response. It handles retrieval + generation in one step, giving you an end-to-end RAG workflow with citations.
Chunking Strategies
- Chunking is the process of splitting documents into smaller pieces before embedding them.
- The quality of chunking directly affects retrieval accuracy, latency, and answer quality in RAG systems.
1.Fixed Size Chunking (Simple and Fast)
“Split the document into equal boxes, regardless of meaning.”
Example:
A 10 page document is cut every 500 words, even if a paragraph or topic is in the middle. As a result, a chunk may contain half of “Applications” mixed with half of “Challenges.”
A 10 page document is cut every 500 words, even if a paragraph or topic is in the middle. As a result, a chunk may contain half of “Applications” mixed with half of “Challenges.”
Problem :
Mixed topics get merged into each chunk, which creates noisy retrieval when the model searches for information. This often results in partially correct answers because the chunks are not aligned with clear, coherent topics.
Consider a typical financial report structure:
2. Semantic Chunking (Meaning-Based)
“Split the document only where the meaning naturally changes.”
Semantic chunking might produce:
- Chunk A: Income Statement
- Chunk B: Balance Sheet
- Chunk C: Entire Cash Flow Statement (as one big chunk)
Problem:
It sees “Cash Flow Statement” as one semantic topic, so it does not keep sub sections separate.
This weakens retrieval when a user asks a specific question.
3.Hierarchical Chunking
Hierarchical chunking creates multiple levels of chunks (sections ,subsections , paragraphs) so the model gets both high level context and fine-grained detail. It preserves the parent child relationships in the document, improving retrieval accuracy and reasoning
“Think of your document as a tree , hierarchical chunking keeps every branch and leaf connected.”
When hierarchical chunking is used, the knowledge base stores the document like a tree:
- Root - entire financial report
- Section - Cash Flow Statements
- Branch - Investing Activities
- Leaf - precise paragraphs with numbers or explanations
During retrieval, the query travels down this tree and pulls the most relevant parts from each level.
This layered retrieval is what prevents hallucinations.
This layered retrieval is what prevents hallucinations.
RAGAS Evaluation:
Retrieval Augmented Generation Assessment (RAGAS) is a systematic evaluation framework designed to measure how accurately a RAG system retrieves information and how faithfully the model generates answers based on that information. The core strength of RAGAS lies in its ability to compare the model’s final answer against two key references: the retrieved context and the ground truth answer. This dual comparison allows RAGAS to determine whether errors arise due to poor retrieval (missing or irrelevant chunks) or due to faulty generation (hallucination, incorrect interpretation, or unsupported statements).
Example Evaluation Scores :
Below is a sample evaluation of a single question using RAGAS metrics.
Ground Truth :
“Operating lease liabilities were $10M in 2022.”
Model Answer:
“The company had $11M in operating lease liabilities in 2022.”
| Metric | Description | Reason for Score | Sample Score |
|---|---|---|---|
| Faithfulness | Whether answer is supported by retrieved context | Answer states $11M instead of $10M unsupported claim | 0.20 |
| Semantic Similarity | Meaning similarity between model answer & ground truth | Meaning is close but number is wrong | 0.60 |
| Answer Correctness | Factual accuracy | Wrong numerical value ,largely incorrect | 0.30 |
| Answer Relevance | Does the answer address the question? | Talks about lease liability as asked | 0.95 |
| Context Precision | Were the right chunks retrieved? | Correct chunk retrieved | 0.90 |
| Context Recall | Did retrieval return all needed info? | All needed info present | 1.00 |
| Entity Recall | Were key entities retrieved? (2022, $10M, liability) | Entities exist in context | 1.00 |
| Harmfulness | Safety compliance | No harmful content | 0.00 (good) |
RAGAS gives a measurable score that shows how well retrieval and generation worked. It helps identify errors and ensures the RAG system stays accurate and reliable.
Autogenerated Filters
Autogenerated Filters in Amazon Bedrock automatically extract key metadata such as dates, departments, products, or regions from a user’s query and apply the appropriate filters without manual input. This enhances search accuracy and ensures results are more relevant and context aware, even when users don’t specify filters.
User asks:
“Give me the 2022 financial statements for the APAC region.”
Bedrock will Identify relevant metadata fields:
| Query phrase | Metadata field |
|---|---|
| 2022 | year |
| APAC | region |
Enabling Autogenerated Filters feature:
1
2
3
4
5
6
7
8
retrievalConfiguration = {
"vectorSearchConfiguration": {
"numberOfResults": 5,
"implicitFilterConfiguration": {
"enableImplicitFiltering": True
}
}
}
Generates automatic filters:
1
2
3
4
{
"year": 2022,
"region": "APAC"
}
Semantic search happens only on filtered subset and Return top results with much higher relevance. This drastically improves retrieval accuracy.
Query Reformulation
Query Reformulation is used when a user asks a complex questions containing multiple topics a single embedding may ignore or dilute parts of the query. Bedrock automatically breaks the query into smaller sub-queries so each part retrieves the correct context. This ensures more accurate retrieval and a complete answer from the FM.
1
2
3
4
5
'orchestrationConfiguration': {
'queryTransformationConfiguration': {
'type': 'QUERY_DECOMPOSITION'
}
}
User Query:
“Where is the company headquarters, and what were last year’s total sales?”
“Where is the company headquarters, and what were last year’s total sales?”
Bedrock Automatically Splits It Into:
1
2
3
4
5
6
<query>
Where is the company headquarters?
</query>
<query>
What were the total sales last year?
</query>
Each sub query focuses on a single topic, allowing retrieval to pull the most relevant chunks for each part of the question. By combining these targeted chunks, the model produces a more accurate, complete, and well grounded final answer.
Re-Ranking:
Re-ranking in Amazon Bedrock improves retrieval by scoring chunks using a deeper query chunk comparison instead of relying only on vector similarity. Vector similarity (cosine distance) shows how “semantically close” a chunk is, but it doesn’t guarantee the chunk actually answers the query. A reranker reads the query and the chunk together and assigns a true relevance score(0-1), pushing the most useful chunks to the top. This gives the model cleaner context, fewer irrelevant chunks, and more accurate answers at lower cost.
Bedrock Reranker Models available:
| Model | Description |
|---|---|
| amazon.rerank-v1:0 | High quality, optimized for AWS workloads |
| cohere.rerank-v3-5:0 | Excellent accuracy, widely used |
To Enable Re-Ranking:
1
2
3
4
5
6
7
8
"rerankingConfiguration": {
"type": "BEDROCK_RERANKING_MODEL",
"bedrockRerankingConfiguration": {
"modelConfiguration": {
"modelArn": f"arn:aws:bedrock:{region}::foundation-model/amazon.rerank-v1:0"
}
}
}
Responsible AI: Guardrails & Contextual Grounding
Guardrails in Amazon Bedrock are safety and quality controls that prevent an AI model from producing unsafe, irrelevant, hallucinated, or ungrounded responses.
They ensure that the model’s answers:
They ensure that the model’s answers:
- Stay factually accurate based on your Knowledge Base
- Stay relevant to the user’s question
- Avoid personal data, harmful content, or made-up information
- Follow Responsible AI guidelines
Contextual Grounding:
Contextual Grounding checks AI output using two scores:
| Score | Meaning | If Score is Low |
|---|---|---|
| Grounding Score | Measures whether the answer is factually supported by the Knowledge Base | Blocked (hallucination detected) |
| Relevance Score | Measures whether the answer actually relates to the user’s question | Blocked (irrelevant answer) |
1
2
3
4
'filtersConfig': [
{ 'type': 'GROUNDING', 'threshold': 0.5 },
{ 'type': 'RELEVANCE', 'threshold': 0.8 }
]
- If grounding < 0.5 - BLOCK
- If relevance < 0.8 - BLOCK
User Query:
"Explain Amazon’s hiring process, and also tell me the private contact details of the HR manager."
With Guardrails Enabled:
The model detects that the second part of the question requests personal information, which is not allowed.
The model will reply:
“Sorry, I cannot respond to this.”
Multimodal RAG with Amazon Bedrock Data Automation (BDA)
LLMs cannot directly understand raw video files (.mp4, .mov) or audio files (.mp3, .wav) because embeddings only work on text, not pixels or sound waves.
This is where Amazon Bedrock Data Automation (BDA) becomes essential. BDA converts raw multimodal data to machine readable text & metadata.
The Knowledge Base then ingests this text using the new BDA parser, creates embeddings, and allows users to query audio/video content just like they query documents. This unlocks powerful multimodal search and grounded generation across podcasts, tutorials, meetings, and videos.
This is where Amazon Bedrock Data Automation (BDA) becomes essential. BDA converts raw multimodal data to machine readable text & metadata.
The Knowledge Base then ingests this text using the new BDA parser, creates embeddings, and allows users to query audio/video content just like they query documents. This unlocks powerful multimodal search and grounded generation across podcasts, tutorials, meetings, and videos.
Processing Audio & Video for Multimodal RAG Using Amazon BDA.
Structured RAG :
Structured RAG is designed for data stored in databases organized into tables, rows, and columns. Unlike unstructured RAG, no embedding or chunking is needed because the information is already structured. Instead, Amazon Bedrock interprets a natural language question, converts it into an appropriate SQL query using the database schema, executes it directly on Redshift, retrieves the results, and summarizes them back into natural language. This allows users to analyze their structured data conversationally, without writing SQL or understanding database internals, while keeping all data securely inside Redshift.
Structured RAG Workflow
Graph RAG Using Amazon Bedrock Knowledge Bases:
Graph RAG in Amazon Bedrock enhances traditional RAG by not only retrieving text based on similarity but also understanding how different facts, sections, and entities in your documents are connected. When documents are ingested, Bedrock automatically builds a knowledge graph in Amazon Neptune capturing relationships such as “belongs to”, “part of section”, “related to” or “describes”.
This graph representation allows the system to retrieve all relevant, connected information, not just isolated chunks that look semantically similar.
As a result, the model can reason across multiple document sections, combine related facts, and produce answers that are more complete, contextual, and explainable.
Graph RAG
Creating a Knowledge Base with Graph Mode:
1
2
3
4
5
6
7
knowledge_base = BedrockKnowledgeBase(
kb_name=knowledge_base_name,
kb_description=knowledge_base_description,
data_sources=data_sources,
chunking_strategy = "GRAPH",
vector_store="NEPTUNE_ANALYTICS"
)
Using
chunking_strategy="GRAPH" and vector_store="NEPTUNE_ANALYTICS", the Knowledge Base automatically extracts entities, sections, and relationships from documents. This creates a graph-enhanced embedding store that lets the system understand how facts connect.Conclusion :
Amazon Bedrock Knowledge Bases significantly strengthen RAG by improving how information is retrieved, filtered, and grounded in trustworthy data. With features like autogenerated filters, query reformulation, and reranking, the system reliably selects the most relevant context for a user’s query. Guardrails add an essential safety layer by blocking hallucinations and off topic responses, while BDA enables multimodal understanding of audio and video content. Structured RAG brings natural language analytics to databases, and GraphRAG introduces relationship-based reasoning for deeper, more connected answers. Together, these capabilities transform RAG into a highly accurate, explainable, and production ready AI system.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article