
Your RAG Has the Right Document - but the Wrong Pieces
Why chunking can quietly weaken retrieval in Amazon Bedrock Knowledge Bases.
When a RAG application gives a weak answer, the model usually gets the blame.
But sometimes the model never received the right information. The answer was hidden inside a document, and that document was divided in a way that made the information difficult to find.
That division process is called chunking. It may sound like a small setup choice, but it has a large effect on what your RAG application can retrieve.
Why documents are divided
Imagine uploading a 100-page policy manual to a knowledge base. When someone asks a question, sending the entire manual to the model would be slow, expensive and unnecessary.
Instead, the document is divided into smaller sections called chunks. Each chunk is converted into an embedding and stored in a vector database.
When a question arrives, the system searches for the chunks that appear most relevant and sends only those pieces to the model.
The basic flow is:
Document → chunks → embeddings → vector store → relevant chunks → AI answer
The quality of the answer now depends partly on where those chunks begin and end.
What happens when chunks are too large?
A large chunk may contain several different topics.
For example, one chunk might contain information about account creation, password resets, billing and account deletion. When all of that text is represented by a single embedding, its meaning becomes less focused.
A large chunk can also send unnecessary text to the model. That increases token usage and may distract the model from the part that actually answers the question.
Large chunks provide more surrounding context, but they can make retrieval less precise.
What happens when chunks are too small?
Small chunks create the opposite problem.
A sentence may be retrieved without the heading that explains what it refers to. A policy rule may appear without its exception. A troubleshooting step may be returned without the error message or product name connected to it.
Small chunks are easier to match precisely, but they can lose the context needed to understand the information.
This is the main chunking trade-off:
- Larger chunks provide more context but may contain extra information.
- Smaller chunks are more focused but may lose important context.
There is no perfect chunk size for every knowledge base.
Why overlap helps
A fact may appear near the boundary between two chunks. If the document is split at exactly that point, part of the meaning could land in one chunk and the rest in another.
Overlap repeats a small amount of text between neighbouring chunks. This gives the retriever another chance to capture the complete idea.
Too little overlap can break related information apart. Too much overlap creates repeated content, increases storage and may return several nearly identical results.
The aim is not maximum overlap. It is enough overlap to preserve meaning across chunk boundaries.
Chunking options in Amazon Bedrock
Amazon Bedrock Knowledge Bases supports several chunking approaches, depending on the type of knowledge base and ingestion configuration. AWS chunking documentation
Fixed-size chunking
Fixed-size chunking divides text into pieces containing roughly the same number of tokens. You choose the maximum chunk size and the overlap between consecutive chunks.
It is predictable, simple and a good starting point for documents with a consistent structure. However, it may split text in an awkward place because it follows size rather than meaning.
Semantic chunking
Semantic chunking looks at the meaning of the text. It attempts to keep related sentences together and creates a new chunk when the topic changes.
This can produce more natural chunks for articles, FAQs and documents containing clear topic changes. It may improve retrieval quality, but it requires additional processing and should still be evaluated against your own documents.
Hierarchical chunking
Hierarchical chunking creates two levels:
- Small child chunks for accurate searching
- Larger parent chunks for additional context
The system searches using the smaller chunks. When it finds a relevant child, it can return the broader parent section to the model.
This offers a useful balance: search with a small target, then answer with more context.
No chunking
With no chunking, each file is treated as one chunk.
This can work when every file is already short and focused. It is usually a poor choice for large documents containing many subjects. AWS recommends preprocessing documents into separate files when using this option. Amazon Bedrock ingestion customization
Chunking should follow the document
A support article, legal contract and product manual do not have the same structure. They should not automatically use the same chunking settings.
Before choosing a strategy, look at:
- The average document length
- How headings and sections are organized
- Whether answers depend on nearby paragraphs
- Whether tables or images contain important information
- The kinds of questions users will ask
- How much context the model needs to answer correctly
The best question is not simply, “What chunk size should I use?”
A better question is, “What information must stay together for this question to be answered?”
Test before committing
Chunking is selected while configuring a data source, and AWS notes that the strategy cannot simply be changed after the data source is connected. A different strategy may require creating and ingesting the data source again.
That makes testing important.
Prepare a small set of realistic questions with the documents or sections you expect to retrieve. Try different chunk sizes, overlaps and strategies. Then inspect the actual chunks returned not only the final AI answer.
Check whether:
- The expected information was retrieved
- The chunk includes enough surrounding context
- Unrelated text was included
- Several results repeat the same information
- Important headings, exceptions or table content were lost
The strategy that works best on your own questions is more useful than a setting copied from another project.
Final takeaway
Chunking is not simply cutting a document into equal pieces. It is deciding which information should stay together.
If chunks are too large, retrieval becomes broad and noisy. If they are too small, important context disappears. Fixed-size chunking provides simplicity, semantic chunking follows meaning, and hierarchical chunking combines precise retrieval with broader context.
A foundation model cannot recover context that was removed before retrieval. Get the chunks right first, and the rest of the RAG pipeline has a much better chance of working.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article