What Actually Happens Inside AWS When You Query a Bedrock Knowledge Base
Managed RAG is still a pipeline. Here is each stage of a Bedrock Knowledge Base, from the ingestion job to the prompt template, and how to inspect every one of them from the CLI.
This is the third one in the series. The first took apart what Nitro does before an EC2 instance boots, the second followed a Bedrock model invocation down through its layers. This one goes after the thing that markets itself hardest as "managed": Knowledge Bases, the RAG pipeline you get without writing a retriever. It is still a pipeline, and every stage of it is a decision you are making, even when you accept the default. Everything below comes from the AWS documentation, and the commands run against your own account.
Ingestion is a job, and you can watch it
The first thing that clears up the mental model: syncing a data source is not a background daemon, it is a job with an ID.
1
2
3
aws bedrock-agent start-ingestion-job \
--knowledge-base-id ABCDEFGHIJ \
--data-source-id KLMNOPQRSTBoth IDs are ten alphanumeric characters. The job reads your data source, chunks it, embeds each chunk, and writes the vectors into your vector store. Nothing is automatic about it. If you drop a new PDF into the bucket and never start a job, the knowledge base does not know it exists.
Chunking is a choice you make even when you do not make it
Take the default. The docs are specific: default chunking "splits content into text chunks of approximately 300 tokens. The chunking process honors sentence boundaries, ensuring that complete sentences are preserved within each chunk."
Three hundred tokens. That is roughly a long paragraph. If your source material answers questions across two pages, the default will cut those answers apart, and no amount of prompt tuning downstream fixes it.
The alternatives are worth knowing before you pick one:
- Fixed-size, where you set a token maximum and an overlap percentage.
- No chunking, where "each document is treated a single text chunk."
- Hierarchical, which "involves organizing information into nested structures of child and parent chunks." Retrieval hits the child, then swaps in the parent for context.
- Semantic, which splits on meaning using a buffer size and a breakpoint percentile threshold for sentence dissimilarity.
Hierarchical has a consequence that surprised me and is documented plainly: since child chunks sharing a parent get collapsed into that parent, "the number of results returned might be less than the requested amount." Your
numberOfResults is a ceiling on child chunks, not a promise about what comes back.Your vector store is holding your text, not just your vectors
This is the part I would go look at first if I were adopting this.
When you bring your own vector store, Bedrock asks you to name three things: a vector field, a text field, and a metadata field. On OpenSearch managed clusters the docs give the conventional names outright,
AMAZON_BEDROCK_TEXT_CHUNK and AMAZON_BEDROCK_METADATA. On Aurora it is columns: id, embedding, chunks, metadata, and an optional custom_metadata.So the raw chunk text lives in the index next to the embedding. Retrieval does not go back to S3 to fetch the passage; it reads it out of the vector store. That changes how you think about the store: it is not an index pointing at your documents, it is a copy of the relevant pieces of them, with everything that implies for access control and deletion.
The supported stores as documented: OpenSearch Serverless, OpenSearch managed clusters, S3 Vectors, Aurora, Neptune Analytics for GraphRAG, Pinecone, Redis Enterprise Cloud, and MongoDB Atlas.
Retrieval, and the filter you are probably missing
By default "Amazon Bedrock returns up to five results in the response," each one a source chunk. You can query it directly, without any generation on top:
1
2
3
4
5
6
7
8
9
10
aws bedrock-agent-runtime retrieve \
--knowledge-base-id ABCDEFGHIJ \
--retrieval-query '{"text": "what is our refund window"}' \
--retrieval-configuration '{
"vectorSearchConfiguration": {
"numberOfResults": 5,
"overrideSearchType": "HYBRID",
"filter": {"equals": {"key": "genre", "value": "entertainment"}}
}
}'Two things in there deserve attention.
overrideSearchType accepts SEMANTIC or HYBRID, and hybrid "combines searching vector embeddings (semantic search) with searching through the raw text", but per the docs it only works on RDS, OpenSearch Serverless and MongoDB stores with a filterable text field, and silently falls back to semantic otherwise. Read that sentence twice before you assume you have hybrid search.The
filter is the underused one. Those keys come from metadata you supply yourself, as a sidecar JSON file next to the document, holding a metadataAttributes object. For a CSV, the docs name the file fileName.csv.metadata.json. Without it, every query searches your whole corpus.The prompt is a template with holes in it
The last stage is the one people assume is a black box. It is not.
RetrieveAndGenerate builds the generation prompt from a template you can replace, and the template has named placeholders: $query$ is the user query, $search_results$ is the retrieved chunks, $current_time$ is the clock, and $output_format_instructions$ carries the citation formatting. That last one has a rule attached: it "is a required field for citations to be displayed in the response." Drop it from a custom template and your citations quietly disappear.There is also query decomposition, which the docs describe as breaking a complex query into smaller sub-queries, and which may run several queries against the knowledge base for one user question. Useful, and worth knowing about before you debug a latency spike.
What I take from this
A managed RAG pipeline is five decisions: how you cut the text, which model embeds it, where the vectors and the text land, how you filter and search them, and what template wraps the results. The defaults are reasonable. They are also decisions someone made for a corpus that is not yours.
Where to read more
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article