Bedrock Knowledge Bases Looked Perfect in My Demo. Production Had Other Plans

I’ll be honest with you. The first time I set up an Amazon Bedrock Knowledge Base, I thought I had cracked the code. Documents in S3, a few clicks in the console, a managed vector store spinning up behind the scenes, and suddenly my prototype was answering domain-specific questions with impressive accuracy. I showed it to stakeholders. They loved it. We moved fast toward production.

That’s where things got interesting.

This article is the first in a series I’m calling Golden Jacket Field Notes. Lessons from the field that I wish someone had written down for me when I was deep in the trenches. Not a tutorial. Not a re:Invent recap. Just what actually happens when you take AWS services past the demo stage and into real workloads, real users, and real consequences.

Let’s talk about Bedrock Knowledge Bases. The good, the painful, and the stuff the documentation won’t warn you about.

Amazon Bedrock Knowledge Bases is a fully managed Retrieval Augmented Generation (RAG) pipeline. You point it at a data source, usually an S3 bucket full of documents, and it handles the heavy lifting: parsing, chunking your content into smaller pieces, generating vector embeddings, and storing them in a vector store so they can be retrieved at query time.

The promise is compelling. You get RAG without managing infrastructure for embedding generation, vector databases, or retrieval orchestration. For teams that want to ship fast, this is powerful. You can go from a pile of PDFs to a working Q&A system in an afternoon.

But here’s the nuance that matters: Bedrock Knowledge Bases is an abstraction layer, and every abstraction trades control for convenience. When your use case fits neatly inside the guardrails of the managed service, you’ll move faster than anyone building a custom pipeline. When it doesn’t, you’ll spend more time fighting the abstraction than you saved by using it.

Knowing where that line is. That’s the skill.

— -

This is the first thing that caught me off guard, and it still catches teams today. When you configure a data source in Bedrock Knowledge Bases, you choose a chunking strategy. Fixed-size, hierarchical, semantic, or no chunking at all. Reasonable options.

What the console doesn’t emphasize enough is that you cannot change the chunking strategy after the data source is created. This isn’t a tunable parameter you iterate on. It’s a one-way door. If you realize three months in that your 300-token fixed-size chunks are splitting critical context across boundaries and degrading retrieval quality, your path forward is to create a new data source from scratch, re-ingest everything, and swap it in.

In practice, this means the chunking decision deserves more time than most teams give it. Here’s what I’ve learned works.

Fixed-size chunking is the safe default. It’s predictable, simple to reason about, and performs well for homogeneous document types. But if your corpus includes documents with strong structural hierarchy , think technical manuals, legal contracts, API documentation, fixed-size chunks will often slice through the middle of a section in ways that destroy meaning.

Hierarchical chunking creates parent and child chunks at two depth levels. The parent preserves broader context while the child offers precision. This sounds ideal for structured documents, and conceptually it is. But there’s a catch: hierarchical chunking stores parent-child relationships as metadata on each vector. If you’re using S3 Vectors as your vector store, AWS explicitly recommends against hierarchical chunking because you’ll run into metadata size limitations when your token counts get high. That’s a painful limitation to discover after you’ve already committed to both the chunking strategy and the vector store.

Semantic chunking uses natural language processing to break text at meaning boundaries rather than arbitrary token counts. It tends to produce cleaner, more coherent chunks. The trade-off is cost…semantic chunking calls a foundation model during ingestion, which adds latency and spend to your data pipeline. For large corpora that change frequently, this cost compounds.

My recommendation: if you’re uncertain, start with fixed-size at 500–800 tokens with a 10–15% overlap. It’s forgiving enough for most document types and gives you a reasonable baseline to evaluate retrieval quality before committing to something more complex.

— -

Bedrock Knowledge Bases supports several vector store options: Amazon OpenSearch Serverless, Aurora PostgreSQL with pgvector, Pinecone, Redis Enterprise Cloud, and…since re:Invent 2025: Amazon S3 Vectors!!

S3 Vectors is the newest addition and the one generating the most conversation right now. AWS positions it as a cost-optimized alternative that can deliver up to 90% savings compared to traditional vector databases, with sub-second query latency at scale. It went generally available in late 2025 with support across 14 AWS Regions, and it handles very large embedding sets without you managing shards or rebalancing clusters.

Sounds like a no-brainer, right? Not quite.

S3 Vectors currently supports semantic search only. If your use case requires hybrid search , combining vector similarity with keyword matching or BM25 scoring , S3 Vectors won’t cover it. You’ll need OpenSearch for that. There’s also a metadata ceiling: each vector can carry up to 1KB of custom metadata with a maximum of 35 filterable keys. For simple document retrieval, that’s usually plenty. For applications that rely on rich metadata filtering to narrow results before the vector comparison step, it can feel tight.

Here’s how I think about the decision. If cost is a primary concern and your retrieval pattern is straightforward semantic similarity, S3 Vectors is a strong choice. If you need hybrid search, complex filtering, or you’re building a retrieval system that goes beyond simple Q&A , think multi-step agent workflows where the retrieval logic itself is nuanced , OpenSearch Serverless gives you significantly more flexibility, at a higher cost.

The mistake I see most often is teams defaulting to the cheapest option without mapping their actual retrieval patterns first. The vector store isn’t just storage. It’s the engine behind every query your application runs. Choose based on what your queries need, not what your ingestion pipeline costs.

— -

This is the section I wish existed when I started.

Debugging retrieval quality is harder than it should be. Bedrock Knowledge Bases gives you CloudWatch metrics for invocation latency and basic error rates. What it doesn’t give you is LLM-specific observability. You can’t easily trace a bad answer back through the retrieval chain to see which chunks were returned, why they ranked the way they did, or whether the generation model hallucinated despite having correct context. If you try to debug a hallucination using CloudWatch alone, you’re going to have a frustrating time. Build observability into your application layer from day one , log the retrieved chunks, their relevance scores, and the generation prompt before you need to troubleshoot.

The “managed” part has limits. Bedrock handles ingestion, embedding, and retrieval, but it won’t tell you when your retrieval quality is degrading. Documents change. New content gets added. Embedding drift is real. You need your own evaluation framework , a set of known queries with expected answers , that you run periodically against your Knowledge Base. Treat it like a test suite for your data.

Query reformulation is underused. Bedrock Knowledge Bases supports automatic query reformulation, where a complex multi-part question gets decomposed into simpler sub-queries before retrieval. This feature dramatically improves accuracy for compound questions, but I rarely see teams enable it. It’s not on by default. Turn it on, especially if your users ask anything more complex than single-topic lookups.

Multimodal ingestion is real but young. Post re:Invent 2025, Knowledge Bases can ingest and retrieve across text, images, audio, and video. The capability exists, but the tooling around evaluating multimodal retrieval quality is still catching up. If you’re building a multimodal RAG application, expect to invest in custom evaluation more heavily than you would for text-only.

Re-ingestion at scale requires planning. If you need to change embedding models, adjust chunking, or reorganize your data source, you’re re-ingesting everything. For large corpora, this isn’t a five-minute operation. Build your pipeline so that re-ingestion is a button press, not a project. Automate it with AWS Step Functions (SFN) or Amazon EventBridge so that when the time comes , and it will , you’re ready.

— -

I get this question a lot, and my answer has evolved over time. Here’s where I’ve landed.

Use Bedrock Knowledge Bases when your RAG requirements are well-served by the managed chunking options, you don’t need exotic retrieval strategies, and your team’s priority is shipping quickly with minimal infrastructure overhead. For internal knowledge assistants, customer support bots, and document Q&A systems where the corpus is reasonably well-structured, it’s an excellent fit. You’ll ship weeks or months faster than a custom pipeline.

Build your own when you need fine-grained control over any part of the RAG chain , custom chunking logic that doesn’t fit the four built-in strategies, hybrid retrieval with reranking, chain-of-thought retrieval where the agent decides how to search, or when you need to swap components independently as your requirements evolve. Teams at this stage often end up using OpenSearch directly with LangChain or LlamaIndex for orchestration, running on infrastructure that gives them full control.

Most teams I work with start with Bedrock Knowledge Bases , and that’s the right call. The key is recognizing the signals that you’re outgrowing it: increasing complaints about retrieval quality that you can’t fix by tuning prompts, retrieval patterns that require hybrid search or custom reranking, or a need to change chunking strategies without rebuilding from scratch.

When those signals appear, you haven’t failed. You’ve graduated.

— -

Bedrock Knowledge Bases is one of the fastest paths from zero to working RAG on AWS. The managed pipeline genuinely removes infrastructure complexity, and for a wide range of use cases, it’s the right answer.

But production is a different game than a demo. The chunking decision is permanent. The vector store choice shapes your retrieval capabilities. Observability doesn’t come for free. And the managed service, by design, trades control for convenience.

The teams that succeed with Bedrock Knowledge Bases in production are the ones that invest time upfront in the decisions that can’t be changed later, build their own evaluation framework to catch quality regressions, and know , before they start , what their exit ramp looks like if they outgrow the abstraction.

That’s not a criticism of the service. That’s just what it means to run things in production.

— -