Building a Document Q&A App with Amazon Bedrock Knowledge Bases
Load files from S3, create and sync a knowledge base, query it from Python with answers and citations, and debug retrieval when results look wrong.
Series: Amazon Bedrock (13 articles)
- …
- 6Building a Document Q&A App with Amazon Bedrock Knowledge Bases This article
Point a managed knowledge base at your own documents in S3, sync it, and ask questions that come back with sources.
A model on its own knows nothing about your policies, your product manuals or your internal notes. Ask it about them and it will either say it doesn't know or, worse, sound confident while making something up.
Amazon Bedrock Knowledge Bases fixes that with retrieval-augmented generation, usually shortened to RAG. It finds the passages in your documents that are most relevant to a question, hands them to a model, and returns an answer with references to where it came from. In this article we'll build a small document Q&A app from an S3 bucket and query it from Python.
How it fits together
Here's the flow in plain terms:
- Your documents sit in an Amazon S3 bucket.
- Ingestion splits them into chunks and turns each chunk into an embedding, the numbers-for-meaning idea from the embeddings article on Medium.
- A vector store keeps those embeddings so they can be searched by similarity.
- At question time, your question is embedded too, the closest chunks are retrieved, and a model writes an answer from them.
- The answer comes back with citations that point to the source documents.
What you need
- The setup from the earlier articles: Python, boto3 and working Bedrock credentials
- A small set of documents you're allowed to use, such as PDFs, text files or Word files. Start with five to ten.
- A model you can invoke for answering, and access to an embedding model
- About an hour
A note on cost before anything else. A knowledge base involves several billable pieces: embedding the documents, storing vectors, and generating answers. Some vector store options also carry a running charge while they exist. Compare the current pricing for Amazon Bedrock, Amazon S3 and your chosen vector store before you begin, and plan to delete what you create when you're done experimenting.
Step 1: Put your documents in S3
- Create an S3 bucket, or use an existing one, in the same Region where you'll use Bedrock.
- Upload your documents under a folder-style prefix, such as
docs/.
- Keep the first set small and clean. Poorly formatted files are the most common reason for disappointing answers.
Don't put anything confidential in a test bucket unless you're sure about who has access.
Step 2: Create the knowledge base
- In the Amazon Bedrock console, open Knowledge bases and choose Create, selecting the option for a knowledge base with a vector store.
- Name it and let the console create a service role. That role lets Bedrock read your bucket and use the embedding model and vector store.
- Choose Amazon S3 as the data source and point it at your bucket and prefix.
- Pick an embedding model.
- Choose a vector store. The console can quick-create one for you. The documentation lists Amazon OpenSearch Serverless, Amazon Aurora PostgreSQL Serverless and Amazon S3 Vectors as options. For a first experiment, one that doesn't require you to run a separate cluster is the simplest, but compare how they're priced and what each is suited to.
- Review the settings and create it.
Write down the knowledge base ID. You'll need it in your code.
Step 3: Sync the data source
Creating the knowledge base doesn't load your documents. You have to sync.
In the console, open your knowledge base, select the data source and choose Sync. Wait until the status shows it's complete. The sync reads your files, chunks them and creates the embeddings.
Remember this step: whenever you add, change or remove files in the bucket, sync again, or your answers will be based on old content.
You can also trigger it from code with the
bedrock-agent client, using the start_ingestion_job call with the knowledge base ID and the data source ID.Step 4: Ask a question from Python
Querying uses the
bedrock-agent-runtime client, the same one we used for flows. The simplest call is retrieve_and_generate , which finds the relevant passages and writes the answer in one go.Python
import boto3REGION = "us-east-1" # use your RegionKB_ID = "YOUR_KNOWLEDGE_BASE_ID"MODEL_ARN = "YOUR_MODEL_OR_INFERENCE_PROFILE_ARN" # copy from the model's cardclient = boto3.client("bedrock-agent-runtime", region_name=REGION)response = client.retrieve_and_generate(input={"text": "What is our refund policy for annual plans?"},retrieveAndGenerateConfiguration={"type": "KNOWLEDGE_BASE","knowledgeBaseConfiguration": {"knowledgeBaseId": KB_ID,"modelArn": MODEL_ARN,},},)print(response["output"]["text"])for citation in response.get("citations", []):for ref in citation.get("retrievedReferences", []):print("Source:", ref.get("location", {}))
The answer is in
output.text , and the citations tell you which passages and files it drew on. Show those to your users. It's a big part of why people trust a RAG app.Note that
modelArn takes the full ARN of a model or an inference profile, not just the short model ID. Copy it from the documentation for the model you chose.Step 5: See what was retrieved
When an answer looks wrong, the first question is whether the right passages were found. The
retrieve call gives you just the retrieved chunks, with no model-written answer, so you can inspect them.Python
response = client.retrieve(knowledgeBaseId=KB_ID,retrievalQuery={"text": "What is our refund policy for annual plans?"},retrievalConfiguration={"vectorSearchConfiguration": {"numberOfResults": 5}},)for result in response["retrievalResults"]:print(round(result.get("score", 0), 2), result["content"]["text"][:150])
If the right passage isn't in the top results, the problem is in retrieval: the document wasn't ingested, the question is worded very differently from the text, or the chunks are too big or too small. If the right passage is there but the answer is still wrong, the problem is in generation, and you should look at the prompt and model.
That split between "did it find the right text" and "did it use it well" is the single most useful habit when debugging a RAG app.
Step 6: Test it properly
Write ten questions that you already know the answers to, covering easy lookups, questions that need two documents, and a couple the documents don't answer at all. Run them and check:
- Is the answer correct?
- Does the citation point to the right file?
- For questions the documents can't answer, does it say so rather than inventing something?
That last one matters most. Keep the list and rerun it after every change to the documents or settings.
My results
[Add your own results here once you've run it: how many documents you used, how many of your ten test questions it answered correctly, anything that surprised you, and roughly what it cost. Real numbers make this section the most useful part of the article.]
When it doesn't work
Answers say nothing was found. Check that the sync finished, that the files are in a supported format and in the prefix you configured, and that you're querying the right knowledge base.
An access error. Check that your identity can query the knowledge base and invoke the model, and that the knowledge base's service role can read the bucket and use the embedding model.
Old information keeps appearing. You changed the files but didn't sync again.
Answers are vague or miss detail. Try more specific questions, check the chunks with
retrieve , and look at the chunking options in the data source settings.The model ARN is rejected. Use the full ARN, and make sure the model is available in your Region.
Tidy up
When you're finished experimenting, delete the knowledge base, the vector store it created and any test files you don't need, so you don't keep paying for storage or a running vector store. Check each of those resources, since deleting the knowledge base alone may not remove the vector store.
What you have now
A managed document Q&A app: your files in S3, a synced knowledge base, and a few lines of Python that return answers with sources, plus a way to see what was retrieved when something goes wrong.
In the next article we'll look at how to keep an AI app affordable, with budgets, alerts and the habits that keep a bill predictable.
Features, vector store options, supported file types, quotas and pricing change often. Check the current Amazon Bedrock documentation before you follow any step here.
Series: Amazon Bedrock (13 articles)
- …
- 6Building a Document Q&A App with Amazon Bedrock Knowledge Bases This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article