
RAG with OpenAI GPT on Amazon Bedrock
You have a Bedrock Knowledge Base full of documents and you want to build a question-answering system that grounds its responses in that corpus. Bedrock's built-in `RetrieveAndGenerate` API couples retrieval and generation into a single call, but you want more control: custom prompting, citation formatting, deduplication logic, or the ability to swap generation models independently of the retrieval layer.
Retrieval-Augmented Generation with OpenAI Models on Amazon Bedrock
Information | Details |
|---|---|
Pattern | Retrieve-then-Generate (two-step RAG) |
Retrieval engine | Amazon Bedrock Knowledge Bases |
LLM provider | openai[bedrock] (GPT models via Amazon Bedrock Mantle) |
LLM model | openai.gpt-5.6-terra (or -sol / -luna ) |
SDK | OpenAI Python SDK with BedrockOpenAI client |
Components | Bedrock Knowledge Bases, Bedrock Mantle, OpenAI Responses API |
Example complexity | Low |
1. Problem and Solution
The problem
Disclaimer: The example corpus used in this guide is for demonstration purposes only — not for actual scientific research. The articles were selected because they are available under permissive copyright terms and may not represent the current state of Opsanus tau research.
You have a Bedrock Knowledge Base full of documents and you want to build a question-answering system that grounds its responses in that corpus. Bedrock's built-in
RetrieveAndGenerate API couples retrieval and generation into a single call, but you want more control: custom prompting, citation formatting, deduplication logic, or the ability to swap generation models independently of the retrieval layer.The solution
A two-step Retrieve-then-Generate pattern that separates concerns:
- Retrieve — Call the Bedrock Knowledge Bases
retrieveAPI directly to get ranked chunks with source locations
- Generate — Pass those chunks as numbered context to an OpenAI model (GPT-5.6) via Bedrock Mantle, with instructions to cite sources inline
This gives you full control over context assembly, citation formatting, and generation parameters while still leveraging Bedrock's managed vector search and OpenAI's frontier models — all authenticated through AWS credentials with no separate API key.
Who benefits
- Teams with existing Bedrock Knowledge Bases who want to use OpenAI models for generation
- Builders who need structured citations — inline
[n]markers that map back to source documents
- Developers who want retrieval/generation decoupled — swap models, tune retrieval
k, add reranking, or inject business logic between steps
2. Prerequisites
- Python 3.10+
- AWS account with Bedrock access (OpenAI models enabled in a supported region)
- AWS credentials configured locally (no OpenAI API key needed —
openai[bedrock]uses AWS auth)
- An existing Bedrock Knowledge Base with documents already ingested
- uv (recommended): docs.astral.sh/uv
Create a Python virtual environment
uv venvsource .venv/bin/activateInstall dependencies
uv pip install 'openai[bedrock]' boto3The
openai[bedrock] package installs the OpenAI SDK with the Bedrock authentication provider. No OPENAI_API_KEY is needed — the SDK uses your local AWS credentials (via STS) to mint short-lived bearer tokens automatically.Set environment variables
export AWS_REGION="us-east-1"
3. Architecture
┌──────────────┐ query ┌───────────────────────────┐│ Client │ ─────────► │ retrieve-and-generate.py ││ (CLI/app) │ ◄───────── │ │└──────────────┘ answer └───────────────────────────┘ │ │ Step 1 │ │ Step 2 (retrieve) │ │ (generate) ▼ ▼ ┌──────────────────┐ ┌──────────────────┐ │ Bedrock │ │ GPT-5.6 via │ │ Knowledge Bases │ │ Bedrock Mantle │ │ (vector search) │ │ (OpenAI SDK) │ └──────────────────┘ └──────────────────┘ │ │ ▼ │ ┌──────────────────┐ │ │ Your documents │ │ │ (S3, Web, etc.) │ │ └──────────────────┘ │ ▼ ┌──────────────────┐ │ Grounded answer │ │ with [n] cites │ └──────────────────┘The two-step separation means you can independently tune retrieval (number of results, hybrid search, metadata filters) and generation (model choice, system prompt, temperature) without either side affecting the other.
4. Step-by-Step Implementation
Step 1: Configure the clients
import boto3from openai import BedrockOpenAI
REGION = "us-east-1"KB_ID = "YOUR_KNOWLEDGE_BASE_ID" # Replace with your KB IDGEN_MODEL = "openai.gpt-5.6-terra" # or -sol / -luna
# Bedrock Agent Runtime client for the Retrieve APIbar = boto3.client("bedrock-agent-runtime", region_name=REGION)
# BedrockOpenAI derives the Mantle base URL from the region and uses# the AWS credential chain for auth — no API key needed.oai = BedrockOpenAI(aws_region=REGION)
Step 2: Retrieve relevant chunks
def retrieve(query: str, k: int = 6) -> list[dict]: """Pull the most relevant chunks from the Bedrock Knowledge Base.""" resp = bar.retrieve( knowledgeBaseId=KB_ID, retrievalQuery={"text": query}, retrievalConfiguration={ "vectorSearchConfiguration": {"numberOfResults": k} }, ) hits = [] for r in resp["retrievalResults"]: hits.append({ "text": r["content"]["text"], "score": r.get("score"), "source": (r.get("location", {}) .get("s3Location", {}) .get("uri") or r.get("location", {}) .get("webLocation", {}) .get("url") or "unknown"), }) return hitsThe
retrieve API returns chunks ranked by relevance score. Each chunk includes the text content, a relevance score, and the source location (S3 URI or web URL).
Step 3: Build numbered context
Number each chunk so the model can produce inline citations like
[1] , [2] :def build_context(hits: list[dict]) -> str: """Number each chunk so the model can cite [1], [2], ...""" blocks = [] for i, h in enumerate(hits, 1): blocks.append(f"[{i}] (source: {h['source']})\n{h['text']}") return "\n\n".join(blocks)
Step 4: Generate a grounded answer
Pass the retrieved context and the user's question to GPT-5.6 via the OpenAI Responses API:
def generate(query: str, hits: list[dict]) -> str: """GPT-5.6 generates a grounded answer via Bedrock Mantle.""" context = build_context(hits) resp = oai.responses.create( model=GEN_MODEL, input=[ {"role": "system", "content": ( "Answer ONLY from the numbered context. " "Cite sources inline as [n] matching the context blocks. " "If the context doesn't contain the answer, say so." )}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}, ], ) return resp.output_textThe system prompt constrains the model to only use the provided context and cite sources explicitly. This prevents hallucination and gives users traceability back to the original documents.
Step 5: Tie it together
def rag(query: str) -> dict: """Full RAG pipeline: retrieve, generate, return answer + citations.""" hits = retrieve(query) answer = generate(query, hits)
citations = {} for i, h in enumerate(hits, 1): citations[i] = h["source"]
return {"answer": answer, "citations": citations}
Step 6: Run it
out = rag("What are the effects of vessel noise on oyster toadfish calling behavior?")print(out["answer"])for n, src in out["citations"].items(): print(f"[{n}] {src}")Example output:
Vessel noise has been shown to significantly impact oyster toadfish acousticcommunication. Studies demonstrate that toadfish modify their calling behaviorin the presence of boat noise [1], reducing call rates and shifting dominantfrequencies [3].
[1] s3://my-bucket/papers/luczkovich-2016.pdf[2] s3://my-bucket/papers/remage-healey-2006.pdf[3] s3://my-bucket/papers/stanley-2017.pdf
5. How It Works End-to-End
- User submits a query — natural language question passed to
rag()
- Retrieve — The Bedrock Knowledge Base performs vector search and returns the top-k chunks with relevance scores
- Build context — Chunks are numbered
[1],[2], ... and concatenated into a single context string
- Generate — GPT-5.6 receives a system prompt constraining it to answer only from context with inline citations, plus the user's question
- Return — The answer and a citation map (
{1: "s3://...", 2: "s3://..."}) are returned to the caller
6. Why This Pattern Works
- No API key management:
BedrockOpenAIuses AWS credentials natively — same auth for retrieval and generation, nothing to rotate or leak.
- Decoupled retrieval and generation: Swap the generation model (GPT-5.6-terra → GPT-5.6-sol) without touching retrieval logic, or tune
kand search type without changing the prompt.
- Grounded answers with citations: The numbered-context + system-prompt pattern reliably produces inline citations that map to real source documents.
- Simple to extend: Add reranking between retrieve and generate, inject metadata filters, or chain multiple Knowledge Bases — the two-step pattern accommodates all of these without architectural changes.
7. Customization Options
Hybrid search
Enable keyword + semantic search for better recall on queries with specific terms:
retrievalConfiguration={ "vectorSearchConfiguration": { "numberOfResults": k, "overrideSearchType": "HYBRID" }}Metadata filtering
Filter retrieval results by metadata attributes (requires metadata ingestion):
retrievalConfiguration={ "vectorSearchConfiguration": { "numberOfResults": k, "filter": { "equals": {"key": "year", "value": "2024"} } }}Swapping models
Change
GEN_MODEL to use a different GPT-5.6 variant:GEN_MODEL = "openai.gpt-5.6-sol" # Faster, lower costGEN_MODEL = "openai.gpt-5.6-luna" # BalancedGEN_MODEL = "openai.gpt-5.6-terra" # Highest quality
8. Next Steps
- Add a reranker: Insert a cross-encoder reranking step between retrieve and generate to improve precision
- Stream responses: Use
oai.responses.create(..., stream=True)for real-time token streaming
- Deploy as an AgentCore runtime: Wrap
rag()in aBedrockAgentCoreAppentry point for serverless hosting
- Add conversation memory: Maintain chat history across turns by appending prior Q&A pairs to the context
- Multi-KB federation: Query multiple Knowledge Bases in parallel and merge results before generation
Additional Resources
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article