AWS Builder Center

Builder Center

Building a Serverless RAG Pipeline with Amazon Bedrock Knowledge Bases and AWS Lambda

Building a Serverless RAG Pipeline with Amazon Bedrock Knowledge Bases and AWS Lambda

A step-by-step guide to building a scalable, cost-guarded Retrieval-Augmented Generation pipeline using Amazon Bedrock Knowledge Bases, S3, and AWS Lambda.

Mahi kumar

Student

Introduction: Moving Beyond Basic Prompting

Directly prompting foundation models (FMs) works well for isolated tasks like summarization, text extraction, or content drafting. However, when deploying generative AI to production, developers inevitably face three major bottlenecks:
  1. Hallucinations: Models lack ground-truth context and invent answers when uncertain.
  2. Context Window Limitations: Entire technical libraries or company document stores cannot fit into single model prompts economically.
  3. Stale Information: Retraining or fine-tuning models continuously to keep up with dynamic data is prohibitively expensive.
Retrieval-Augmented Generation (RAG) solves these challenges by anchoring responses to private, verified data sources.
Traditionally, implementing RAG required self-hosting vector databases, writing custom chunking scripts, spinning up orchestrators like LangChain on long-lived EC2 instances, and managing embedding pipelines manually. With Amazon Bedrock Knowledge Bases and Amazon OpenSearch Serverless (AOSS), AWS turns this entire architecture into a fully managed, serverless pipeline.

Architectural Breakdown

The architecture is divided into two decoupled pipelines: an Ingestion Pipeline and a Retrieval/Query Pipeline.
1
2
3
4
5
6
7
8
9
10
11
12
13
[Document Upload] ──> [Amazon S3] ──> [EventBridge / S3 Notification] ──> [AWS Lambda (Sync)]
│
▼
[Bedrock Knowledge Base]
│
(Chunk & Embed via Titan)
│
▼
[Amazon OpenSearch Serverless]

[User Request] ──> [API Gateway] ──> [AWS Lambda (Query)] ──> [Bedrock RetrieveAndGenerate] ──> [Claude 3.5 / Nova]
│
(Pulls Context Chunks)

1. Ingestion Pipeline (Data Synchronization)

  • Storage Layer: Documents (.pdf, .md, .txt, .docx) are uploaded to an Amazon S3 source bucket.
  • Orchestration Layer: S3 emits an ObjectCreated event, triggering an ingestion AWS Lambda function via Amazon EventBridge.
  • Vector Transformation: The Lambda function initiates Bedrock's ingestion job, which extracts text, chunks it, converts chunks to vector embeddings using Amazon Titan Text Embeddings v2, and indexes them directly into Amazon OpenSearch Serverless.

2. Query Pipeline (Context Retrieval & Generation)

  • API Entrypoint: An incoming query reaches an AWS Lambda function behind an API Gateway.
  • Retrieve & Generate: The function invokes the RetrieveAndGenerate API in Amazon Bedrock runtime.
  • Semantic Search: Bedrock queries OpenSearch Serverless for the top-$K$ most semantically relevant text chunks.
  • Response Synthesis: Bedrock injects the retrieved context into the system prompt of a foundational model (such as Claude 3.5 Haiku or Amazon Nova), returning a grounded answer with precise source citations.

Implementation Guide

Step 1: Create the Amazon S3 Bucket

Create an Amazon S3 bucket to act as the primary document data store:
1
aws s3 mb s3://my-bedrock-knowledgebase-source-bucket --region us-east-1
Enable bucket versioning to track document updates and maintain consistent sync states.

Step 2: Configure the Bedrock Knowledge Base

  1. Navigate to the Amazon Bedrock Console > Knowledge bases > Create knowledge base.
  2. Data Source: Select Amazon S3 and point it to s3://my-bedrock-knowledgebase-source-bucket.
  3. Embeddings Model: Select Titan Text Embeddings v2. Set vector dimensions to 512 (for lower latency and reduced storage costs) or 1024 (for dense technical material).
  4. Vector Store: Choose Quick create a new vector store - Recommended. Bedrock will automatically provision:
  • An OpenSearch Serverless vector search collection.
  • Vector indexing configurations.
  • Access and encryption policies.
  1. Note down the generated Knowledge Base ID and Data Source ID.

Step 3: Automate Ingestion with AWS Lambda

When documents are uploaded to S3, the Knowledge Base should sync automatically.

1. Lambda IAM Permissions

Attach the following policy to the Lambda execution role:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"bedrock:StartIngestionJob",
"bedrock:GetIngestionJob"
],
"Resource": "arn:aws:bedrock:*:*:knowledge-base/*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}

2. Lambda Code (ingestion_handler.py)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import json
import os
import boto3

bedrock_agent = boto3.client('bedrock-agent')

KNOWLEDGE_BASE_ID = os.environ['KNOWLEDGE_BASE_ID']
DATA_SOURCE_ID = os.environ['DATA_SOURCE_ID']

def lambda_handler(event, context):
try:
response = bedrock_agent.start_ingestion_job(
knowledgeBaseId=KNOWLEDGE_BASE_ID,
dataSourceId=DATA_SOURCE_ID,
description="Auto-sync triggered by S3 upload"
)

job_id = response['ingestionJob']['ingestionJobId']
print(f"Ingestion job started successfully. Job ID: {job_id}")

return {
"statusCode": 200,
"body": json.dumps({
"message": "Ingestion job started",
"jobId": job_id
})
}
except Exception as e:
print(f"Error starting ingestion job: {str(e)}")
raise e

Step 4: Build the Query & Generation Lambda

The query function leverages Bedrock's unified retrieve_and_generate endpoint, eliminating manual prompt construction, chunk stitching, and citation parsing.

Lambda Code (query_handler.py)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
import json
import os
import boto3

bedrock_runtime = boto3.client('bedrock-agent-runtime')

KNOWLEDGE_BASE_ID = os.environ['KNOWLEDGE_BASE_ID']
MODEL_ARN = "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-haiku-20241022-v1:0"

def lambda_handler(event, context):
try:
body = json.loads(event.get('body', '{}'))
user_query = body.get('query')

if not user_query:
return {
"statusCode": 400,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({"error": "Missing 'query' parameter"})
}

response = bedrock_runtime.retrieve_and_generate(
input={'text': user_query},
retrieveAndGenerateConfiguration={
'type': 'KNOWLEDGE_BASE',
'knowledgeBaseConfiguration': {
'knowledgeBaseId': KNOWLEDGE_BASE_ID,
'modelArn': MODEL_ARN,
'retrievalConfiguration': {
'vectorSearchConfiguration': {
'numberOfResults': 3
}
}
}
}
)

output_text = response['output']['text']
citations = response.get('citations', [])

return {
"statusCode": 200,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({
"answer": output_text,
"citations": citations
})
}

except Exception as e:
print(f"Inference error: {str(e)}")
return {
"statusCode": 500,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({"error": str(e)})
}

Cost Optimization & Production Safeguards

To prevent unexpected billing while maintaining production reliability:
  • Cap OpenSearch Serverless OCUs: OpenSearch Serverless charges per OpenSearch Compute Unit (OCU). Configure maximum indexing and search limits to 0.5 or 1 OCU under collection settings during initial rollout to prevent over-allocation.
  • Optimize Foundation Model Tiers: For routine document lookup and Q&A, smaller models like Anthropic Claude 3.5 Haiku or Amazon Nova Micro reduce token latency and costs by more than 60% compared to top-tier models like Claude 3.5 Sonnet.
  • Configure S3 Lifecycle Rules: Apply S3 lifecycle policies to transition raw, ingested source PDFs to S3 Standard-Infrequent Access (IA) or Glacier Flexible Retrieval once embeddings are indexed.
  • Implement Amazon Bedrock Guardrails: Attach guardrails to filter sensitive customer data (PII), mask output information, and block prompt-injection attacks before queries reach the model.

Conclusion

By combining Amazon Bedrock Knowledge Bases, Amazon OpenSearch Serverless, and AWS Lambda, you can build a resilient, enterprise-grade RAG pipeline without provisioning a single server or managing vector index pipelines manually.
From here, the natural progression is connecting this Knowledge Base to an Amazon Bedrock Agent, enabling multi-step reasoning, external API executions, and full conversational autonomy.
# aws-lambda# ai-engineering# serverless# rag# amazon-bedrock
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article