AWS Builder Center
Building a Self-Optimizing Multi-Agent AI System on AWS with Amazon Bedrock, Step Functions, and OpenSearch Serverless

Building a Self-Optimizing Multi-Agent AI System on AWS with Amazon Bedrock, Step Functions, and OpenSearch Serverless

The system addresses a real problem in production AI workloads: over-spending on compute by sending every request regardless of complexity to a single expensive model. By combining intelligent routing, semantic caching with vector search, and automated quality verification, Agentic Mesh achieves up to 60% cost reduction without sacrificing response quality.

Overview

In this post, I'll walk through the architecture and implementation of Agentic Mesh — a fully serverless, multi-agent orchestration system that dynamically routes tasks to specialized AI agents, verifies output quality using an LLM-as-a-Judge pattern, and self-corrects failures using Amazon Bedrock foundation models.
All infrastructure is deployed via AWS SAM, making it reproducible and easy to extend.
What you'll learn:
  • How to design a broker-worker-verifier pattern using AWS Lambda and Step Functions
  • Using Amazon Bedrock's Converse-compatible APIs for multi-model invocation
  • Building a semantic task cache with OpenSearch Serverless k-NN and Amazon Titan Embeddings V2
  • Implementing LLM-as-a-Judge verification and self-correction loops
  • Publishing custom metrics to Amazon CloudWatch for cost and quality observability



The Core Problem: LLM Cost Inefficiency at Scale

Most organizations building on Amazon Bedrock start by routing all inference requests to a single high-capability model like Claude Sonnet. This works, but it ignores a fundamental mismatch: not all tasks have the same complexity.
A request to "summarize these three bullet points" costs the same as "design and implement a thread-safe LRU cache in Python" if both hit the same model. Over thousands of daily requests, this adds up fast.
Additionally, without semantic caching, every request is a cold start. Identical or near-identical queries get processed from scratch, burning tokens that could have been served from memory.
Agentic Mesh solves both problems with a layered approach:
  1. Smart routing — A cheap broker model classifies tasks and routes them to the right specialist
  1. Semantic caching — OpenSearch Serverless + Titan Embeddings V2 serve repeated tasks instantly
  1. Quality gating — Claude Sonnet 4.5 acts as a judge, preventing low-quality outputs from reaching users
  1. Self-correction — Failed outputs are regenerated with enhanced context before being returned



Architecture Overview

The system follows the Well-Architected Framework's operational excellence and cost optimization pillars. Here's the high-level flow:
Client
│
▼
Amazon API Gateway (REST)
│
▼
Amazon SQS (FIFO decoupling)
│
▼
AWS Lambda — Orchestrator
│ (starts execution)
▼
AWS Step Functions State Machine
│
├─► [1] Lambda: Guardrail Check
│ └─ PII redaction + injection detection
│
├─► [2] Lambda: Broker Agent (Llama 3 8B via Bedrock)
│ ├─ Embed task → search OpenSearch Serverless
│ ├─ Cache hit? → skip to Verification
│ └─ No hit → classify type + complexity
│
├─► [3] Lambda: Worker Agent (routed by Broker)
│ ├─ Coder → Claude Sonnet 4.5
│ ├─ Researcher → Claude Sonnet 4.5 (or Haiku 4.5)
│ └─ Summarizer → Claude Haiku 4.5
│
├─► [4] Lambda: Verification Agent (Claude Sonnet 4.5 as Judge)
│ ├─ Score ≥ 7.0 → PASS
│ └─ Score < 7.0 → trigger Self-Correction
│
├─► [5] Lambda: Self-Correction (conditional, Claude Sonnet 4.5)
│
└─► [6] Lambda: Save Results
├─ DynamoDB (task results + TTL)
├─ OpenSearch Serverless (update vector cache)
└─ CloudWatch (custom metrics)

All 10 Lambda functions are defined in a single  template.yaml  and deploy as a unit via  sam deploy .



Deep Dive: Key AWS Services and How They're Used

Amazon Bedrock — Multi-Model Invocation

The system uses four foundation models via Amazon Bedrock, each serving a distinct role:
Model
Bedrock Identifier
Role
Cost Tier
Meta Llama 3 8B Instruct
meta.llama3-8b-instruct-v1:0
Broker routing
~$0.0003/call
Claude Haiku 4.5
us.anthropic.claude-haiku-4-5-20251001-v1:0
Summarization
~$0.001/call
Claude Sonnet 4.5
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Coding, research, verification
~$0.005/call
Titan Embed Text V2
amazon.titan-embed-text-v2:0
Task embeddings
~$0.0002/call
Important note on model IDs: Claude 4+ models require cross-region inference profiles (the  us.  prefix) rather than direct model IDs. Attempting to invoke  anthropic.claude-sonnet-4-5-...  directly will return an error in most regions. Always use  us.anthropic.claude-sonnet-4-5-...  for current-generation Claude models.
The  bedrock_client.py  module provides a unified invocation interface that handles both the Anthropic Messages API format and the Meta Llama prompt format:
# Anthropic Claude — Messages API
body = {
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": max_tokens,
"temperature": temperature,
"messages": messages,
"system": system_prompt,
}

# Meta Llama — Prompt format with special tokens
prompt = "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n"
f"{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n"
f"{user_message}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

The Bedrock client uses  botocore.config.Config  with adaptive retry mode and a 120-second read timeout to handle the latency variability of large model inference:
bedrock_config = Config(
retries={"max_attempts": 3, "mode": "adaptive"},
read_timeout=120,
connect_timeout=10,
)




Amazon Bedrock Guardrails — Pre-Processing Safety Layer

Before any task reaches a language model, it passes through a guardrail check. This combines two mechanisms:
Local pattern matching (no LLM cost):
  • PII detection and redaction using compiled regex (SSN, email, credit card, phone, IP)
  • Prompt injection detection against known attack signatures
INJECTION_PATTERNS = [
re.compile(r"ignore\s+(all\s+)?(previous|prior|above)\s+instructions", re.IGNORECASE),
re.compile(r"system\s*prompt", re.IGNORECASE),
re.compile(r"jailbreak", re.IGNORECASE),
re.compile(r"DAN\s+mode", re.IGNORECASE),
# 10+ patterns total
]

Amazon Bedrock Guardrails API (optional, for content filtering):
response = bedrock_runtime.apply_guardrail(
guardrailIdentifier=guardrail_id,
guardrailVersion="DRAFT",
source="INPUT",
content=[{"text": {"text": task_text}}],
)
# action == "NONE" means safe to proceed
is_safe = response.get("action") == "NONE"

The guardrail fails open (allows requests) if the Bedrock Guardrails service is temporarily unavailable. This is intentional — a guardrail service outage shouldn't take down your entire pipeline.



AWS Step Functions — State Machine Orchestration

The pipeline is a Step Functions Express Workflow with 13 states defined in Amazon States Language (ASL). Using Step Functions over direct Lambda chaining provides three critical benefits:
1. Conditional branching with Choice states:
"CheckVerification": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.verification_result.passed",
"BooleanEquals": false,
"Next": "SelfCorrection"
}
],
"Default": "PrepareSuccessResult"
}

2. Retry with exponential backoff on every Lambda invocation:
"Retry": [
{
"ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException"],
"IntervalSeconds": 3,
"MaxAttempts": 2,
"BackoffRate": 2
}
]

3. Structured error handling with Catch blocks that route failures to a graceful terminal state rather than letting exceptions propagate unpredictably.
The state machine receives the full execution context (guardrail result, broker decision, worker output, verification score) in the Step Functions state object. Each Lambda function reads what it needs and writes its output to a dedicated  ResultPath  — clean, auditable, and debuggable in the Step Functions console.



OpenSearch Serverless — Semantic Task Cache

This is the highest-leverage component for cost optimization. Every successfully completed task (quality score ≥ 7.0) is embedded with Titan Embeddings V2 and stored in an OpenSearch Serverless collection as a k-NN vector document.
Index configuration:
index_body = {
"settings": {"index": {"knn": True, "knn.algo_param.ef_search": 100}},
"mappings": {
"properties": {
"task_embedding": {
"type": "knn_vector",
"dimension": 1024,
"method": {
"engine": "nmslib",
"space_type": "cosinesimil",
"name": "hnsw",
"parameters": {"ef_construction": 256, "m": 48},
},
},
"task_text": {"type": "text"},
"quality_score": {"type": "float"},
"answer": {"type": "text"},
# ...
}
},
}

Query pattern — k-NN search with threshold:
query = {
"size": 3,
"query": {
"knn": {
"task_embedding": {
"vector": task_embedding,
"k": 3,
}
}
},
}

response = client.search(index=INDEX_NAME, body=query)

# Only accept results above similarity threshold
results = [
hit["_source"] | {"score": hit["_score"]}
for hit in response["hits"]["hits"]
if hit["_score"] >= 0.85
]

Authentication uses AWS SigV4 via  requests-aws4auth , targeting the  aoss  service name for OpenSearch Serverless:
awsauth = AWS4Auth(
credentials.access_key,
credentials.secret_key,
region,
"aoss", # Note: "aoss" not "es" for Serverless
session_token=credentials.token,
)

The cache uses a deterministic document ID (SHA-256 hash of task text) to prevent duplicate entries for identical tasks.



Amazon DynamoDB — Task Results Store

Task results are persisted in DynamoDB with on-demand capacity and a 30-day TTL. The schema is flat and designed for single-key access by  task_id :
item = {
"task_id": task_id, # Partition key
"task": task_text,
"status": "SUCCESS|CORRECTED|BLOCKED|FAILED",
"answer": answer,
"agent": "coder|researcher|summarizer|cache",
"model": model_id,
"quality_score": Decimal(str(quality_score)),
"cost_estimate": Decimal(str(total_cost)),
"escalated": escalated,
"cache_hit": cache_hit,
"completed_at": datetime.now(timezone.utc).isoformat(),
"ttl": int(time.time()) + (30 * 24 * 60 * 60),
}

Important: DynamoDB does not accept Python  float  values. Always convert floats to  Decimal(str(value))  before writing. The  _to_decimal()  helper function in  save_results.py  handles this recursively across the entire item dictionary.



Amazon CloudWatch — Custom Metrics and Observability

Every Lambda publishes structured metrics to CloudWatch under the  AgenticMesh  namespace. The full list of custom metrics:
Metric Name
Dimensions
Description
CostPerTask
Agent
USD cost per invocation
TaskLatency
Agent
Processing time in milliseconds
QualityScore
Agent
Verification score (0–10)
RoutingDecision
Agent, CacheHit
Task routing distribution
CacheHitRate
—
Fraction of tasks served from cache
EscalationRate
—
Fraction of tasks triggering self-correction
TokensUsed
Agent
Total tokens consumed
VerificationPassRate
—
Fraction of tasks passing quality check
Metric publishing pattern:
cloudwatch.put_metric_data(
Namespace="AgenticMesh",
MetricData=[
{
"MetricName": "CostPerTask",
"Dimensions": [{"Name": "Agent", "Value": agent}],
"Timestamp": datetime.now(timezone.utc),
"Value": cost,
"Unit": "None",
},
],
)

All Lambda functions use  AWS Lambda Powertools for Python  for structured JSON logging and AWS X-Ray tracing:
from aws_lambda_powertools import Logger, Tracer

logger = Logger(service="agentic-mesh")
tracer = Tracer(service="agentic-mesh")

@tracer.capture_lambda_handler
@logger.inject_lambda_context(log_event=True)
def lambda_handler(event, context):
...

Lambda Powertools'  inject_lambda_context  decorator automatically appends  cold_start ,  function_name , and  request_id  to every log entry — invaluable for debugging multi-Lambda pipelines.



The Broker Routing Logic

The Broker is the central decision-maker. It handles two scenarios:
Scenario A — Cache Hit:
embed(task) → search OpenSearch → similarity ≥ 0.85 AND quality ≥ 7.0
→ Return cached answer (zero worker cost, ~100ms)

Scenario B — Cache Miss:
invoke Llama 3 8B → parse JSON → extract task_type + complexity
→ Route to appropriate worker

The Llama 3 8B prompt is designed to return structured JSON with strict constraints:
BROKER_SYSTEM_PROMPT = """You are a task routing agent. Respond ONLY with valid JSON:
{"task_type": "coding|research|summarize", "complexity": "low|medium|high", "reasoning": "..."}

Rules:
- "coding": writing code, debugging, algorithms, technical implementation
- "research": analysis, comparison, explaining concepts, pros/cons
- "summarize": condensing, extracting key points, reformatting
"""

If the LLM returns malformed JSON (which happens occasionally with smaller models), the Broker catches the parse error and defaults to  researcher  with  medium  complexity — a safe, conservative fallback that won't fail.



Cost Calculation Implementation

Per-task cost tracking is implemented in  cost_tracker.py  with a model pricing dictionary:
MODEL_PRICING = {
"meta.llama3-8b-instruct-v1:0": {
"input_per_1k": 0.0003,
"output_per_1k": 0.0006,
"tier": "cheap",
},
"us.anthropic.claude-haiku-4-5-20251001-v1:0": {
"input_per_1k": 0.0008,
"output_per_1k": 0.004,
"tier": "cheap",
},
"us.anthropic.claude-sonnet-4-5-20250929-v1:0": {
"input_per_1k": 0.003,
"output_per_1k": 0.015,
"tier": "elite",
},
}

def calculate_cost(model_id, input_tokens, output_tokens):
pricing = MODEL_PRICING.get(model_id)
return round(
(input_tokens / 1000) * pricing["input_per_1k"] +
(output_tokens / 1000) * pricing["output_per_1k"],
8
)

The  save_results.py  Lambda aggregates costs across the full pipeline before writing to DynamoDB:
total_cost = worker_cost + verification_cost + correction_cost # if escalated

This gives you an accurate end-to-end cost per task, not just the worker inference cost.



Deployment: End-to-End with AWS SAM

The system deploys entirely via SAM. Here's the minimal deployment path:
Prerequisites:
  • Python 3.10+
  • AWS CLI v2 configured with appropriate credentials
  • AWS SAM CLI 1.x
  • Amazon Bedrock model access enabled for all four models
Enable Bedrock model access in the  AWS Console → Bedrock → Model Access :
  • ✅ Meta Llama 3 8B Instruct
  • ✅ Anthropic Claude Haiku 4.5
  • ✅ Anthropic Claude Sonnet 4.5
  • ✅ Amazon Titan Embed Text V2
Build and deploy:
git clone https://github.com/yourusername/agentic-mesh.git
cd agentic-mesh

# Install dependencies
pip install -r requirements.txt

# Build Lambda deployment packages
sam build

# First-time guided deploy (sets stack name, region, S3 bucket)
sam deploy --guided

# Subsequent deploys
sam deploy --no-confirm-changeset

SAM provisions all resources — API Gateway, 10 Lambda functions, Step Functions state machine, SQS queue, DynamoDB table, OpenSearch Serverless collection, CloudWatch dashboard, and all IAM roles — in a single CloudFormation stack.
Test your deployment:
# Submit a coding task
TASK_ID=$(curl -s -X POST https://YOUR_API/Prod/task \
-H "Content-Type: application/json" \
-d '{"task": "Write a Python merge sort with type hints", "type_hint": "coding"}' \
| jq -r '.task_id')

# Poll for results
curl https://YOUR_API/Prod/task/$TASK_ID




IAM Least-Privilege Patterns

Each Lambda function has a scoped-down IAM role. The pattern used throughout  template.yaml :
# Worker Lambda — only needs Bedrock InvokeModel and CloudWatch PutMetricData
Policies:
- Statement:
- Effect: Allow
Action:
- bedrock:InvokeModel
Resource:
- !Sub "arn:aws:bedrock:${AWS::Region}::foundation-model/us.anthropic.claude-sonnet-4-5-20250929-v1:0"
- !Sub "arn:aws:bedrock:${AWS::Region}::foundation-model/us.anthropic.claude-haiku-4-5-20251001-v1:0"
- Effect: Allow
Action:
- cloudwatch:PutMetricData
Resource: "*"
Condition:
StringEquals:
cloudwatch:namespace: "AgenticMesh"

The Orchestrator Lambda — which starts Step Functions executions — is the only function with  states:StartExecution  permission, and only on the specific state machine ARN. No Lambda has wildcard resource access.



Performance Characteristics

Based on  us-east-1  benchmarks with warm Lambda invocations:
Scenario
Latency
Cost
Cache hit (vector match)
< 100ms
~$0.0002
Summarization (Haiku 4.5)
~3–5s
~$0.001
Research (Sonnet 4.5)
~8–12s
~$0.005
Coding (Sonnet 4.5)
~5–15s
~$0.003–0.008
Self-correction (+ Sonnet 4.5)
+8–12s
+$0.005
Cold start overhead
~2–3s
—
Scalability limits and defaults:
  • Lambda concurrency: 1,000 concurrent executions (default, adjustable)
  • SQS throughput: up to 3,000 messages/second
  • DynamoDB: on-demand, no capacity limits
  • OpenSearch Serverless: auto-scales compute units and storage



Lessons Learned and Gotchas

1. Cross-region inference profile requirement for Claude 4+ Direct model IDs ( anthropic.claude-sonnet-4-5-... ) fail for current-generation Claude models. You must use the inference profile format ( us.anthropic.claude-sonnet-4-5-... ). This is documented but easy to miss.
2. OpenSearch Serverless auth uses  aoss  not  es  When building  AWS4Auth  for OpenSearch Serverless, the service name is  aoss . Using  es  (the service name for managed OpenSearch) will result in authentication errors.
3. DynamoDB float serialization  boto3 's DynamoDB resource client will raise  TypeError: Float types are not supported  on any Python  float  value. Always convert to  Decimal(str(value)) . Build a recursive helper function rather than handling each field individually.
4. Step Functions state size limits Step Functions has a 256KB limit on the state object. If worker outputs are large (long code generation or research answers), you'll hit this. The solution is to store large outputs in DynamoDB or S3 and pass only a reference key through the state machine.
5. OpenSearch Serverless cold start OpenSearch Serverless collections have a warm-up period when idle. The first vector search after a period of inactivity can add 2–5 seconds. Build this into your SLA expectations or implement a scheduled keepalive.



What's Next

The roadmap focuses on expanding the system's capabilities while maintaining the serverless-first principle:
  • WebSocket support — Replace polling with API Gateway WebSocket for real-time pipeline progress
  • Multi-modal inputs — Process images and PDFs using Claude Vision via Bedrock
  • Agent collaboration — Multi-step pipelines where agent outputs become inputs to subsequent agents
  • Cognito authentication — Add API Gateway authorizer for multi-tenant deployments
  • S3 + CloudFront — Move the web dashboard from local Python server to global CDN hosting



Summary

Agentic Mesh demonstrates a production-viable pattern for cost-optimized, reliable multi-agent AI on AWS. The key architectural principles:
  • Use cheap models for cheap jobs — Llama 3 8B for routing decisions, Haiku 4.5 for summarization
  • Cache semantically, not literally — OpenSearch k-NN finds similar tasks, not just identical ones
  • Verify before you return — LLM-as-a-Judge catches low-quality outputs automatically
  • Self-correct, don't fail — A single retry with explicit feedback is more cost-effective than error-handling by the caller
  • Observe everything — Custom CloudWatch metrics on cost, latency, quality, and cache performance give you the data to optimize over time
The full project — SAM template, all Lambda handlers, Step Functions ASL, and the web dashboard available on GitHub:  https://github.com/shubham5027/Agentic-Mesh-AWS  .



Resources



Built with Amazon Bedrock, AWS Lambda, Step Functions, OpenSearch Serverless, and DynamoDB. All infrastructure deployed via AWS SAM.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article