
Building a Self-Optimizing Multi-Agent AI System on AWS with Amazon Bedrock, Step Functions, and OpenSearch Serverless
The system addresses a real problem in production AI workloads: over-spending on compute by sending every request regardless of complexity to a single expensive model. By combining intelligent routing, semantic caching with vector search, and automated quality verification, Agentic Mesh achieves up to 60% cost reduction without sacrificing response quality.
Overview
In this post, I'll walk through the architecture and implementation of Agentic Mesh — a fully serverless, multi-agent orchestration system that dynamically routes tasks to specialized AI agents, verifies output quality using an LLM-as-a-Judge pattern, and self-corrects failures using Amazon Bedrock foundation models.
All infrastructure is deployed via AWS SAM, making it reproducible and easy to extend.
What you'll learn:
- How to design a broker-worker-verifier pattern using AWS Lambda and Step Functions
- Using Amazon Bedrock's Converse-compatible APIs for multi-model invocation
- Building a semantic task cache with OpenSearch Serverless k-NN and Amazon Titan Embeddings V2
- Implementing LLM-as-a-Judge verification and self-correction loops
- Publishing custom metrics to Amazon CloudWatch for cost and quality observability
The Core Problem: LLM Cost Inefficiency at Scale
Most organizations building on Amazon Bedrock start by routing all inference requests to a single high-capability model like Claude Sonnet. This works, but it ignores a fundamental mismatch: not all tasks have the same complexity.
A request to "summarize these three bullet points" costs the same as "design and implement a thread-safe LRU cache in Python" if both hit the same model. Over thousands of daily requests, this adds up fast.
Additionally, without semantic caching, every request is a cold start. Identical or near-identical queries get processed from scratch, burning tokens that could have been served from memory.
Agentic Mesh solves both problems with a layered approach:
- Smart routing — A cheap broker model classifies tasks and routes them to the right specialist
- Semantic caching — OpenSearch Serverless + Titan Embeddings V2 serve repeated tasks instantly
- Quality gating — Claude Sonnet 4.5 acts as a judge, preventing low-quality outputs from reaching users
- Self-correction — Failed outputs are regenerated with enhanced context before being returned
Architecture Overview
The system follows the Well-Architected Framework's operational excellence and cost optimization pillars. Here's the high-level flow:
Client │ ▼Amazon API Gateway (REST) │ ▼Amazon SQS (FIFO decoupling) │ ▼AWS Lambda — Orchestrator │ (starts execution) ▼AWS Step Functions State Machine │ ├─► [1] Lambda: Guardrail Check │ └─ PII redaction + injection detection │ ├─► [2] Lambda: Broker Agent (Llama 3 8B via Bedrock) │ ├─ Embed task → search OpenSearch Serverless │ ├─ Cache hit? → skip to Verification │ └─ No hit → classify type + complexity │ ├─► [3] Lambda: Worker Agent (routed by Broker) │ ├─ Coder → Claude Sonnet 4.5 │ ├─ Researcher → Claude Sonnet 4.5 (or Haiku 4.5) │ └─ Summarizer → Claude Haiku 4.5 │ ├─► [4] Lambda: Verification Agent (Claude Sonnet 4.5 as Judge) │ ├─ Score ≥ 7.0 → PASS │ └─ Score < 7.0 → trigger Self-Correction │ ├─► [5] Lambda: Self-Correction (conditional, Claude Sonnet 4.5) │ └─► [6] Lambda: Save Results ├─ DynamoDB (task results + TTL) ├─ OpenSearch Serverless (update vector cache) └─ CloudWatch (custom metrics)
All 10 Lambda functions are defined in a single
template.yaml and deploy as a unit via sam deploy .
Deep Dive: Key AWS Services and How They're Used
Amazon Bedrock — Multi-Model Invocation
The system uses four foundation models via Amazon Bedrock, each serving a distinct role:
Model | Bedrock Identifier | Role | Cost Tier |
|---|---|---|---|
Meta Llama 3 8B Instruct | meta.llama3-8b-instruct-v1:0 | Broker routing | ~$0.0003/call |
Claude Haiku 4.5 | us.anthropic.claude-haiku-4-5-20251001-v1:0 | Summarization | ~$0.001/call |
Claude Sonnet 4.5 | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | Coding, research, verification | ~$0.005/call |
Titan Embed Text V2 | amazon.titan-embed-text-v2:0 | Task embeddings | ~$0.0002/call |
Important note on model IDs: Claude 4+ models require cross-region inference profiles (theus.prefix) rather than direct model IDs. Attempting to invokeanthropic.claude-sonnet-4-5-...directly will return an error in most regions. Always useus.anthropic.claude-sonnet-4-5-...for current-generation Claude models.
The
bedrock_client.py module provides a unified invocation interface that handles both the Anthropic Messages API format and the Meta Llama prompt format:# Anthropic Claude — Messages APIbody = { "anthropic_version": "bedrock-2023-05-31", "max_tokens": max_tokens, "temperature": temperature, "messages": messages, "system": system_prompt,}
# Meta Llama — Prompt format with special tokensprompt = "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n" f"{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n" f"{user_message}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
The Bedrock client uses
botocore.config.Config with adaptive retry mode and a 120-second read timeout to handle the latency variability of large model inference:bedrock_config = Config( retries={"max_attempts": 3, "mode": "adaptive"}, read_timeout=120, connect_timeout=10,)
Amazon Bedrock Guardrails — Pre-Processing Safety Layer
Before any task reaches a language model, it passes through a guardrail check. This combines two mechanisms:
Local pattern matching (no LLM cost):
- PII detection and redaction using compiled regex (SSN, email, credit card, phone, IP)
- Prompt injection detection against known attack signatures
INJECTION_PATTERNS = [ re.compile(r"ignore\s+(all\s+)?(previous|prior|above)\s+instructions", re.IGNORECASE), re.compile(r"system\s*prompt", re.IGNORECASE), re.compile(r"jailbreak", re.IGNORECASE), re.compile(r"DAN\s+mode", re.IGNORECASE), # 10+ patterns total]
Amazon Bedrock Guardrails API (optional, for content filtering):
response = bedrock_runtime.apply_guardrail( guardrailIdentifier=guardrail_id, guardrailVersion="DRAFT", source="INPUT", content=[{"text": {"text": task_text}}],)# action == "NONE" means safe to proceedis_safe = response.get("action") == "NONE"
The guardrail fails open (allows requests) if the Bedrock Guardrails service is temporarily unavailable. This is intentional — a guardrail service outage shouldn't take down your entire pipeline.
AWS Step Functions — State Machine Orchestration
The pipeline is a Step Functions Express Workflow with 13 states defined in Amazon States Language (ASL). Using Step Functions over direct Lambda chaining provides three critical benefits:
1. Conditional branching with Choice states:
"CheckVerification": { "Type": "Choice", "Choices": [ { "Variable": "$.verification_result.passed", "BooleanEquals": false, "Next": "SelfCorrection" } ], "Default": "PrepareSuccessResult"}
2. Retry with exponential backoff on every Lambda invocation:
"Retry": [ { "ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException"], "IntervalSeconds": 3, "MaxAttempts": 2, "BackoffRate": 2 }]
3. Structured error handling with Catch blocks that route failures to a graceful terminal state rather than letting exceptions propagate unpredictably.
The state machine receives the full execution context (guardrail result, broker decision, worker output, verification score) in the Step Functions state object. Each Lambda function reads what it needs and writes its output to a dedicated
ResultPath — clean, auditable, and debuggable in the Step Functions console.
OpenSearch Serverless — Semantic Task Cache
This is the highest-leverage component for cost optimization. Every successfully completed task (quality score ≥ 7.0) is embedded with Titan Embeddings V2 and stored in an OpenSearch Serverless collection as a k-NN vector document.
Index configuration:
index_body = { "settings": {"index": {"knn": True, "knn.algo_param.ef_search": 100}}, "mappings": { "properties": { "task_embedding": { "type": "knn_vector", "dimension": 1024, "method": { "engine": "nmslib", "space_type": "cosinesimil", "name": "hnsw", "parameters": {"ef_construction": 256, "m": 48}, }, }, "task_text": {"type": "text"}, "quality_score": {"type": "float"}, "answer": {"type": "text"}, # ... } },}
Query pattern — k-NN search with threshold:
query = { "size": 3, "query": { "knn": { "task_embedding": { "vector": task_embedding, "k": 3, } } },}
response = client.search(index=INDEX_NAME, body=query)
# Only accept results above similarity thresholdresults = [ hit["_source"] | {"score": hit["_score"]} for hit in response["hits"]["hits"] if hit["_score"] >= 0.85]
Authentication uses AWS SigV4 via
requests-aws4auth , targeting the aoss service name for OpenSearch Serverless:awsauth = AWS4Auth( credentials.access_key, credentials.secret_key, region, "aoss", # Note: "aoss" not "es" for Serverless session_token=credentials.token,)
The cache uses a deterministic document ID (SHA-256 hash of task text) to prevent duplicate entries for identical tasks.
Amazon DynamoDB — Task Results Store
Task results are persisted in DynamoDB with on-demand capacity and a 30-day TTL. The schema is flat and designed for single-key access by
task_id :item = { "task_id": task_id, # Partition key "task": task_text, "status": "SUCCESS|CORRECTED|BLOCKED|FAILED", "answer": answer, "agent": "coder|researcher|summarizer|cache", "model": model_id, "quality_score": Decimal(str(quality_score)), "cost_estimate": Decimal(str(total_cost)), "escalated": escalated, "cache_hit": cache_hit, "completed_at": datetime.now(timezone.utc).isoformat(), "ttl": int(time.time()) + (30 * 24 * 60 * 60),}
Important: DynamoDB does not accept Pythonfloatvalues. Always convert floats toDecimal(str(value))before writing. The_to_decimal()helper function insave_results.pyhandles this recursively across the entire item dictionary.
Amazon CloudWatch — Custom Metrics and Observability
Every Lambda publishes structured metrics to CloudWatch under the
AgenticMesh namespace. The full list of custom metrics:Metric Name | Dimensions | Description |
|---|---|---|
CostPerTask | Agent | USD cost per invocation |
TaskLatency | Agent | Processing time in milliseconds |
QualityScore | Agent | Verification score (0–10) |
RoutingDecision | Agent, CacheHit | Task routing distribution |
CacheHitRate | — | Fraction of tasks served from cache |
EscalationRate | — | Fraction of tasks triggering self-correction |
TokensUsed | Agent | Total tokens consumed |
VerificationPassRate | — | Fraction of tasks passing quality check |
Metric publishing pattern:
cloudwatch.put_metric_data( Namespace="AgenticMesh", MetricData=[ { "MetricName": "CostPerTask", "Dimensions": [{"Name": "Agent", "Value": agent}], "Timestamp": datetime.now(timezone.utc), "Value": cost, "Unit": "None", }, ],)
All Lambda functions use AWS Lambda Powertools for Python for structured JSON logging and AWS X-Ray tracing:
from aws_lambda_powertools import Logger, Tracer
logger = Logger(service="agentic-mesh")tracer = Tracer(service="agentic-mesh")
@tracer.capture_lambda_handler@logger.inject_lambda_context(log_event=True)def lambda_handler(event, context): ...
Lambda Powertools'
inject_lambda_context decorator automatically appends cold_start , function_name , and request_id to every log entry — invaluable for debugging multi-Lambda pipelines.
The Broker Routing Logic
The Broker is the central decision-maker. It handles two scenarios:
Scenario A — Cache Hit:
embed(task) → search OpenSearch → similarity ≥ 0.85 AND quality ≥ 7.0→ Return cached answer (zero worker cost, ~100ms)
Scenario B — Cache Miss:
invoke Llama 3 8B → parse JSON → extract task_type + complexity→ Route to appropriate worker
The Llama 3 8B prompt is designed to return structured JSON with strict constraints:
BROKER_SYSTEM_PROMPT = """You are a task routing agent. Respond ONLY with valid JSON:{"task_type": "coding|research|summarize", "complexity": "low|medium|high", "reasoning": "..."}
Rules:- "coding": writing code, debugging, algorithms, technical implementation- "research": analysis, comparison, explaining concepts, pros/cons- "summarize": condensing, extracting key points, reformatting"""
If the LLM returns malformed JSON (which happens occasionally with smaller models), the Broker catches the parse error and defaults to
researcher with medium complexity — a safe, conservative fallback that won't fail.
Cost Calculation Implementation
Per-task cost tracking is implemented in
cost_tracker.py with a model pricing dictionary:MODEL_PRICING = { "meta.llama3-8b-instruct-v1:0": { "input_per_1k": 0.0003, "output_per_1k": 0.0006, "tier": "cheap", }, "us.anthropic.claude-haiku-4-5-20251001-v1:0": { "input_per_1k": 0.0008, "output_per_1k": 0.004, "tier": "cheap", }, "us.anthropic.claude-sonnet-4-5-20250929-v1:0": { "input_per_1k": 0.003, "output_per_1k": 0.015, "tier": "elite", },}
def calculate_cost(model_id, input_tokens, output_tokens): pricing = MODEL_PRICING.get(model_id) return round( (input_tokens / 1000) * pricing["input_per_1k"] + (output_tokens / 1000) * pricing["output_per_1k"], 8 )
The
save_results.py Lambda aggregates costs across the full pipeline before writing to DynamoDB:total_cost = worker_cost + verification_cost + correction_cost # if escalated
This gives you an accurate end-to-end cost per task, not just the worker inference cost.
Deployment: End-to-End with AWS SAM
The system deploys entirely via SAM. Here's the minimal deployment path:
Prerequisites:
- Python 3.10+
- AWS CLI v2 configured with appropriate credentials
- AWS SAM CLI 1.x
- Amazon Bedrock model access enabled for all four models
- ✅ Meta Llama 3 8B Instruct
- ✅ Anthropic Claude Haiku 4.5
- ✅ Anthropic Claude Sonnet 4.5
- ✅ Amazon Titan Embed Text V2
Build and deploy:
git clone https://github.com/yourusername/agentic-mesh.gitcd agentic-mesh
# Install dependenciespip install -r requirements.txt
# Build Lambda deployment packagessam build
# First-time guided deploy (sets stack name, region, S3 bucket)sam deploy --guided
# Subsequent deployssam deploy --no-confirm-changeset
SAM provisions all resources — API Gateway, 10 Lambda functions, Step Functions state machine, SQS queue, DynamoDB table, OpenSearch Serverless collection, CloudWatch dashboard, and all IAM roles — in a single CloudFormation stack.
Test your deployment:
# Submit a coding taskTASK_ID=$(curl -s -X POST https://YOUR_API/Prod/task \ -H "Content-Type: application/json" \ -d '{"task": "Write a Python merge sort with type hints", "type_hint": "coding"}' \ | jq -r '.task_id')
# Poll for resultscurl https://YOUR_API/Prod/task/$TASK_ID
IAM Least-Privilege Patterns
Each Lambda function has a scoped-down IAM role. The pattern used throughout
template.yaml :# Worker Lambda — only needs Bedrock InvokeModel and CloudWatch PutMetricDataPolicies: - Statement: - Effect: Allow Action: - bedrock:InvokeModel Resource: - !Sub "arn:aws:bedrock:${AWS::Region}::foundation-model/us.anthropic.claude-sonnet-4-5-20250929-v1:0" - !Sub "arn:aws:bedrock:${AWS::Region}::foundation-model/us.anthropic.claude-haiku-4-5-20251001-v1:0" - Effect: Allow Action: - cloudwatch:PutMetricData Resource: "*" Condition: StringEquals: cloudwatch:namespace: "AgenticMesh"
The Orchestrator Lambda — which starts Step Functions executions — is the only function with
states:StartExecution permission, and only on the specific state machine ARN. No Lambda has wildcard resource access.
Performance Characteristics
Based on
us-east-1 benchmarks with warm Lambda invocations:Scenario | Latency | Cost |
|---|---|---|
Cache hit (vector match) | < 100ms | ~$0.0002 |
Summarization (Haiku 4.5) | ~3–5s | ~$0.001 |
Research (Sonnet 4.5) | ~8–12s | ~$0.005 |
Coding (Sonnet 4.5) | ~5–15s | ~$0.003–0.008 |
Self-correction (+ Sonnet 4.5) | +8–12s | +$0.005 |
Cold start overhead | ~2–3s | — |
Scalability limits and defaults:
- Lambda concurrency: 1,000 concurrent executions (default, adjustable)
- SQS throughput: up to 3,000 messages/second
- DynamoDB: on-demand, no capacity limits
- OpenSearch Serverless: auto-scales compute units and storage
Lessons Learned and Gotchas
1. Cross-region inference profile requirement for Claude 4+ Direct model IDs (
anthropic.claude-sonnet-4-5-... ) fail for current-generation Claude models. You must use the inference profile format ( us.anthropic.claude-sonnet-4-5-... ). This is documented but easy to miss.2. OpenSearch Serverless auth uses
aoss not es When building AWS4Auth for OpenSearch Serverless, the service name is aoss . Using es (the service name for managed OpenSearch) will result in authentication errors.3. DynamoDB float serialization
boto3 's DynamoDB resource client will raise TypeError: Float types are not supported on any Python float value. Always convert to Decimal(str(value)) . Build a recursive helper function rather than handling each field individually.4. Step Functions state size limits Step Functions has a 256KB limit on the state object. If worker outputs are large (long code generation or research answers), you'll hit this. The solution is to store large outputs in DynamoDB or S3 and pass only a reference key through the state machine.
5. OpenSearch Serverless cold start OpenSearch Serverless collections have a warm-up period when idle. The first vector search after a period of inactivity can add 2–5 seconds. Build this into your SLA expectations or implement a scheduled keepalive.
What's Next
The roadmap focuses on expanding the system's capabilities while maintaining the serverless-first principle:
- WebSocket support — Replace polling with API Gateway WebSocket for real-time pipeline progress
- Multi-modal inputs — Process images and PDFs using Claude Vision via Bedrock
- Agent collaboration — Multi-step pipelines where agent outputs become inputs to subsequent agents
- Cognito authentication — Add API Gateway authorizer for multi-tenant deployments
- S3 + CloudFront — Move the web dashboard from local Python server to global CDN hosting
Summary
Agentic Mesh demonstrates a production-viable pattern for cost-optimized, reliable multi-agent AI on AWS. The key architectural principles:
- Use cheap models for cheap jobs — Llama 3 8B for routing decisions, Haiku 4.5 for summarization
- Cache semantically, not literally — OpenSearch k-NN finds similar tasks, not just identical ones
- Verify before you return — LLM-as-a-Judge catches low-quality outputs automatically
- Self-correct, don't fail — A single retry with explicit feedback is more cost-effective than error-handling by the caller
- Observe everything — Custom CloudWatch metrics on cost, latency, quality, and cache performance give you the data to optimize over time
The full project — SAM template, all Lambda handlers, Step Functions ASL, and the web dashboard available on GitHub: https://github.com/shubham5027/Agentic-Mesh-AWS .
Resources
Built with Amazon Bedrock, AWS Lambda, Step Functions, OpenSearch Serverless, and DynamoDB. All infrastructure deployed via AWS SAM.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article