12 lessons for reducing multi-agent latency on AWS
A practitioner's guide to latency optimization on Amazon Bedrock AgentCore with the Strands Agents SDK
Multi-agent latency matters more than you think
When a multi-agent chatbot takes 40 seconds to answer a simple question, latency becomes a product-launch risk. One customer built such a chatbot on Amazon Bedrock AgentCore to answer end-user questions about their accounts, transactions, and service requests. The architecture followed the recommended pattern: an orchestrator agent routes requests to specialized sub-agents, each sub-agent calls tools through an AgentCore Gateway, and the gateway invokes AWS Lambda functions that reach back-end systems.
The first end-to-end test took over 40 seconds. For a customer-facing chatbot, that number is not acceptable. Over the next ten weeks, we instrumented every phase of the request. We identified bottlenecks in the LLM inference layer, the agent framework, the infrastructure, and the corporate network, then resolved them one at a time. The result: a reduction of over 80% in our testing, with the same architecture and the same foundation model.
This post distills twelve high-impact lessons from that engagement. Each lesson includes the symptom that surfaced it, the fix, and a link to the relevant documentation. Although the examples reference Amazon Bedrock AgentCore and the Strands Agents SDK, the principles apply to most multi-agent systems built on foundation models.
The architecture
The system follows a standard multi-agent orchestration pattern on Amazon Bedrock AgentCore:
- An orchestrator agent receives the user message and selects a sub-agent based on intent.
- The selected sub-agent (for example, a domain-specific sub-agent) connects to an AgentCore Gateway that exposes tools over the Model Context Protocol (MCP).
- The gateway invokes an AWS Lambda function for each tool call.
- The Lambda function calls the enterprise back-end systems and returns structured data.
- The sub-agent formats the response and returns it to the orchestrator, which relays it to the user.
The foundation model is Anthropic Claude Haiku, called through a global cross-region inference profile. The entire stack runs serverless: AgentCore Runtime for compute, AgentCore Gateway for tool discovery and invocation, AgentCore Identity for authentication, AgentCore Memory for conversation history, AgentCore Observability for tracing, and Lambda for the API access layer.
Note: why Lambda and not directly API endpoints ? In our case, the Lambda was necessary to proxy access to back-end servers for data manipulation, but AgentCore Gateway can directly integrate with OpenAPI-compatible endpoints.
LLM inference optimization
Lesson 1: Scope tools per sub-agent
When a sub-agent discovers tools from a shared gateway, it may receive far more tools than it needs. In this case, a domain-specific sub-agent received 40 tools from the gateway even though it only needed 2. The LLM spent 15 seconds reasoning through 40 tool definitions before selecting one.
The fix: pass only the relevant tools when constructing the agent. In the Strands Agents SDK, the tools parameter on the Agent() constructor controls both which tools the LLM sees in its prompt and which tools the agent can invoke. Restricting it from 40 tools to 2 cut tool-selection time from 15 seconds to under 2 seconds.
Before (all tools from gateway):
# Sub-agent sees all 40 gateway toolsagent = Agent(model=model, tools=[mcp_client])After (scoped to 2):
# Filter to only the tools this sub-agent needsneeded = [t for t in mcp_client.tools()if t.name in ("get_account_status", "get_transaction_details")]agent = Agent(model=model, tools=needed)Lesson 2: Prompt caching eliminates repeated processing
System prompts and tool definitions are static between deployments. Without caching, the model reprocesses these tokens on every invocation. In our case, with 27,000 input tokens of system prompt and tool definitions, that overhead adds up quickly.
In the Strands Agents SDK, set CacheConfig(strategy="auto") on the BedrockModel constructor. This places cache checkpoints on tools, the system prompt, and the last user message. After the first invocation (a cache WRITE), subsequent calls hit the cache and skip reprocessing. In our case, 27,000 tokens were served from cache on every warm call. Cache entries are shared across users and sessions at the account-and-model-and-region level, matched by content prefix.
Prompt caching is not model-agnostic. As of this writing, Amazon Bedrock supports it for Anthropic Claude models, OpenAI GPT models and Amazon Nova models. Each model family has different minimum token thresholds for cache checkpoints. Check the prompt caching documentation for the current list of supported models and thresholds.
from strands import Agentfrom strands.models import BedrockModelfrom strands.models.bedrock import CacheConfigmodel = BedrockModel(model_id="global.anthropic.claude-haiku-v1:0",region_name="<your-region>",cache_config=CacheConfig(strategy="auto"), # <-- one line)Reference: Amazon Bedrock prompt caching | Strands CacheConfig | Well-Architected caching best practices
Lesson 3: The agent loop synthesis tax
This was the single largest bottleneck, and the hardest to spot. After a sub-agent calls a tool and receives structured data, the Strands agent loop makes a second LLM call to "synthesize" the result into a natural-language response. In a multi-agent chain, this multiplies: the sub-agent synthesizes its tool output (16 seconds, roughly 3,000 output tokens at 250 tokens per second), then the orchestrator re-synthesizes the sub-agent response (13 seconds, roughly 4,000 output tokens). Two synthesis passes consumed 29 of the ~40 total seconds.
The fix: use limits={"turns": 1} when invoking the agent. This is a TypedDict called Limits from strands.types.agent. It stops the agent loop after one turn (the LLM picks a tool, the tool executes, and the loop exits without a synthesis call). The stop reason is limit_turns. You extract the raw tool result from agent.messages[-1] and pass it upstream directly. Apply this to both the sub-agent and the orchestrator.
# Invoke with a one-turn limit: LLM picks a tool, tool runs, loop exits.# No synthesis call. stop_reason == "limit_turns".result = sub_agent(user_input,limits={"turns": 1},)# The raw tool output is in the last messagetool_result = sub_agent.messages[-1]# Pass it to the orchestrator without re-generationreturn tool_result["content"]
This solution might not always be desirable. In lesson 12, we look at another option to accelerate the output part with Generative UI.
Another alternative: intercept the tool result with a lifecycle hook. The turns limit is a blunt instrument because it caps the whole loop. If you need to suppress synthesis for some tool calls but allow it for others, a Strands lifecycle hook is more surgical. Register a callback on the AfterToolCallEvent (from strands.hooks): it fires after a tool returns, and the typed event carries the tool result, so you can read the output there and short-circuit the response rather than capping turns globally. For more information, refer to Using Strands hooks to deterministically control your agent .
Lesson 4: System prompt instructions have limits
Before discovering the turns limit, we tried a prompt-based approach: instruct the sub-agent with "return the JSON payload from the tool call as-is, do not summarize or reformat." This reduced commentary but did not eliminate token generation. The LLM still produced 3,000 output tokens at roughly 250 tokens per second, because it reformatted the JSON, added field labels, and wrapped the output in a markdown code fence.
Prompt instructions can influence tone and structure, but they cannot prevent the model from generating output tokens. Only a code-level constraint (invocation limits) or a lifecycle hook can enforce a hard stop. When you need deterministic control over the agent loop, use the SDK, not the prompt.
Infrastructure optimization
Lesson 5: Lambda cold starts dominate tool latency
The tool Lambda was a container-image function with 1 GB of memory, deployed in a VPC. Cold starts measured 16.9 seconds. With provisioned concurrency (PC), warm invocations dropped to 533 milliseconds (a 30x reduction measured in our load tests).
Gotcha: provisioned concurrency must be configured on a Lambda alias or a published version, not on $LATEST. If your Terraform or CDK code points the gateway target at the unqualified function ARN (aws_lambda_function.fn.arn) instead of the alias ARN (aws_lambda_alias.live.arn), PC is allocated but never used, and every invocation still cold-starts. Check the InitDuration Amazon CloudWatch metric: if it appears on calls you expect to be warm, the alias wiring is wrong.
For cost efficiency, combine a baseline of provisioned concurrency during business hours with AWS Application Auto Scaling for burst traffic. A scheduled scaling policy that drops PC to zero overnight can reduce Lambda fixed costs by roughly 60% (based on the difference between 24/7 and business-hours-only provisioning).
Terraform example (alias wiring):
resource "aws_lambda_provisioned_concurrency_config" "pc" {function_name = aws_lambda_function.tool.function_namequalifier = aws_lambda_alias.live.nameprovisioned_concurrent_executions = 5}# Gateway target MUST point to the alias, not $LATEST# WRONG: function_arn = aws_lambda_function.tool.arn# RIGHT:resource "aws_bedrock_agentcore_gateway_target" "tool" {lambda_arn = aws_lambda_alias.live.arn # <-- alias ARN}Lesson 6: Parallelize independent back-end calls
The tool Lambda called three separate back-end APIs sequentially. Each call went through a corporate network proxy, adding latency. Total tool execution was 11.6 seconds.
Using asyncio.gather() to run the three calls concurrently reduced tool execution to 2.8 seconds, bounded by the slowest individual call. This is a standard Python pattern, but it is often overlooked as the original code can be written synchronously and still "work".
# Before: sequential (11.6s)policy = await fetch_account(customer_id)history = await fetch_transactions(customer_id)status = await fetch_entitlements(customer_id)# After: concurrent (2.8s, bounded by slowest call)policy, history, status = await asyncio.gather(fetch_account(customer_id),fetch_transactions(customer_id),fetch_entitlements(customer_id),)
Lesson 7: Offload token management and cache connections at module scope
The original agent code managed its own OAuth token lifecycle: it fetched a new JWT from Amazon Cognito on every invocation, created a new MCP client connection, and loaded conversation memory from scratch. That added over 1.5 seconds of overhead per call before the LLM even started.
The better approach: delegate token management to AgentCore Identity. The
@requires_access_token decorator handles the full OAuth lifecycle (acquisition, caching in the Token Vault, and automatic refresh before expiry) with zero custom code. For MCP tool connections, cache the client and the discovered tools list at module scope so they persist across invocations within the same microVM session. Running any remaining initialization steps concurrently with asyncio.gather() saved another 500 milliseconds.Similarly, memory writes were synchronous and blocking, adding 3 seconds per invocation. Switching to async fire-and-forget writes (the memory store is eventually consistent anyway) brought that down to 4 milliseconds.
Note that memory writes can also be batched. On the AgentCore Strands SDK integration, set batch_size greater than 1 on AgentCoreMemoryConfig: messages buffer in memory and flush to AgentCore Memory in a single API call once the buffer fills, cutting write calls in multi-turn conversations and complementing the async fire-and-forget pattern.
Regarding the new AgentCore Runtime (V2): V2 restores instances from a snapshot taken after startup, so anything you compute at module scope is frozen into that snapshot and shared by every restored instance. A tool catalog fetched from AgentCore Gateway looks like an ideal startup value because it is slow to load, but caching it at startup pins your tool inventory to the moment the snapshot was taken, and a tool added to the gateway afterward stays invisible until you redeploy. On V2, cache connection objects and tokens at module scope, but fetch anything that can change at runtime (such as the gateway tool list) inside the request path, or accept that changes require a redeployment. For more information, refer to Optimize your agent for AgentCore Runtime V2 .
Reference: AgentCore Identity token management | AgentCore Identity onboarding sample (06_identity) | AgentCore Memory batch write example
Observability
Lesson 8: Network proxies are invisible bottlenecks
After fixing Lambda cold starts, provisioned concurrency was active and the Lambda init duration disappeared from metrics. Yet tool call latency was still not satisfying. The root cause: every back-end API call routed through a corporate network proxy before reaching the on-premises systems. That proxy added 2 to 4 seconds per call, and the latency was identical whether the Lambda was cold or warm.
This bottleneck was invisible until we added per-call timing instrumentation inside the Lambda function itself. Amazon CloudWatch metrics at the Lambda level showed healthy durations (the proxy latency was inside the function execution, not an initialization cost). Only wrapping each HTTP call with a high-resolution timer revealed where the time went.
Lesson: when tool latency persists after infrastructure optimization, instrument individual network calls, not only function-level metrics.
Lesson 9: Observability gaps compound debugging time
Several wrong conclusions during this engagement came from inferring latency by calculating gaps between log timestamps. The orchestrator appeared to take 25 seconds on cold start, but that "cold start" was actually user think time between conversation turns. A Lambda that appeared slow was actually fast; the slow part was the HTTP call inside it.
You get built-in observability through Amazon CloudWatch Application Signals, including per-invocation traces with span-level granularity. Request AgentCore traces early in the optimization process. Verify provisioned concurrency with the
InitDuration metric. Check prompt cache hits with cacheReadInputTokens in the model invocation metrics (note: the key is camelCase in the Converse API response, not snake_case).Gateway configuration
Lesson 10: Semantic search does not replace tool scoping
The AgentCore Gateway supports
searchType: SEMANTIC on the tools/list MCP endpoint, which adds an x_amz_bedrock_agentcore_search meta-tool that the agent can call to find relevant tools. This is useful when the tool catalog is large and frequently changing. However, it does not filter the tool list that the Strands SDK passes to the LLM. The agent still sees all 40 tools in its prompt, and the search tool is an additional one on top.Worse, when using Strands, tools discovered through semantic search that are not in the agent's tool registry cause a "tool not found" error at invocation time. For fixed-domain sub-agents where you know the tools at build time, hardcode the subset in the Agent(tools=[...]) constructor. Use semantic search for discovery-oriented orchestrators that work with large, evolving tool catalogs.
Perceived latency and generative UI
Lesson 11: Stream UI tokens, not final payloads
Reducing actual latency is half the battle. The other half is perceived latency: how long the user feels they are waiting. A traditional approach buffers the complete response, then renders it. The user stares at a loading spinner for the entire generation time.
An alternative approach uses a generative UI framework (such as OpenUI ) where the LLM emits a compact UI language instead of prose. The frontend parser renders components as tokens arrive. In a customer-service prototype built on AgentCore, the first data card appeared 2 to 3 seconds after the request. Generating all 30 cards took 21 seconds, but the user saw a response assembling itself from the first token. Users no longer stare at a blank screen.
The key architectural insight: the component library on the frontend and the system prompt fed to the agent are generated from a single source file. A build step compiles the React components and their Zod schemas into the agent prompt. Adding a field means editing one file and rerunning the generator; the component contract and the model instructions cannot drift apart.
Here is how that works in practice. A single file defines each UI component with both its React rendering logic and a Zod schema (Zod is a TypeScript validation library that describes the shape and types of an object) defining the props the model is allowed to generate:
// frontend/src/openui/library.tsx (single source of truth)const DataCard = defineComponent({name: "DataCard",description: "A single record rendered as a card.",props: z.object({recordId: z.string(),status: z.enum(["Active", "Pending", "Closed"]),amount: z.number(),}),component: ({ props }) => (<Card><Badge>{props.status}</Badge>...</Card>),});A build step (npm run gen:prompt) compiles all Zod schemas from that file into the agent system prompt. The model then knows exactly what components exist and what fields they accept. Add a prop to the schema, rerun the generator, redeploy: the frontend renderer and the model instructions update from the same source, so they cannot drift apart.
SSE wire format (each line is one event):
{"t":"boot", "coldStart":true, "moduleTotalMs":2840}{"t":"status", "phase":"thinking", "ms":3}{"t":"tool", "name":"get_data", "state":"start", "ms":812}{"t":"tool", "name":"get_data", "state":"end", "ms":1240, "durationMs":428}{"t":"text", "v":"root = AssistantResponse(\"You have 30 records\", [c1,c2..])"}{"t":"text", "v":"c1 = ClaimCard(\"CLM-0111\",\"Motor\",\"Open\"...)"}{"t":"done", "ms":8047, "firstTokenMs":1310}// The frontend parser renders each DataCard as tokens arrive.// The shell ("root") appears first; children fill in behind it.Progressive rendering turns an output-volume problem (30 cards at 250 tokens per second) into a time-to-first-card problem (one card in 2 seconds). For data-heavy agentic applications, this is the single most effective UX optimization.
Reference: OpenUI framework (GitHub) | Well-Architected Agentic AI Lens, performance design principles
Lesson 12: Warm up the session on page load
AgentCore Runtime provisions a microVM for each session. The cold start cost (staging the deployment artifact, starting the Python interpreter, importing libraries) ranges from 3 to 5 seconds on the original runtime (AgentCore Runtime v1) and grew with image size. If the user pays this on their first question, it adds to the perceived wait time. A workaround: send a lightweight warm-up request
{"warmup": true} payload via the InvokeAgentRuntime HTTP API when the web page loads. This is not a built-in AgentCore feature; it is a pattern you implement in your agent code. The agent receives the payload, detects the warmup flag, and returns immediately without calling the model. The microVM is now booted and ready. By the time the user types a question, the session is warm and overhead drops to roughly 500 milliseconds.The frontend calls the Runtime directly over HTTPS (
POST /runtimes/{arn}/invocations , authenticated with a Bearer JWT from Amazon Cognito). There is no API Gateway or Lambda relay in between. Combine this with session reuse across browser reloads (persisting the session ID in localStorage rather than minting a new one per page load). Precompiled Python bytecode can halve import time further. Together, these reduced first-request overhead from 5 seconds to under 700 milliseconds in our prototype.Warm-up on page load (TypeScript):
// Fire a no-op invocation while the user reads the landing page.// The agent boots the microVM without calling the model.useEffect(() => {const controller = new AbortController();invokeAgent({sessionId,payload: { warmup: true },signal: controller.signal,});return () => controller.cancel();}, []);Update for the new AgentCore Runtime (V2): in September 2026, AWS made a new AgentCore Runtime generally available. It restores each instance from a snapshot of the initialized agent process, which delivers consistent cold starts regardless of image size (AWS reports a P75 of roughly 2 seconds across 200 MB to 2 GB images, compared with 5 to 30 seconds before). On V2, the page-load warm-up above matters much less, because the platform removes most of the image-size-dependent cold-start variability. Opt in by setting platformVersion to V2 when you create or update the runtime. For more information, refer to launch blog post .
Summary: latency breakdown
The following table shows the effect of each optimization, measured on multiple different prototypes with a similar multi-agent architecture. It provides an high-level order of magnitude of each possible optimization.
Optimization | Layer | Before | After |
|---|---|---|---|
Scope tools (40 to 2) | LLM inference | 15s | 1.5s |
Prompt caching | LLM inference | 27K tokens reprocessed | Cache HIT |
Invocation limits (turns: 1) | Agent loop | 29s (two synthesis passes) | 0s |
Provisioned concurrency | Lambda | 16.9s cold | 533ms warm |
Parallel back-end calls | Lambda / network | 11.6s sequential | 2.8s parallel |
Module-level caching | Agent init | 1s per call | 2ms |
Async memory writes | Agent init | 3s blocking | 4ms fire-and-forget |
Network proxy instrumentation | Observability | Hidden 2-4s per call | Identified and mitigated |
Generative UI streaming | UX | Blank screen for 21s | First card in 2s |
Session warm-up on page load | UX | 5s cold start | 500ms warm |
Putting it all together
Multi-agent architectures on Amazon Bedrock AgentCore distribute work across specialized components. Each boundary (orchestrator to sub-agent, agent to gateway, gateway to Lambda, Lambda to back-end) is a place where latency hides. Optimization requires visibility at every layer.
The highest-impact lesson from this engagement: the LLM synthesis pass in a multi-agent chain. Two unnecessary LLM calls consumed 80% of total latency, and they were invisible until we measured token output from each agent separately. The fix was one parameter change.
Start by instrumenting each phase of your agent flow with span-level tracing. Identify whether your bottleneck is in LLM inference (tool selection, synthesis), infrastructure (cold starts, network), or the agent framework (connection setup, memory). Then apply the targeted fix. The twelve lessons in this post address common bottlenecks we saw in production multi-agent deployments.
Related resources
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article