Where a voice agent's 2-second budget goes: benchmarking a Bedrock AgentCore backend
I built the backend of an IVR-style cancellation agent on Bedrock AgentCore Runtime, MCP Gateway, and Lambda and ran 16 latency tests (text in/out, no telephony). Pre-warming brings session cold start to 224 ms, but a full 3-tool turn is about 4.8 s, before ASR and TTS. The main cost is the LLM loop.
Summary
I measured whether the backend of a multi-turn agent, built on Amazon Bedrock AgentCore Runtime (with pre-warming) plus AWS Lambda, can fit a 2 second budget. This is a small benchmark (tens of iterations per test, not a load test), and it covers the agent compute leg only: text in, text out, invoked over
InvokeAgentRuntime . No telephony, ASR, or TTS. Sixteen tests, summarized:- Pre-warming brings session cold start from 1,019 ms to 224 ms p50 (under the 500 ms target), effective with as little as a 0.5 s warm-up lead.
- A full 3 tool turn is about 4.8 s p50 (TTFT about 4.4 s), roughly 2.5x the 2 s target, and that is before any speech I/O.
- The largest component is the three sequential LLM rounds, not the infrastructure.
- Strands and LangGraph performed about the same.
- Provisioned Concurrency and framework choice made no measurable difference to latency in this setup. The data below explains why.
The use case
A generic airline reservation cancellation agent, representative of a multi-tool, multi-turn IVR flow with a roughly 2 second response expectation: retrieve booking, check eligibility, determine refund method, explain in one or two sentences, and on confirmation execute. I benchmarked the backend of this agent, the part that would sit behind a voice frontend.
Architecture
What I measured, the agent backend:
Python client (boto3, HTTPS) | InvokeAgentRuntime (text in, text out) vAgentCore Runtime (Strands or LangGraph agent, container; LLM: Claude Haiku 4.5) | vAgentCore Gateway (MCP, AWS_IAM / SigV4) | v4 x Lambda (ARM64, 512 MB) | vDynamoDBIn production this backend would sit behind an Amazon Connect voice frontend (IVR, ASR, TTS). That frontend was not part of this benchmark. The Gateway uses AWS_IAM (SigV4) inbound auth, so the runtime and client sign requests directly, with no Cognito to stand up.
Assumptions and method
Scope
- This measures the agent backend only, invoked as text over
InvokeAgentRuntimefrom a laptop. No Amazon Connect, telephony, ASR, or TTS was involved. A production voice turn adds those legs (typically several hundred ms of ASR plus TTS, plus the media path) on top of every number here, so the real end to end is higher than shown.
Benchmark setup
- Region
us-east-1, single AWS account, one point in time. Bedrock latency varies, so treat run to run gaps under about 1 s as noise.
- Model: Claude Haiku 4.5 via the
us-east-1cross-region inference profile, max output capped at 512 tokens.
- Runtime in
networkMode=PUBLIC(no VPC), no SnapStart. Lambdas ARM64, 512 MB. Provisioned Concurrency = 5 on each of 4 functions (20 instances).
- Iterations: 20 per test by default; 5 to 10 for LLM-heavy tests; 15 concurrent for the burst test. Percentiles are nearest-rank on raw samples. This is a small sample, not a statistically large run.
- Client ran from a laptop over the public internet (about 70 ms RTT to
us-east-1). Absolute numbers are conservative versus an in-region caller; relative comparisons are unaffected.
- "Cold" = a new session per iteration. "Pre-warmed" = a warmup ping, a wait, then the measured call. Burst warm/cold is classified by a 1,500 ms latency threshold, a heuristic, not a ground-truth cold-start signal.
- Framework comparison: one container image switched by an
AGENT_FRAMEWORKenv var, state-matched with per-session memory, n=10 back to back.
Cost model
- Volume: 1,000 calls/day = 30,000/month, where one call is one full 3-tool turn. A real cancellation call has at least two turns (the confirm turn also runs the tools and the LLM), so a full conversation would cost roughly double.
- LLM tokens are measured, not assumed: about 4,110 input and 375 output tokens per turn (accumulated across the three LLM rounds; multi-round tool calling replays tool schemas and context each round).
- Rates (on-demand, retrieved Aug 2026, verify before relying): Claude Haiku 4.5 $1.00 / M input and $5.00 / M output; Lambda Provisioned Concurrency $0.0000041667 / GB-s plus duration $0.0000097222 / GB-s plus $0.20 / M requests. No batch or prompt-cache discounts applied.
- Provisioned Concurrency is billed 24/7 (730 hr/month) whether idle or not.
- Excluded from the total: AgentCore Runtime consumption (vCPU-s + GB-s), any AgentCore Gateway per-invocation charge, DynamoDB (on-demand, negligible at this volume), data transfer, and CloudWatch. The real all-in figure is therefore higher than the roughly $290/mo shown.
What the 16 tests covered
# | Test |
|---|---|
T1 | Lambda tool latency, Provisioned Concurrency |
T2 | Lambda tool latency, $LATEST |
T3 | Tool latency through the MCP Gateway |
T4 | Runtime cold start, container |
T5 | Runtime cold start, code / ZIP |
T6 | Pre-warmed session ping |
T7 | Round-robin across 3 runtime endpoints |
T8 | Warm same-session ping |
T9 | Full end-to-end turn, with and without pre-warm |
T10 | Burst of 15 concurrent new sessions |
T11 | Streaming first-byte (reframed from an original WebSocket test; AgentCore has no client WebSocket) |
T12 | Streaming time-to-first-token |
T13 | Multi-turn, second turn on a warm session |
T14 | Pre-warm timing curve (0.5 to 10 s wait) |
T15 | Parallel vs sequential tool execution |
T16 | Integration Lambda hop vs direct invoke |
The sections below report the ones that carried the findings.
Results
Lambda tool latency and Provisioned Concurrency
Config | p50 | p95 |
|---|---|---|
Direct invoke, Provisioned Concurrency | 113 ms | 389 ms |
Direct invoke, $LATEST | 108 ms | 170 ms |
Via MCP Gateway | 411 ms | n/a |
Provisioned Concurrency made no measurable difference here. At p50 it matched
$LATEST (113 vs 108 ms), and its p95 was higher (a first-call sampling artifact), not lower. The reason: the handler is a trivial DynamoDB read that stays warm between calls, so there is no cold start for PC to prevent. PC helps against cold starts under bursty or idle traffic, a condition this benchmark did not create at the tool layer. It is not a way to speed up an already-warm function.The MCP Gateway added about 300 ms per call as measured from the client (per-call SigV4 requests, no HTTP keep-alive). From inside the runtime (in region, persistent MCP session) the same calls ran about 175 to 260 ms. Either way it adds measurable latency.
Cold start and pre-warming
Test | p50 |
|---|---|
Container cold start | 1,019 ms |
Code / ZIP cold start | ~4,756 ms |
Pre-warmed (2 s wait) | 224 ms |
Warm session ping | 209 ms |
Two findings: pre-warming works (a 0.5 s lead was enough; latency was flat from 0.5 s to 10 s of warm-up wait), and container deployment cold-started roughly 4 to 5 times faster than the code / ZIP path. In my packaging, the code path also required vendoring dependencies into the ZIP rather than relying on an install step, which is worth confirming against your own setup.
End to end agent turn (backend, no speech I/O)
Metric | p50 |
|---|---|
Full 3 tool turn (pre-warmed) | ~4,800 ms |
Full turn (no pre-warm) | ~5,450 ms |
Streaming TTFT | ~4,400 ms |
Burst, 15 concurrent (warm by latency threshold) | 11 / 15 |
Integration Lambda hop overhead (see note) | +57 ms |
The integration hop was measured with a small Lambda that itself calls
InvokeAgentRuntime , standing in for the Connect-integration Lambda. That is a simulation of the hop, not Amazon Connect.Where the time goes (one representative turn, about 5 s)
LLM round 1 (plan) ~1.3 s -> get_reservation ~0.2 sLLM round 2 (plan) ~1.8 s -> check_eligibility + get_refund_method ~0.3 s (parallel)LLM round 3 (compose) ~1.5 sAbout 4.6 s of this turn is LLM time across three sequential rounds; tools are about 0.5 s. Across all runs the pre-warmed turn p50 was about 4.8 s. Cold start, after pre-warming, is negligible, and streaming does not reduce TTFT here because the first token of the response cannot be produced until tool results return.
Strands vs LangGraph
Same agent both ways, from one container image switched by an
AGENT_FRAMEWORK env var (identical Gateway, tools, model, prompt, and timing contract), state-matched with per-session memory, n=10 back to back.Metric | Strands | LangGraph |
|---|---|---|
Full E2E p50 | 4,805 ms | 4,646 ms |
Multi-turn (2nd turn) p50 | 4,408 ms | 4,037 ms |
Tool parallelism | 10 / 10 | 10 / 10 |
Avg input tokens per turn | 4,120 | 3,832 |
The two are about the same, within roughly 1 s of run to run variance. Both parallelize independent tools in every run. The only consistent difference is Strands using about 7 to 13 percent more tokens.
Cost at 1,000 calls / day
Using the published rates and measured token usage from the Assumptions section:
Item | $ / mo |
|---|---|
Lambda Provisioned Concurrency (20 idle instances at 512 MB) | ~$109 |
LLM tokens (Haiku 4.5, measured usage) | ~$180 |
Lambda duration and requests | ~$0.15 |
Total (excludes AgentCore Runtime consumption) | ~$290 |
A note on token assumptions: a "500 in / 200 out tokens" estimate puts LLM cost at about $45/mo. The measured turns used about 4,110 input tokens, because multi-round tool calling replays the tool schemas and context every round, which makes it about $180/mo, roughly 4x higher. Measure token usage rather than assuming it.
What I would try next (not yet measured)
Since about 4.6 s of the turn is the LLM loop, that is where the time is:
- Reduce the number of LLM rounds. For a fixed workflow, the read tools can be fired in parallel on call arrival, followed by a single LLM call to phrase the answer, rather than letting the model plan each step.
- Prompt caching for the static system prompt and tool schemas, to lower per-round latency and token cost.
- Stream the response to TTS so speech can begin as phrasing starts.
- Co-locate or bypass the Gateway for latency-critical tools.
Takeaways
- Pre-warming brings AgentCore session cold start to about 224 ms, under the 500 ms target.
- Provisioned Concurrency did not help a warm, trivial handler. It is worth using where cold starts actually occur.
- The MCP Gateway hop adds measurable latency, about 175 to 300 ms per call.
- For a multi-tool agentic turn, the 2 s target is limited by LLM orchestration, not infrastructure. Reducing round trips matters more than adding infrastructure.
- This is the backend only. ASR, TTS, and telephony are additive and not measured here, so a real voice deployment starts further from 2 s than these numbers show.
I'll be publishing the code sample soon so stayed tuned!
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article