AWS Builder Center

Where a voice agent's 2-second budget goes: benchmarking a Bedrock AgentCore backend

I built the backend of an IVR-style cancellation agent on Bedrock AgentCore Runtime, MCP Gateway, and Lambda and ran 16 latency tests (text in/out, no telephony). Pre-warming brings session cold start to 224 ms, but a full 3-tool turn is about 4.8 s, before ASR and TTS. The main cost is the LLM loop.

Summary

I measured whether the backend of a multi-turn agent, built on Amazon Bedrock AgentCore Runtime (with pre-warming) plus AWS Lambda, can fit a 2 second budget. This is a small benchmark (tens of iterations per test, not a load test), and it covers the agent compute leg only: text in, text out, invoked over  InvokeAgentRuntime . No telephony, ASR, or TTS. Sixteen tests, summarized:
  • Pre-warming brings session cold start from 1,019 ms to 224 ms p50 (under the 500 ms target), effective with as little as a 0.5 s warm-up lead.
  • A full 3 tool turn is about 4.8 s p50 (TTFT about 4.4 s), roughly 2.5x the 2 s target, and that is before any speech I/O.
  • The largest component is the three sequential LLM rounds, not the infrastructure.
  • Strands and LangGraph performed about the same.
  • Provisioned Concurrency and framework choice made no measurable difference to latency in this setup. The data below explains why.

The use case

A generic airline reservation cancellation agent, representative of a multi-tool, multi-turn IVR flow with a roughly 2 second response expectation: retrieve booking, check eligibility, determine refund method, explain in one or two sentences, and on confirmation execute. I benchmarked the backend of this agent, the part that would sit behind a voice frontend.

Architecture

What I measured, the agent backend:
Python client (boto3, HTTPS)
| InvokeAgentRuntime (text in, text out)
v
AgentCore Runtime (Strands or LangGraph agent, container; LLM: Claude Haiku 4.5)
|
v
AgentCore Gateway (MCP, AWS_IAM / SigV4)
|
v
4 x Lambda (ARM64, 512 MB)
|
v
DynamoDB
In production this backend would sit behind an Amazon Connect voice frontend (IVR, ASR, TTS). That frontend was not part of this benchmark. The Gateway uses AWS_IAM (SigV4) inbound auth, so the runtime and client sign requests directly, with no Cognito to stand up.

Assumptions and method

Scope
  • This measures the agent backend only, invoked as text over  InvokeAgentRuntime  from a laptop. No Amazon Connect, telephony, ASR, or TTS was involved. A production voice turn adds those legs (typically several hundred ms of ASR plus TTS, plus the media path) on top of every number here, so the real end to end is higher than shown.
Benchmark setup
  • Region  us-east-1 , single AWS account, one point in time. Bedrock latency varies, so treat run to run gaps under about 1 s as noise.
  • Model: Claude Haiku 4.5 via the  us-east-1  cross-region inference profile, max output capped at 512 tokens.
  • Runtime in  networkMode=PUBLIC  (no VPC), no SnapStart. Lambdas ARM64, 512 MB. Provisioned Concurrency = 5 on each of 4 functions (20 instances).
  • Iterations: 20 per test by default; 5 to 10 for LLM-heavy tests; 15 concurrent for the burst test. Percentiles are nearest-rank on raw samples. This is a small sample, not a statistically large run.
  • Client ran from a laptop over the public internet (about 70 ms RTT to  us-east-1 ). Absolute numbers are conservative versus an in-region caller; relative comparisons are unaffected.
  • "Cold" = a new session per iteration. "Pre-warmed" = a warmup ping, a wait, then the measured call. Burst warm/cold is classified by a 1,500 ms latency threshold, a heuristic, not a ground-truth cold-start signal.
  • Framework comparison: one container image switched by an  AGENT_FRAMEWORK  env var, state-matched with per-session memory, n=10 back to back.
Cost model
  • Volume: 1,000 calls/day = 30,000/month, where one call is one full 3-tool turn. A real cancellation call has at least two turns (the confirm turn also runs the tools and the LLM), so a full conversation would cost roughly double.
  • LLM tokens are measured, not assumed: about 4,110 input and 375 output tokens per turn (accumulated across the three LLM rounds; multi-round tool calling replays tool schemas and context each round).
  • Rates (on-demand, retrieved Aug 2026, verify before relying): Claude Haiku 4.5 $1.00 / M input and $5.00 / M output; Lambda Provisioned Concurrency $0.0000041667 / GB-s plus duration $0.0000097222 / GB-s plus $0.20 / M requests. No batch or prompt-cache discounts applied.
  • Provisioned Concurrency is billed 24/7 (730 hr/month) whether idle or not.
  • Excluded from the total: AgentCore Runtime consumption (vCPU-s + GB-s), any AgentCore Gateway per-invocation charge, DynamoDB (on-demand, negligible at this volume), data transfer, and CloudWatch. The real all-in figure is therefore higher than the roughly $290/mo shown.

What the 16 tests covered

#
Test
T1
Lambda tool latency, Provisioned Concurrency
T2
Lambda tool latency,  $LATEST 
T3
Tool latency through the MCP Gateway
T4
Runtime cold start, container
T5
Runtime cold start, code / ZIP
T6
Pre-warmed session ping
T7
Round-robin across 3 runtime endpoints
T8
Warm same-session ping
T9
Full end-to-end turn, with and without pre-warm
T10
Burst of 15 concurrent new sessions
T11
Streaming first-byte (reframed from an original WebSocket test; AgentCore has no client WebSocket)
T12
Streaming time-to-first-token
T13
Multi-turn, second turn on a warm session
T14
Pre-warm timing curve (0.5 to 10 s wait)
T15
Parallel vs sequential tool execution
T16
Integration Lambda hop vs direct invoke
The sections below report the ones that carried the findings.

Results

Lambda tool latency and Provisioned Concurrency

Config
p50
p95
Direct invoke, Provisioned Concurrency
113 ms
389 ms
Direct invoke,  $LATEST 
108 ms
170 ms
Via MCP Gateway
411 ms
n/a
Provisioned Concurrency made no measurable difference here. At p50 it matched  $LATEST  (113 vs 108 ms), and its p95 was higher (a first-call sampling artifact), not lower. The reason: the handler is a trivial DynamoDB read that stays warm between calls, so there is no cold start for PC to prevent. PC helps against cold starts under bursty or idle traffic, a condition this benchmark did not create at the tool layer. It is not a way to speed up an already-warm function.
The MCP Gateway added about 300 ms per call as measured from the client (per-call SigV4 requests, no HTTP keep-alive). From inside the runtime (in region, persistent MCP session) the same calls ran about 175 to 260 ms. Either way it adds measurable latency.

Cold start and pre-warming

Test
p50
Container cold start
1,019 ms
Code / ZIP cold start
~4,756 ms
Pre-warmed (2 s wait)
224 ms
Warm session ping
209 ms
Two findings: pre-warming works (a 0.5 s lead was enough; latency was flat from 0.5 s to 10 s of warm-up wait), and container deployment cold-started roughly 4 to 5 times faster than the code / ZIP path. In my packaging, the code path also required vendoring dependencies into the ZIP rather than relying on an install step, which is worth confirming against your own setup.

End to end agent turn (backend, no speech I/O)

Metric
p50
Full 3 tool turn (pre-warmed)
~4,800 ms
Full turn (no pre-warm)
~5,450 ms
Streaming TTFT
~4,400 ms
Burst, 15 concurrent (warm by latency threshold)
11 / 15
Integration Lambda hop overhead (see note)
+57 ms
The integration hop was measured with a small Lambda that itself calls  InvokeAgentRuntime , standing in for the Connect-integration Lambda. That is a simulation of the hop, not Amazon Connect.

Where the time goes (one representative turn, about 5 s)

LLM round 1 (plan) ~1.3 s
-> get_reservation ~0.2 s
LLM round 2 (plan) ~1.8 s
-> check_eligibility + get_refund_method ~0.3 s (parallel)
LLM round 3 (compose) ~1.5 s
About 4.6 s of this turn is LLM time across three sequential rounds; tools are about 0.5 s. Across all runs the pre-warmed turn p50 was about 4.8 s. Cold start, after pre-warming, is negligible, and streaming does not reduce TTFT here because the first token of the response cannot be produced until tool results return.

Strands vs LangGraph

Same agent both ways, from one container image switched by an  AGENT_FRAMEWORK  env var (identical Gateway, tools, model, prompt, and timing contract), state-matched with per-session memory, n=10 back to back.
Metric
Strands
LangGraph
Full E2E p50
4,805 ms
4,646 ms
Multi-turn (2nd turn) p50
4,408 ms
4,037 ms
Tool parallelism
10 / 10
10 / 10
Avg input tokens per turn
4,120
3,832
The two are about the same, within roughly 1 s of run to run variance. Both parallelize independent tools in every run. The only consistent difference is Strands using about 7 to 13 percent more tokens.

Cost at 1,000 calls / day

Using the published rates and measured token usage from the Assumptions section:
Item
$ / mo
Lambda Provisioned Concurrency (20 idle instances at 512 MB)
~$109
LLM tokens (Haiku 4.5, measured usage)
~$180
Lambda duration and requests
~$0.15
Total (excludes AgentCore Runtime consumption)
~$290
A note on token assumptions: a "500 in / 200 out tokens" estimate puts LLM cost at about $45/mo. The measured turns used about 4,110 input tokens, because multi-round tool calling replays the tool schemas and context every round, which makes it about $180/mo, roughly 4x higher. Measure token usage rather than assuming it.

What I would try next (not yet measured)

Since about 4.6 s of the turn is the LLM loop, that is where the time is:
  1. Reduce the number of LLM rounds. For a fixed workflow, the read tools can be fired in parallel on call arrival, followed by a single LLM call to phrase the answer, rather than letting the model plan each step.
  1. Prompt caching for the static system prompt and tool schemas, to lower per-round latency and token cost.
  1. Stream the response to TTS so speech can begin as phrasing starts.
  1. Co-locate or bypass the Gateway for latency-critical tools.

Takeaways

  • Pre-warming brings AgentCore session cold start to about 224 ms, under the 500 ms target.
  • Provisioned Concurrency did not help a warm, trivial handler. It is worth using where cold starts actually occur.
  • The MCP Gateway hop adds measurable latency, about 175 to 300 ms per call.
  • For a multi-tool agentic turn, the 2 s target is limited by LLM orchestration, not infrastructure. Reducing round trips matters more than adding infrastructure.
  • This is the backend only. ASR, TTS, and telephony are additive and not measured here, so a real voice deployment starts further from 2 s than these numbers show.
I'll be publishing the code sample soon so stayed tuned!
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article