Inference on AWS: Precision, Memory and Six Backends Behind One Strands Agent
The companion to the talk: what bits per weight, memory bandwidth and the serving engine do to Gemma 4 on Bedrock, SageMaker, EC2 g6 and g5g, Inferentia2 and Trainium, with the cost of each, measured on one day.
This article is the written companion to the talk "Inference on AWS". One Strands agent sends the same question to six AWS backends: Amazon Bedrock, a SageMaker endpoint, vLLM on EC2 g6 and g5g, and a hand-ported Gemma 4 on Inferentia2 and Trainium. The talk adds two ideas that explain most of the results: the precision a model is stored and computed in, and the memory bandwidth of the chip that serves it.
The slides: https://docs.google.com/presentation/d/1Ax8KGF8T0EYffshY7WfUM61QcHmLk0q72EE7G_hIwgM/edit
What Is the Problem?
AWS sells a model as an API, as a managed endpoint, as a GPU by the hour and as two families of its own chips. The same small model, Gemma 4 E2B, runs at 36 tokens a second on one of them and 132 on another, and the hourly prices do not explain the gap. Three properties do: how many bits each weight takes, how fast the chip can read them, and how well the serving engine uses the chip.
Which Backends?
| Backend | What serves the model | Model |
|---|---|---|
| Amazon Bedrock | managed API | us.amazon.nova-micro-v1:0 |
SageMaker ml.g6.xlarge | AWS vLLM 0.31.0 container | xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 |
| EC2 g6.xlarge (NVIDIA L4) | vLLM 0.30 | the same repack |
| EC2 g5g.2xlarge (Graviton2 + T4G) | vLLM 0.27.2rc0, patched for sm_75 | google/gemma-4-E2B-it, fp16 |
| EC2 inf2.xlarge (Inferentia2) | hand-ported Gemma 4, xbill9/gemma4-optb:slim | Gemma 4 E2B, compiled for Neuron |
| EC2 trn1.2xlarge (Trainium) | the same image | the same build |
Step 1 — Precision: What You Trade
Every format below stores the same Gemma 4 E2B. Fewer bits per weight means less memory, less disk and less data to move for every token. It buys speed only where the chip multiplies that format natively, and it costs accuracy unless the weights were trained for it.
| Format | E2B weights in memory | Suite accuracy, 3,880 records |
|---|---|---|
| bf16 | 9.42 GiB | 0.683 |
| fp8, from Google's QAT weights | 7.07 GiB | 0.670 |
| int8, from Google's QAT weights | 7.07 GiB | 0.686 |
| int8, rounded from bf16 | — | 0.670 |
| int4, QAT | 6.32 GiB | 0.678 |
| int4 with int4 embedding tables | 2.85 GiB | 0.681 |
Two results carry the slide. The int4 builds made from Google's quantization-aware-trained (QAT) checkpoint stay within half a point of bf16, while an int8 build rounded from bf16 loses 1.3 points. And E2B's embedding tables are most of its weight: packing them to int4 takes the model from 6.32 GiB to 2.85 GiB.
Native support decides whether the smaller format is also faster:
| Chip | bf16 | fp8 | int8 | int4 |
|---|---|---|---|---|
| NVIDIA L4 (g6, SageMaker) | yes | yes | yes | widened to 16-bit |
| NVIDIA T4G (g5g) | emulated | no | yes | widened to 16-bit |
| NeuronCore-v2 (inf2, trn1) | yes | in hardware; compiler support pending | yes | widened to bf16, 27x per call |
Memory figures come from the E2B data-type sweep on one AMD MI300X (2026-10-07) and accuracy from the same checkpoint files on one TPU v5e (2026-09-29), both in
gemma4-dev.Step 2 — Memory and Bandwidth Across AWS
Generating a token reads every weight once, so memory bandwidth sets the ceiling on tokens per second for one request. The engine decides how close a backend gets to that ceiling.
| Backend | Accelerator | Memory | Bandwidth | tok/s, one request |
|---|---|---|---|---|
| g6.xlarge | NVIDIA L4 | 24 GB | 300 GB/s | 131.9 |
| SageMaker ml.g6.xlarge | NVIDIA L4 | 24 GB | 300 GB/s | 96.5 |
| g5g.2xlarge | NVIDIA T4G | 16 GB | 320 GB/s, 277 measured | 34.9 |
| inf2.xlarge | Inferentia2 | 2 x 16 GiB | ~817 GB/s | 36.5 |
| trn1.2xlarge | Trainium | 2 x 16 GiB | ~613 GB/s | 36.0 |
| Bedrock | Nova Micro | not disclosed | not disclosed | 130.4 |
Host RAM is 16 GiB on every instance except trn1.2xlarge, which has 32 GiB.
The Neuron chips have 2.0 to 2.7 times the L4's bandwidth and decode at 0.28 times its speed. Their 32 GB is two 16 GiB pools, one per NeuronCore, so a model either splits across both cores or fits in 16 GiB. AWS publishes Neuron bandwidth per instance; the per-chip figures here divide the largest instance's 9.8 TB/s by its chip count.
Tip: Quota Is Permission to Ask
Quota does not reserve a machine. On the rehearsal day the first zone tried in us-east-2 had no capacity for g6 or trn1, and on an earlier run inf2 found no capacity in us-east-2a with 80 vCPUs of quota free.
stage/up.sh tries every zone and then the next region, so a launch script that does the same saves a demo.Step 3 — Bedrock and SageMaker: the Managed Pair
Bedrock needs an IAM permission and an inference profile id, and bills per token: $0.035 per million input tokens and $0.14 per million output tokens for Nova Micro.
1
2
from strands.models import BedrockModel
model = BedrockModel(model_id="us.amazon.nova-micro-v1:0", region_name="us-east-1")SageMaker runs a container you choose on an instance you choose:
Three API calls build it,
create-model, create-endpoint-config and create-endpoint, and vLLM flags pass as SM_VLLM_* environment variables. On the rehearsal day the endpoint reached InService about 9 minutes after deploy on the AWS vLLM 0.31.0 container. It serves the same weights on the same GPU as the g6, and each request carries about 0.24 s of extra time through the managed path: 87 tokens took 0.902 s against 0.660 s at the g6's rate.Step 4 — vLLM on an L4
The g6.xlarge runs vLLM 0.30 with two flags that turn on tool calls:
1
--enable-auto-tool-choice --tool-call-parser gemma4| NVIDIA L4 | |
|---|---|
| Family | Ada Lovelace, compute capability 8.9 |
| Memory | 24 GB |
| Bandwidth | 300 GB/s |
| BF16 / FP16 tensor | 121 TFLOPS dense |
| FP8 tensor | 242.5 TFLOPS dense |
| Power | 72 W, single-slot PCIe |
It was ready 16 minutes after launch and decoded 131.9 tokens a second for one request, the fastest of the five instances.
Step 5 — Arm and CUDA, Then and Now
The g5g pairs a Graviton2 Arm host with an NVIDIA T4G. AWS calls G5g "the first Arm-based instances in a major cloud to feature GPU acceleration." Serving Gemma 4 on it needs vLLM built from source for sm_75 with a shared-memory patch, because the arm64 image ships without Turing kernels.
The same pairing reaches the desk this fall. NVIDIA announced RTX Spark on May 31, 2026:
| EC2 g5g | NVIDIA RTX Spark | |
|---|---|---|
| CPU | AWS Graviton2, Arm | 20-core NVIDIA Grace, Arm |
| GPU | NVIDIA T4G, Turing | Blackwell RTX, 6,144 CUDA cores |
| CPU to GPU | PCIe | NVLink-C2C |
| Memory | 16 GB GPU + 16 GB host | up to 128 GB unified |
The aarch64 and CUDA build work that the g5g required is the platform those machines run.
Step 6 — Why Neuron Is Hard to Serve On
The Inferentia2 and Trainium rows run the same hand-ported image, measured on
inf2.xlarge:- No vLLM release serves Gemma 4 on Neuron, so the engine is a hand port and ignores tool definitions.
- The graph compiles ahead of time for fixed shapes, 512 tokens in total and 128 for the prompt. A new shape costs about 3.9 s to compile.
- Memory is 2 x 16 GiB, one per NeuronCore. A single gather above roughly 4 to 5 GB returns zeros with no error.
- int4 weights get a dense bf16 copy before each matrix multiply: 27 times slower per call than plain bf16.
That is the 0.28x in Step 2: the chips have the bandwidth, and the software stack for this model has not caught up.
Step 7 — One Strands Agent, Every Backend
The agent asks how many of eleven ids are 10 or more and has a
count_ids tool that computes it. Every backend that supports tool calls, three runs each on the rehearsal day:1
2
3
4
bedrock run 1/3 ok quoted=8 filters=['id >= 10'] 1.19s
g6 run 2/3 ok quoted=8 filters=['id >= 10'] 0.4s
g5g run 1/3 ok quoted=8 filters=['id >= 10'] 2.35s
sagemaker run 1/3 ok quoted=8 filters=['id >= 10'] 0.63sAll twelve runs answered 8 and sent the filter
id >= 10. Grading the filter as well as the number matters: a wrong filter returns an exact number with a clean source.A second agent on Bedrock calls all six backends as tools and quotes a scoreboard sorted in code:
1
2
3
| vLLM on EC2 g6 (L4) | Paris | 0.151 | 13.2 |
| SageMaker endpoint (vLLM, L4) | Paris | 0.426 | 4.7 |
| Amazon Bedrock | Paris | 0.861 | 2.3 |Step 8 — What Each Backend Costs
cost.py pulls On-Demand prices from the AWS Pricing API and does the arithmetic:| Backend | $/hr | $/month 24x7 | $ per 1M output tokens, 1 request | $ per 1M output tokens, 16 parallel |
|---|---|---|---|---|
| EC2 g6.xlarge | 0.8048 | 588 | 1.69 | 0.150 |
| SageMaker ml.g6.xlarge | 1.1267 | 822 | 3.24 | 0.291 |
| EC2 g5g.2xlarge | 0.556 | 406 | 4.43 | not measured |
| EC2 inf2.xlarge | 0.7582 | 553 | 5.77 | not measured |
| EC2 trn1.2xlarge | 1.3438 | 981 | 10.37 | not measured |
| Bedrock, Nova Micro | — | 0 when idle | 0.14 | 0.14 |
A g6 running all month costs more than Bedrock Nova Micro until it averages 710 output tokens a second around the clock, at five input tokens per output token.
Tip: Debug With the Small Model
Every path in the talk was brought up with E2B first. Flags, port forwards and tool parsing fail the same way at any size, and E2B loads on Neuron in about 2 minutes against close to 4 for the 26B.
Compare and Contrast
| Backend | Precision served | tok/s | $ per 1M output tokens, 1 request | Tool calls |
|---|---|---|---|---|
| EC2 g6, vLLM | int4 weights, int4 tables | 131.9 | 1.69 | yes |
| Bedrock | managed | 130.4 | 0.14 | yes |
| SageMaker, vLLM | int4 weights, int4 tables | 96.5 | 3.24 | yes |
| EC2 g5g, vLLM | fp16 | 34.9 | 4.43 | yes |
| EC2 inf2, hand-ported | compiled for Neuron | 36.5 | 5.77 | no |
| EC2 trn1, hand-ported | compiled for Neuron | 36.0 | 10.37 | no |
So, Which One?
Start on Bedrock when one of its models does the job: nothing to run and the lowest cost per token here. Move to vLLM on a g6 for your own weights, quantized from a QAT checkpoint so the smaller format keeps its accuracy; it was the fastest and cheapest instance. SageMaker runs the same container as a managed endpoint for about twice the cost per token. The g5g is the Arm and CUDA path, and Inferentia2 and Trainium pay off once the serving stack supports the model.
Summary
The goal of this article was to explain the talk's results through precision, memory bandwidth and the serving engine. The key to the solution was one Strands agent driving six backends on the same day, with every count, ranking and price computed in code. The results were:
- int4 from QAT weights stays within half a point of bf16, and int4 embedding tables shrink E2B from 6.32 GiB to 2.85 GiB.
- vLLM on a g6 decoded fastest, 131.9 tok/s for one request at $1.69 per million output tokens, $0.150 at 16 in parallel.
- Every tool-calling backend picked the right filter, 12 of 12 runs.
- SageMaker adds about 0.24 s per request on the same GPU and costs about twice the g6 per token.
- Bandwidth sets a ceiling the engine has to reach: Neuron has 2.0 to 2.7 times the L4's bandwidth and decodes at 0.28 times its speed.
- No vLLM release serves Gemma 4 on Neuron, so those backends run a hand port with fixed shapes and no tool calls.
Scope: one rehearsal on 2026-10-12 in us-east-2, with the g5g and Bedrock in us-east-1, three timed requests per backend, client on a laptop through SSM port forwards. Memory per format comes from one MI300X and accuracy from one TPU v5e, both serving the same Gemma 4 E2B checkpoint files, and the 16-parallel prices from earlier compare.py runs of the E2B QAT build on a g6 and an ml.g6.xlarge. Bedrock serves Nova Micro, a different model from the Gemma 4 builds on the other rows. On-demand list prices from the AWS Pricing API on 2026-10-12.
The strategy for serving Gemma 4 on AWS from Bedrock to Trainium behind one Strands agent was validated with an incremental step by step approach.
References
- aws-inference-strands | GitHub
- Inference on AWS slides | Google Slides
- gemma4-dev | GitHub
- Strands Agents | GitHub
- NVIDIA L4 | NVIDIA
- NVIDIA and Microsoft Reinvent Windows PCs for the Age of Personal AI | NVIDIA Newsroom
- Amazon EC2 G5g instances | AWS
- Amazon EC2 Inf2 instances | AWS
- Amazon EC2 Trn1 instances | AWS
- AWS Neuron data types | AWS
- Amazon SageMaker AI real-time inference | AWS
- Amazon Bedrock pricing | AWS
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article