Running Strands Decider 2B on Amazon Bedrock AgentCore: microVMs vs GPU Instances

Where a 2B decision model fits on AWS, what it costs, how often to trust it, and the traps that silently cost you a GPU.

Series: Decision models on AWS (3 articles)

1
Running Strands Decider 2B on Amazon Bedrock AgentCore: microVMs vs GPU Instances This article
2
Decision Models Explained: where they came from, how to read the leaderboards, and how to choose one on AWS
3
When does a decision model pay for itself?

Akshay_B

Published Oct 6, 2026

Last modified Oct 10, 2026

Strands Labs released Strands Decider 2B on 1 October. It's a small, open source decision model: you give it some state, a question and a fixed set of options, and it picks one, with a calibrated confidence. The community moved fast. Within three days there was a local setup guide and a careful Lambda experiment . Both are worth reading.

I build agents with startups on AWS, so my question was different: can Decider run next to my agents on Amazon Bedrock AgentCore, how fast is it there, and can I trust its answers? I deployed the same container to five hosts in eu-west-1. I ran the same 100 routing questions through each, with Claude Haiku 4.5 on Amazon Bedrock as the baseline.

What's a decision model?

Most of the small calls inside an agent aren't writing tasks. Which team should get this ticket? Is this request in scope? Should this tool call run? An LLM answers by generating text, which you then parse and hope is one of your options. A decision model does only that narrower job: you give it some state, one or more questions and a fixed set of options, and it returns one of your options with a probability. It can't write prose or hold a conversation; in exchange it always answers in your format, it's small enough to run in your own account, and it tells you how sure it is.

Strands Decider 2B is an open source decision model from Strands Labs (Apache-2.0, released 1 October 2026). It's Qwen3.5-2B-Base with the text-generation head replaced by a small pointer head that scores each option, fine-tuned with a LoRA adapter. It answers three question types (yes/no, pick one of several options, and scores on a scale) in a single pass, and its HTTP API uses the same request shape as TypeSafe's Jev.

The results first

There are two separate questions here: how fast is Decider on each host, and how accurate is it compared with an LLM.

1. Hosting: same model, different hosts

Every row runs the same Strands Decider 2B container. "Time per decision" is the model's own time (server p50).

Where Decider runs	Hardware	Time per decision	New session / cold start	Verdict
SageMaker AI real-time endpoint	ml.g6.xlarge, NVIDIA L4	131 ms	Always on	Fastest; newer GPU
AgentCore Runtime Instances	g4dn.xlarge, NVIDIA T4	475 ms	174–194 s per new session; 3.7 min for the first call, which launches the instance	Fast, agent-native, more setup
SageMaker AI real-time endpoint	ml.g4dn.xlarge, NVIDIA T4	483 ms	Always on	Same speed on the same GPU; simpler
AWS Lambda	arm64, 10 GB, 6 vCPU	4.3 s	39–49 s	Batch only
AWS Lambda MicroVMs	Graviton, 8 GB / 4 vCPU baseline, bursts to 16 vCPU	2.4 s (fp32), 2.8 s (bf16)	6–7 s to start from a snapshot; first decision in under 2 s on a brand-new MicroVM, with the model primed before the snapshot; ~2.4 s after resume	Fastest CPU option; scales from zero
AgentCore Runtime microVM (V1)	2 vCPU, 8 GB, CPU only	9.1 s	10–17 s	Works; too slow for a live agent

The answers were the same on every host: 72% routing accuracy on all 100 items for each GPU run. The CPU runs used the first 20 items, because each decision takes seconds, and they matched the GPU runs on those 20. Where you host it changes speed and cost, not the answers.

2. Accuracy: Decider against an LLM

The test set is 100 synthetic support messages for a fictional invoicing SaaS: 80 straightforward and 20 deliberately ambiguous. Each one asks two questions: which team should handle this message (billing, technical, sales, account, or unknown) and is it about the product at all (yes/no). The same two questions went to Decider and to Claude Haiku 4.5.

Model	Routing accuracy	Hard items	In scope
Strands Decider 2B (any host above)	72%	55%	95%
Claude Haiku 4.5 on Amazon Bedrock	93%	75%	97%
Decider first, Haiku only when Decider's confidence is below 0.6	90%	—	—

With the 0.6 rule, Decider answers 45% of requests on its own.

Four takeaways:

Decider needs a GPU for anything a user waits on. On CPU it takes seconds per decision on every platform I tried.
On AgentCore, that means Runtime Instances, not microVMs. The microVM path works on V1, at about 9 seconds a decision.
Check that you're actually on the GPU. On both AgentCore and SageMaker, my first GPU deployments silently ran on CPU, for two different reasons. You pay GPU prices for CPU speed, and nothing errors.
Out of the box, Haiku is more accurate. Decider's value today is a cheap first pass that knows when to hand off. The last two sections cover that, and when it's worth it.

Why run a decision model in your own account

Most of the startups I work with sell to banks, insurers or health providers, and the first questionnaire they get asks where data is processed. Running Decider in your own account and Region means:

prompts and decisions stay in your AWS account
access is controlled by IAM
there's no third-party key or shared rate limit
it's all on one bill

The original decision model, TypeSafe's Jev, is a hosted API. That's a good choice when residency isn't a constraint; This Builder post shows a nice pattern with it.

What makes this model awkward to host

Most of the problems below come back to three facts about the model:

4.6 GB of weights. Decider is a 67 MB LoRA adapter and a 4 MB pointer head on top of Qwen3.5-2B-Base, which ships as a single 4.55 GB bf16 file.
fp32 on CPU. On CPU, the package upcasts the model to float32 because, by its own measurement, bf16 is about 2× slower there. That takes about 7 GiB, and both copies briefly exist while it converts.
Slow fallback kernels. Qwen3.5 uses linear-attention layers (gated delta rule). Without the GPU kernels from flash-linear-attention, transformers falls back to a reference PyTorch implementation that it describes as "much slower". That's the main reason CPU numbers are in seconds.

I used one FastAPI app that speaks both the AgentCore and SageMaker AI container contracts: GET /ping and POST /invocations on port 8080. It copies the weights from Amazon S3 at startup, so the image stays small, and it reports which GPU served each request:

1
2
3
4
5
6
7
8
9
10
@app.get("/ping")
def ping():
    return {"status": "Healthy"}

@app.post("/invocations")
def invocations(body: dict):
    out = ENGINE.evaluate(SystemOneRequest(**body))   # strands_decider's engine
    payload = out.model_dump()
    payload["gpu"] = STATS.get("gpu")                 # e.g. "Tesla T4", or None on CPU
    return payload

AgentCore Runtime microVMs: what I found

This section is for anyone considering it. MicroVMs are the serverless side of AgentCore Runtime, and they have fixed limits:

2 vCPU and 8 GB per session
a 2 GB container image

Making it fit. The package's default fp32 upcast doesn't fit in 8 GB, so the process is killed while converting. I upcast one decoder layer at a time and kept the large embedding table in bf16. The answers matched the package default, and peak memory stayed under the limit. Copying 4.6 GB from S3 took about 51 seconds, and the container was healthy after 60–64 seconds.

V1. The same container on V1 created in about 70 seconds and worked. New sessions answered in 10–17 seconds; the one-minute model load didn't show up, so sessions appear to start on pre-initialised microVMs. Warm decisions took about 9.1 seconds (server p50, 20 items).

The guidance: a microVM-hosted Decider is too slow for routing or guardrails inside a live agent, where a user is waiting. It can make sense for background work: overnight triage, labelling agent transcripts after the fact, or async checks where seconds don't matter.

At the V1 list price ($0.0895 per vCPU-hour, $0.00945 per GB-hour), 9 seconds on 2 vCPU and 8 GB is roughly $0.64 per 1,000 decisions, before system overhead.

What would change this verdict: GPU-backed microVMs, more vCPUs per session, or the model on Bedrock.

Lambda MicroVMs: what I found

Lambda MicroVMs start each MicroVM from a snapshot of your already-running application, so the one-minute model load happens once, at build time. Sizes go up to 8 GB and 4 vCPU at baseline, and burst to 32 GB and 16 vCPU while busy.

Making it fit. A snapshot of the fp32 model (about 7.5 GB resident; 9.35 GB at peak) built, but MicroVMs restored from it terminated before starting, including with the weight files deleted. What worked was keeping the model in bf16 (under 1 GB resident) and serving it in bf16: on Graviton with PyTorch 2.14, bf16 gave the same answer as fp32 on all 100 test items, at 2.8 s per decision warm instead of 2.35 s. Upcasting to fp32 after the restore also works, but on my builds without priming it added about 46 seconds to the first request. (My first bf16 test took almost 8 minutes for one decision, but that was on a MicroVM still reading its memory from the snapshot.)

Prime the model before the snapshot. Lambda calls two hooks while it builds the image: /ready before it takes the snapshot, and /validate on a test MicroVM started from that snapshot (image build hooks ). Load the model and run a couple of real inferences before /ready returns 200, so the snapshot holds a model that has already answered. Then run one more inference in /validate: Lambda records which parts of the snapshot your application reads there and prefetches them when later MicroVMs start. Lambda polls both hooks, so return 503 straight away, do the work in a background thread, and return 200 when it's done. When I ran the inference inside the request, every build failed with "MicroVM terminated before it started". My build had also been copying a second checkpoint by mistake (9.3 GB of weights instead of 4.7 GB), and builds with the warm-up only passed once I fixed that too. Eric's post Baking a Fast Lambda MicroVM explains the hooks well.

Speed. From run-microvm to RUNNING took 6–7 seconds. With the model primed before the snapshot, the first request on a brand-new MicroVM took 1.6–1.9 seconds on four MicroVMs, the same as the second request on each (a short two-question request), so the first answer arrived 7–9 seconds after the MicroVM was started. Two earlier setups were much slower. With nothing in /validate, the first decision took about 7 minutes, because memory pages were read from the snapshot on first touch (the upcast took 402–429 s, against 3 s during the build), and the endpoint dropped that response. With an inference in /validate but none before the snapshot, it took about 48 seconds. Warm, decisions took 2.4 seconds in fp32 (server p50 2.35 s, p95 2.7 s, 100 items, 72% accuracy, the same as every other host) and 2.8 seconds in bf16. The burst helps: 3.1 s on 4 threads, 2.8 s on 8, 2.5 s on 16 (fp32). A suspended MicroVM resumed and answered in 2.4 s, with no penalty.

The guidance: the fastest CPU option I measured, and suspend/resume makes idle time cheap. It's still seconds per decision, so it's for background work, not a live agent. Serve the model in bf16, prime it before the snapshot, and run one inference in /validate. A fresh MicroVM then answers within about 8 seconds of being started, so you can scale from zero; keep one suspended if you need answers sooner.

At US East list prices, a 2.8-second bf16 decision bursting to 16 vCPU costs roughly $1.33 per 1,000 decisions. In fp32 (2.4 s) it would be about $1.10, and about $0.45 at the 4-vCPU baseline (3.1 s).

AgentCore Runtime Instances with a GPU

Runtime Instances run your agent on EC2 instances that AgentCore manages in your account, and they can have GPUs. You create a capacity provider (instance types, VPC, volumes, lifecycle) and point a runtime at it.

Getting it working took six fixes. Here they are in one place:

Pick instance types that exist in your Region. eu-west-1 has no EC2 g6 instances, though SageMaker AI does offer ml.g6. g5.xlarge on its own hit insufficient capacity that afternoon. Allowing g5.xlarge, g5.2xlarge and g4dn.xlarge let AgentCore place the runtime on a g4dn.xlarge (NVIDIA T4).
Don't set networkConfiguration on the runtime. The VPC comes from the capacity provider.
Keep the runtime's maxLifetime at or below the capacity provider's.
The 2 GB image limit applies to Instances too. CUDA PyTorch alone is about 3 GB. I shipped a 182 MB image and copy a prebuilt PyTorch from S3 at startup, built with the image's own Python version (3.12).
The runtime's network interface has no public IP. It needs VPC endpoints for Amazon ECR (api and dkr), Amazon S3 (gateway), Amazon CloudWatch Logs and AWS STS. Without them you get "The agent artifact could not be downloaded".
The host's NVIDIA driver is mounted at /usr/lib64. Debian-based images don't look there. More on that below.

Once it runs, it's fast. On the T4, a decision took 475 ms (server p50) over 100 items. Accuracy was 72%, the same as every other host.

Cold sessions are the trade-off.

The first call took 3.7 minutes: AgentCore launched the instance and pulled the image. The container then copied 4.6 GB of weights (21 s) and 4 GB of CUDA PyTorch (74 s), and was ready at 113 s.
Later new sessions took 174–194 seconds.
Sessions then stay warm for up to 14 days, and the instance idles down when you stop them.

Throughput. Within one warm session, requests are handled one at a time:

one caller gets 1.6 decisions a second
four or eight callers still get about 2.0 a second, with latency rising in step (p50 625 ms, 2.0 s, 3.9 s)

The stock strands-decider server handles one request at a time. Spreading load across sessions doesn't help either, because each new session costs about 3 minutes to start. The practical model is one warm session, reused. Batching would be the next step.

Cost. g4dn.xlarge is $0.587/hour on demand in eu-west-1, and AgentCore adds a 7.8% management fee for G-series instances, so about $0.63/hour. At about 2 decisions a second, that's roughly $0.09 per 1,000 decisions at full use, and $0.63 an hour whether you use it or not.

The SageMaker AI baseline, on the same GPU

To separate the platform from the hardware, I ran the same container on a SageMaker AI real-time endpoint on ml.g4dn.xlarge, the same Tesla T4 that AgentCore gave me. Client times include the round trip from London.

Same T4, same container	Server p50	Client p50	Client p95	Decisions per second, 1 / 4 / 8 callers
AgentCore Runtime Instances	475 ms	622 ms	854 ms	1.6 / 2.0 / 2.0
SageMaker AI endpoint	483 ms	510 ms	517 ms	2.0 / 2.0 / 2.0

The model time is the same, as it should be. The difference is the invoke path: about 110 ms more at the median through InvokeAgentRuntime, and a wider tail.

On newer hardware it's much faster. An ml.g6.xlarge (NVIDIA L4) answered in 131 ms, 3.7× faster than the T4, at about 6.7 decisions a second. If your Region has L4 or newer capacity, use it.

Price for the same GPU in eu-west-1:

SageMaker AI hosting on ml.g4dn.xlarge is $0.821/hour, about $0.11 per 1,000 decisions at full use. It's simpler and has a tighter latency tail.
AgentCore Instances is about $0.63/hour, about 23% cheaper per hour, and gives you sessions and co-located agents.

Plan for capacity. ml.g5.xlarge failed twice with InsufficientInstanceCapacity that afternoon, once after sitting in "Creating" for 75 minutes. Allowing several instance types, as an AgentCore capacity provider lets you, is the safer pattern.

Check that you're actually on the GPU

This cost me the most time, and nothing flags it unless you look. Both platforms gave me a GPU and a working endpoint, and both ran Decider on CPU at about 5.5 seconds a decision:

AgentCore Runtime Instances mounts the host's NVIDIA driver at /usr/lib64/libcuda.so.1. Debian-based images such as python:3.12-slim don't search there, so PyTorch reports "Found no NVIDIA driver" while /dev/nvidia0 is present. Fix: load the library by path before importing torch.
SageMaker AI on ml.g4dn defaults to a host driver that only supports CUDA 11.4, so a CUDA 12 PyTorch build reports "The NVIDIA driver on your system is too old". Fix: set the endpoint configuration's InferenceAmiVersion to al2023-ami-sagemaker-inference-gpu-4-1 (NVIDIA driver 580).

1
2
3
4
5
6
7
8
9
10
11
import ctypes, os

# AgentCore Runtime Instances: load the host driver before torch looks for it
for lib in ("libcuda.so.1", "libnvidia-ml.so.1"):
    path = f"/usr/lib64/{lib}"
    if os.path.exists(path):
        ctypes.CDLL(path, mode=ctypes.RTLD_GLOBAL)

import torch
assert torch.cuda.is_available(), "GPU deployment running on CPU"
print("Serving on", torch.cuda.get_device_name(0))

1
2
3
4
5
6
7
8
# SageMaker AI (CDK, Python): a newer GPU host image for ml.g4dn
sagemaker.CfnEndpointConfig.ProductionVariantProperty(
    variant_name="AllTraffic",
    model_name=model.attr_model_name,
    initial_instance_count=1,
    instance_type="ml.g4dn.xlarge",
    inference_ami_version="al2023-ami-sagemaker-inference-gpu-4-1",
)

The SageMaker AI monitoring tab shows it clearly: CPU near 350% and GPU at 0% for half an hour, until the fix lands.

My advice: log torch.cuda.get_device_name() at startup, return it with every response, and fail the health check if a GPU deployment has no GPU.

Against Claude Haiku as the decider

My test set is 100 synthetic support messages for a fictional invoicing SaaS:

80 straightforward, 20 deliberately ambiguous
each labelled with the team that should handle it (five options) and whether it's in scope for the product

The same two questions went to Decider and, as a forced tool call, to Claude Haiku 4.5 through the EU cross-Region inference profile.

Haiku is clearly better here: 93% against 72% on routing. Decider under-predicts "sales" and over-predicts "unknown", and option wording matters (rewording the "unknown" description moved accuracy by 2 points). One person labelled the set, so treat these as indicative, not a benchmark.

Decide cheap, escalate when unsure

Decider's confidence is informative:

When Decider was wrong, its average confidence was 0.43. When it was right, 0.64.
At 0.8 and above, it was right every time on this set.

So let it answer when it's confident and hand everything else to Haiku.

Confidence threshold	Decider answers alone	Overall accuracy
none	100%	72%
0.5	61%	87%
0.6	45%	90%
0.7	32%	92%
0.8	19%	93%

1
2
3
4
5
6
7
answer = decider(state, questions)       # ~0.5 s on a T4, ~0.13 s on an L4
route = answer["answers"]["route"]

if route["confidence"] >= 0.6:
    team = route["choice"]               # 45% of requests stop here
else:
    team = ask_haiku(state)              # the rest go to the LLM

Is it worth it? My Haiku calls averaged 878 input and 71 output tokens. At Haiku 4.5's list price of $1 and $5 per million input and output tokens, that's about $1.23 per 1,000 decisions on Haiku alone. With escalation:

Haiku handles 55% of requests, about $0.68 per 1,000.
A fully used T4 adds about $0.09–0.11 per 1,000.
The total is roughly $0.79 per 1,000: about 36% less, for 3 points of accuracy.

Below full use, the GPU's hourly cost dominates, so this pays off only at steady volume. The other wins are latency for the confident half, and that those requests never leave your account.

To be clear about where this leaves Decider: on a zero-shot routing task, a small decision model doesn't beat a strong LLM. Where I expect it to win is a narrow decision you make millions of times, with Decider tuned on your own labels and asked several questions in one pass. That's what Part 2 tests.

Which should you pick?

If you need…	Use
Decisions inside a live agent, with AgentCore sessions	AgentCore Runtime Instances with a GPU
The simplest fast endpoint	SageMaker AI real-time endpoint on a GPU
Background or batch decisions, scale to zero	Lambda MicroVMs (bf16, model primed before the snapshot), Lambda functions, or AgentCore microVM (V1); seconds per decision
The best accuracy with no tuning	Claude Haiku on Bedrock, or Decider first with Haiku as the fallback

Cost and clean-up

The whole investigation cost about $10:

roughly 6 GPU instance-hours across both platforms
a few hours of AWS CodeBuild
VPC endpoints while the Instances runtime was up

Everything was deployed with the AWS CDK. Delete GPU endpoints and capacity providers the same day you test.

What's next

Part 2 looks for the case where a decision model wins:

Decider fine-tuned on a few thousand of your own labels, on the same routing set
several questions per request against several LLM calls
batching on the GPU
a break-even calculator for when a decision model beats LLM tokens

# startups # strands-agents # amazon-bedrock-agentcore # amazon-sagemaker # agentic-ai

Series: Decision models on AWS (3 articles)

1
Running Strands Decider 2B on Amazon Bedrock AgentCore: microVMs vs GPU Instances This article
2
Decision Models Explained: where they came from, how to read the leaderboards, and how to choose one on AWS
3
When does a decision model pay for itself?

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Running Strands Decider 2B on Amazon Bedrock AgentCore: microVMs vs GPU Instances

Series: Decision models on AWS (3 articles)

Akshay_B

What's a decision model?

The results first

1. Hosting: same model, different hosts

2. Accuracy: Decider against an LLM

Why run a decision model in your own account

What makes this model awkward to host

AgentCore Runtime microVMs: what I found

Lambda MicroVMs: what I found

AgentCore Runtime Instances with a GPU

The SageMaker AI baseline, on the same GPU

Check that you're actually on the GPU

Against Claude Haiku as the decider

Decide cheap, escalate when unsure

Which should you pick?

Cost and clean-up

What's next

Series: Decision models on AWS (3 articles)