Deploy the Strands Decider model on Amazon SageMaker for fast, calibrated routing decisions

This post walks through deploying Strands Decider as a real-time Amazon SageMaker endpoint with a thin FastAPI container built via CodeBuild, validating the deployment with a golden-parity test, and wiring it into an LLM router that sends easy queries to a small model and hard ones to a large one

Meena Thandavarayan

Published Oct 6, 2026

Last modified Oct 6, 2026

Most of agent orchestration is a sequence of small decisions: which team owns this ticket, is this request urgent, should the agent call this tool, does this query need the big model or the small one. The default reflex is to hand each of these to a general-purpose LLM with a carefully worded prompt. That works, but you pay for it in latency, token cost, and output you then have to parse and hope is well-formed.

Strands Decider is a 2B-parameter model built specifically for that class of decision. Instead of generating free text, it answers structured, typed questions about a piece of input and returns calibrated probabilities. This post covers what the model does, when to reach for it, how to deploy it as a real-time Amazon SageMaker endpoint, how to validate the deployment, and a worked example: an LLM router that sends easy queries to a small model and hard ones to a large one.

What Strands Decider is

Strands Decider is a classification/decision model (the variant used here is strands-decider-2B-hobson-v19, built on a Qwen3.5-2B base with a LoRA adapter and a pointer head). You give it a state — the text to reason about — and one or more named questions. Each question is one of three types:

choice — pick one option from a labeled set. Returns the chosen label, a confidence, and a probability for every option.
noul — a yes/no judgment. Returns P(true).
score — an ordinal rating over an ordered list of levels (e.g. ["Low", "Medium", "High", "Critical"]).

Two properties make it a good fit for control-flow decisions. First, the output is structured by construction — you get probabilities and a chosen label, not a sentence you have to regex. Second, the probabilities are calibrated, so a confidence of 0.55 genuinely means the model is unsure, which lets you set thresholds and fall back to a human or a stronger model on low-confidence cases.

When to use it

Reach for the Decider when a decision is well-defined, happens on the hot path, and runs often enough that latency and cost matter:

Routing — ticket triage, intent routing, LLM tier selection (small vs. large).
Gating — allow / ask / block an agent's tool call before it executes.
Scoring — urgency, frustration, lead quality, severity.
Filtering — content screening (PII, prompt injection, abuse), judging whether an LLM answer passes or needs escalation.

It is not a generative model. If you need the model to write the summary, draft the reply, or produce the code, that is still a job for a generative LLM. The Decider's job is to decide what happens next, quickly and cheaply, and route the expensive generation only where it is warranted.

Architecture

The deployment has two build inputs — the model artifact and the serving container — that come together in a SageMaker real-time endpoint.

The serving container is deliberately thin. The SageMaker contract is just two HTTP routes on port 8080: GET /ping for health and POST /invocations for inference. A small FastAPI app served by Uvicorn implements both and delegates to the model's own inference engine:

1
2
3
4
5
6
7
8
9
10
# container/app.py (excerpt)
from strands_decider import server
from strands_decider.schema import SystemOneRequest

app = server.create_app(CHECKPOINT, device=DEVICE, model_name=MODEL_NAME)
_lock = threading.Lock()  # the engine isn't concurrency-safe; serialize calls

@app.post("/invocations")
def invocations(request: SystemOneRequest) -> JSONResponse:
    return JSONResponse(content=_evaluate(request))

A detail worth calling out: the container warms up the model (triton kernel compilation, CUDA init) during startup before /ping returns healthy, so cold-start cost is absorbed before SageMaker reports the endpoint InService. The first real request is already fast.

Deploying to SageMaker

The deployment is four scripts run from the project root, in order. They assume AWS credentials are configured, the model weights are present locally under model/, and (for staging) an S3 bucket named sagemaker-<region>-<account>.

1. Stage the model to S3

1
python stage_model.py

This packs the Decider checkpoint and its base model into model.tar.gz, rewrites the checkpoint config so the base-model path points at /opt/ml/model/base (where SageMaker extracts the archive inside the container), uploads to s3://sagemaker-<region>-<account>/strands-decider/model/model.tar.gz, and records the URI in build/model_uri.txt.

2. Build and push the serving image

1
python build_image.py          # tag defaults to "v1"

The image is built with AWS CodeBuild, not local Docker — no daemon or GPU needed on your machine, and the image is built in the same partition it is pushed to. The script idempotently creates the ECR repository, a CodeBuild role, and a CodeBuild project, zips the container context (Dockerfile, app.py, serve) to S3, runs the build (ECR login → docker build → docker push), and writes the image URI to build/image_uri.txt.

One build-time note from the Dockerfile: it installs the CUDA 12.6 build of PyTorch, because the default PyPI wheel targets CUDA 13 and needs a newer driver than the standard SageMaker GPU hosts carry.

3. Create the endpoint

1
python deploy.py

This reads both URIs, ensures a scoped SageMaker execution role exists, and uses the SageMaker SDK ModelBuilder to create the model and deploy the real-time endpoint strands-decider on an ml.g5.xlarge instance. Two settings matter:

1
2
3
4
5
6
7
8
9
10
builder.deploy(
    endpoint_name="strands-decider",
    initial_instance_count=1,
    instance_type="ml.g5.xlarge",
    container_timeout_in_seconds=900,
    # Default GPU host AMI ships a CUDA 11.4 driver; torch cu126 needs a 12.x driver.
    inference_ami_version="al2-ami-sagemaker-inference-gpu-3-1",
    update_endpoint=exists,
    wait=True,
)

The inference_ami_version pins a GPU host AMI with a CUDA 12.x driver — the piece that makes the cu126 torch build in the container actually run. The deploy waits until the endpoint reaches InService.

4. Tear down when done

An ml.g5.xlarge endpoint bills by the hour, so delete it when you are finished:

1
python destroy.py              # deletes endpoint, config, and model

Testing the deployment

Deploying without a correctness check is how silent regressions ship. The validation here compares the live endpoint against a CPU-generated golden reference and also checks latency, concurrency, and error handling.

1
python test_endpoint.py --endpoint strands-decider --calls 50

The golden file (tests/golden.json) is produced once on CPU with make_golden.py, running the same request set through the model locally. The endpoint test asserts that every answer matches within a 0.02 tolerance and that choice picks are identical, so a GPU/driver/version drift that changes outputs is caught immediately. It then measures serial latency (asserting server-side p50 under 300 ms), fires 4 concurrent clients × 10 calls to confirm the serialized engine stays correct under load, and sends malformed requests to confirm they are rejected rather than silently mishandled.

Running every bundled use case against the live endpoint produces a per-case accuracy line. These are smoke tests over 10–24 hand-labeled examples each — sanity checks, not benchmarks — but they confirm the full path works end to end:

1
2
3
4
5
6
7
8
01_ticket_triage       20/20  100%
02_intent_router       12/12  100%
03_incident_triage     17/20   85%
04_tool_call_gate       8/10   80%
05_llm_router          20/24   83%
...
09_lead_scoring        10/10  100%
10_llm_judge_filter    10/10  100%

Worked example: an LLM router

A common, high-value use is routing queries to the right model tier: cheap model for the easy stuff, expensive model only when the request actually needs reasoning. The Decider makes this a single structured call. Define the decision once as two questions — a choice over tiers and a noul for whether retrieval is needed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from common import ask, choice, yes_no

QUESTIONS = {
    "tier": choice(
        "Which model tier should answer this request?",
        {
            "small": "Simple facts, rewriting, translation, chit-chat, short lookups",
            "large": "Multi-step reasoning, code, system design, analysis of several documents",
        },
    ),
    "needs_search": yes_no(
        "Does answering require fresh or private information that must be looked up?"
    ),
}

The ask helper wraps a single endpoint invocation — one call answers both questions about the query:

1
2
3
4
5
6
7
8
# usecases/common.py (excerpt)
def ask(state, questions):
    resp = _runtime.invoke_endpoint(
        EndpointName=ENDPOINT,
        ContentType="application/json",
        Body=json.dumps({"state": state, "questions": questions}),
    )
    return json.loads(resp["Body"].read())["answers"]

Routing a query is then just reading the structured answer:

1
2
3
4
5
6
7
8
9
10
11
query = "Design a sharded database schema for a multi-tenant SaaS and explain the tradeoffs"
a = ask(query, QUESTIONS)

tier = a["tier"]["choice"]                 # "large"
confidence = a["tier"]["confidence"]       # e.g. 0.91
needs_search = a["needs_search"]["noul"] > 0.5  # False

if tier == "large" or confidence < 0.6:
    model = "claude-sonnet"   # reasoning-grade model, or hedge on low confidence
else:
    model = "claude-haiku"    # cheap model for easy queries

Because confidence is calibrated, the confidence < 0.6 clause is a real hedge: borderline queries get escalated to the stronger model instead of being miscategorized. The needs_search flag independently drives whether you attach a retrieval step, so a simple-but-fresh query like "What's the current price of AAPL?" routes to the small model and triggers a lookup.

Here is the shape of the decision across a representative query set — easy requests go small, reasoning-heavy ones go large, and the retrieval flag fires on the queries that need current or private data:

Query	tier	needs_search
Rewrite this sentence to be more formal	small	no
What's the capital of Australia?	small	no
Design a sharded multi-tenant DB schema and explain tradeoffs	large	no
What were the top news headlines today?	small	yes
Summarize our company's refund policy for a customer	small	yes
Compare 2024 vs 2025 EU AI Act obligations and draft a plan	large	yes
Debug this stack trace, fix spans three services	large	no

One endpoint call per query, structured output, calibrated confidence for the fallback — and the expensive model only sees the queries that justify it.

Wrapping up

Strands Decider fills a specific gap: the fast, repeated, structured decisions that sit between the generative steps of an agent or pipeline. Deploying it on SageMaker is a thin FastAPI container over the model's own engine, built with CodeBuild and deployed with ModelBuilder, with a golden-parity test that catches environment drift before it reaches production. The LLM router shows the pattern — define the decision as typed questions once, make a single call, and branch on a calibrated answer — and it generalizes directly to triage, gating, scoring, and filtering.

Full source, the container definition, the deploy scripts, and all eleven use-case demos are in the https://github.com/frugalagents/blueprints/tree/main/strands_decider_on_sagemaker

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Deploy the Strands Decider model on Amazon SageMaker for fast, calibrated routing decisions

Meena Thandavarayan

What Strands Decider is

When to use it

Architecture

Deploying to SageMaker

1. Stage the model to S3

2. Build and push the serving image

3. Create the endpoint

4. Tear down when done

Testing the deployment

Worked example: an LLM router

Wrapping up