Deploying Strands Decider 2B On SageMaker AI
Host the open source Strands Decider 2B decision model on an Amazon SageMaker AI endpoint, from container build to your first request with calibrated probabilities.
Strands Labs recently announced Strands Decider 2B , an open source decision model optimized for experimentation. Decision models, also referred to as system one models, have been exciting developers since Jev was launched by TypeSafe AI, primarily because of their speed and cost effectiveness at close to frontier levels accuracy on many classification tasks.
Why Decision Models
Many tasks that we apply LLM reasoning to can be expressed as decision or classification tasks. A common example is choosing how to route a prompt, whether in the context of a RAG pipeline, or agentic coding. In RAG applications, you may implement a router to decide which tool or data source to send a query to, based on its content or complexity. In agentic coding and agentic knowledge work more broadly, we typically care about optimizing the cost per task, and may want to only utilize the most capable models, which are often the most expensive, on tasks that actually need that kind of capability. Decision models are able to give you a quick answer to your query, selecting from one of the choices of possible answers that you give it. More specifically, they assign probabilities to the different options, which allows your agent to know when to act with confidence, or escalate in the face of uncertainty. What is most exciting though is the fact that decision models are able to do many of these classification tasks at close to frontier levels of accuracy at a fraction of the cost and the time that frontier models may take.
Why Strands Decider 2B
Strands Decider 2B is a decision model geared towards experimentation. It is easy to get started with, and is open source so you can understand the building blocks and build on top of it. While you can run it locally during your development cycle, you can also deploy it on AWS infrastructure where you can test the effectiveness of introducing a decision model into your architecture in your actual test environment, iterate on it in an environment that mirrors production, and allow other developers to use it without each of them setting up a decision model themselves. You can do all of this within your AWS security boundary, without sending data to any provider or service outside of this perimeter. Strands Decider 2B also ships with a training recipe, so you can easily fine tune and adapt the model to your specific classification tasks. This post will walk you through the process of deploying Strands Decider 2B on SageMaker AI.
Deploying Strands Decider 2B
What We Will Build
In this walkthrough, we will deploy Strands Decider 2B to a SageMaker AI real-time inference endpoint, which can serve inference requests within your AWS account. We will use CodeBuild to build our model serving image and to download the model weights from Hugging Face, and ECR and S3 to store the container image and the model weights respectively. We can then send inference requests to the deployed model. Access is granted to invoke the model through IAM, so any application we give access to can invoke it.
AWS CodeBuild builds the serving image into Amazon ECR and copies the pinned weights from Hugging Face to Amazon S3. Amazon SageMaker AI combines the image and the weights into a real-time endpoint, and agents call it with InvokeEndpoint to get a choice with probabilities.
Setup
Strands Decider 2B is a 2 billion parameter model, small enough to run locally or on a modest cloud based instance. Here we choose to run it on an ml.g6.xlarge on SageMaker AI, which comes with an NVIDIA L4 GPU with 24 GB of memory. Strands Decider 2B consumes ~5GB of memory for the model weights, so the L4 GPU provides plenty of headroom. SageMaker AI provides a variety of GPU instances allowing you to rightsize your workload to achieve your target performance and meet your requirements. We also set up the supporting infrastructure and permissions to build our model serving container image and deploy it to SageMaker AI.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import json, time, zipfile, io
from pathlib import Path
import boto3
REGION = "us-east-2"
PREFIX = "strands-decider"
ENDPOINT = "strands-decider-v19"
INSTANCE = "ml.g6.xlarge"
IMAGE_TAG = "v19-0.1.0"
REPO = Path.cwd()
BUILD_DIR = REPO / "build"
(BUILD_DIR / "container").mkdir(parents=True, exist_ok=True)
(BUILD_DIR / "weights").mkdir(parents=True, exist_ok=True)
# Pinned Hugging Face revisions
CKPT_REV = "bb282d786bc251fd4e3068de3ada9ddbb38127cd" # StrandsAgents/strands-decider-2B-hobson-v19
BASE_REV = "b1485b2fa6dfa1287294f269f5fb618e03d52d7c" # Qwen/Qwen3.5-2B-Base
session = boto3.Session(region_name=REGION)
ACCT = session.client("sts").get_caller_identity()["Account"]
BUCKET = f"sagemaker-{REGION}-{ACCT}"
MODEL_S3 = f"s3://{BUCKET}/{PREFIX}/model/v19/"
iam, s3, ecr = session.client("iam"), session.client("s3"), session.client("ecr")
cb, sm = session.client("codebuild"), session.client("sagemaker")
smr, cw = session.client("sagemaker-runtime"), session.client("cloudwatch")
print("account", ACCT, "| bucket", BUCKET, "| model prefix", MODEL_S3)The Model Serving Container
Inference runs in PyTorch, so we must add it to our container image, in addition to the strands-decider package, alongside additional dependencies.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
%%writefile {BUILD_DIR}/container/Dockerfile
FROM python:3.12-slim
# Triton compiles a small C launcher at runtime, so the image needs a C compiler.
RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends build-essential \
&& rm -rf /var/lib/apt/lists/*
ENV PIP_NO_CACHE_DIR=1 PYTHONUNBUFFERED=1
RUN pip install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu126
# Versions from strands-decider docs/inference.md + training/README.md (CUDA setup).
RUN pip install \
strands-decider==0.1.0 \
transformers==5.17.0 \
peft==0.21.0 \
flash-linear-attention==0.5.2 \
fla-core==0.5.2 \
&& python -c "import importlib.util, torch, triton; assert importlib.util.find_spec('fla'), 'fla missing'; print('torch', torch.__version__, 'triton', triton.__version__)"
ENV HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TRITON_CACHE_DIR=/tmp/triton
WORKDIR /app
COPY batching.py serve.py ./
EXPOSE 8080
# SageMaker runs the container with the argument "serve".
ENTRYPOINT ["python", "/app/serve.py"]Models deployed to SageMaker AI need to implement
/invocations and /ping endpoints in order to be invocable. We therefore create a custom serve.py file to provide these paths. /invocations wraps the invocation call to the strands-decider server.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
async def invocations(request: Request) -> JSONResponse:
body = await request.body()
try:
req = SystemOneRequest.model_validate_json(body)
except ValidationError as exc:
return JSONResponse(status_code=400, content={"error": json.loads(exc.json())})
attrs = request.headers.get("x-amzn-sagemaker-custom-attributes", "")
try:
job = await run_in_threadpool(batcher.prepare, req,
force_unbatched="unbatched" in attrs)
payload = await batcher.submit(job)
except ValueError as exc: # caller error, e.g. options truncated away
return JSONResponse(status_code=422, content={"error": str(exc)})
return JSONResponse(content=payload)The default inference engine that is shipped with strands-decider handles one request at a time. Processing a request with a single question leaves the GPU mostly idle. In order to improve throughput for single-question requests, we can implement a custom batcher to batch requests before sending them to strands-decider, and use it like this:
1
2
job = await run_in_threadpool(batcher.prepare, req, force_unbatched="unbatched" in attrs)
payload = await batcher.submit(job)Once our container image spec and serving code is set up, we can build the model serving image. In this example, we utilize CodeBuild.
1
2
3
4
5
6
7
8
9
10
11
12
13
%%writefile {BUILD_DIR}/container/buildspec.yml
version: 0.2
phases:
pre_build:
commands:
- REGISTRY="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
- aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS --password-stdin "$REGISTRY"
build:
commands:
- docker build --platform linux/amd64 -t "${REGISTRY}/${ECR_REPO_NAME}:${IMAGE_TAG}" .
post_build:
commands:
- docker push "${REGISTRY}/${ECR_REPO_NAME}:${IMAGE_TAG}"We also need to copy the weights from HuggingFace into S3, where SageMaker AI can find them at deploy time and copy them into the model serving container. We utilize CodeBuild for this as well.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
%%writefile {BUILD_DIR}/weights/buildspec.yml
version: 0.2
phases:
install:
commands:
- pip install --quiet "huggingface_hub==1.5.0"
build:
commands:
- |
python3 - <<'PY'
import json, os, shutil
from huggingface_hub import snapshot_download
ckpt = snapshot_download("StrandsAgents/strands-decider-2B-hobson-v19",
revision=os.environ["CKPT_REV"], local_dir="/tmp/model/checkpoint")
base = snapshot_download("Qwen/Qwen3.5-2B-Base",
revision=os.environ["BASE_REV"], local_dir="/tmp/model/base")
for d in ("/tmp/model/checkpoint/.cache", "/tmp/model/base/.cache"):
shutil.rmtree(d, ignore_errors=True)
# The published v19 uses the legacy config name.
cfg_path = next(p for p in ("/tmp/model/checkpoint/strands_decider_config.json",
"/tmp/model/checkpoint/hobson_config.json") if os.path.exists(p))
shutil.copy(cfg_path, cfg_path + ".orig")
cfg = json.load(open(cfg_path))
assert cfg["base_model"] == "Qwen/Qwen3.5-2B-Base", cfg["base_model"]
cfg["base_model"] = "/opt/ml/model/base"
json.dump(cfg, open(cfg_path, "w"), indent=2)
print("rewrote", cfg_path)
PY
- du -sh /tmp/model/*
- aws s3 sync /tmp/model "s3://${BUCKET}/${PREFIX}/" --delete --only-show-errors
- aws s3 ls --recursive "s3://${BUCKET}/${PREFIX}/" | head -50Deployment
Once the model serving container image has been built, and the weights downloaded to S3, we can configure an endpoint and deploy the model to that endpoint.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
MODEL_NAME = f"{ENDPOINT}-batch"
sm.create_model(
ModelName=MODEL_NAME, ExecutionRoleArn=SM_ROLE,
PrimaryContainer={
"Image": IMAGE, "Mode": "SingleModel",
"ModelDataSource": {"S3DataSource": {"S3Uri": MODEL_S3, "S3DataType": "S3Prefix",
"CompressionType": "None"}},
"Environment": {"MODEL_NAME": "strands-decider-2B-hobson-v19",
"MAX_BATCH": "32", "MAX_BATCH_TOKENS": "16384", "MAX_DELAY_MS": "5"}})
sm.create_endpoint_config(
EndpointConfigName=MODEL_NAME,
ProductionVariants=[{"VariantName": "AllTraffic", "ModelName": MODEL_NAME,
"InstanceType": INSTANCE, "InitialInstanceCount": 1,
"ModelDataDownloadTimeoutInSeconds": 1200,
"ContainerStartupHealthCheckTimeoutInSeconds": 900,
"InferenceAmiVersion": "al2-ami-sagemaker-inference-gpu-3-1"}])
sm.create_endpoint(EndpointName=ENDPOINT, EndpointConfigName=MODEL_NAME)
t0 = time.time()
sm.get_waiter("endpoint_in_service").wait(EndpointName=ENDPOINT, WaiterConfig={"Delay": 30, "MaxAttempts": 60})
print(f"InService after {(time.time() - t0) / 60:.1f} min")Invoking The Model
Once the model is in service, we can invoke it, passing in a System One shaped request.
1
2
3
4
5
6
7
8
9
10
11
12
def ask(body, unbatched=False):
kw = {"CustomAttributes": "unbatched"} if unbatched else {}
r = smr.invoke_endpoint(EndpointName=ENDPOINT, ContentType="application/json",
Body=json.dumps(body), **kw)
return json.loads(r["Body"].read())
one = ask({"state": {"request": "Translate 'good morning' into Spanish."},
"questions": {"tier": {"type": "choice", "instructions": "How capable a model does `request` need?",
"criteria": {"basic": "A quick, routine reply.",
"standard": "Ordinary professional work needing some care.",
"advanced": "Expert-level or high-stakes work."}}}})
print(json.dumps(one, indent=2))This is the output of the request:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
"model": "strands-decider-2B-hobson-v19",
"answers": {
"tier": {
"type": "choice",
"choice": "basic",
"probabilities": {
"basic": 0.7548,
"standard": 0.1883,
"advanced": 0.0568
},
"confidence": 0.6322
}
},
"usage": {
"input_tokens": 101,
"output_tokens": 1
},
"batch_size": 1,
"latency_ms": 73.72
}With this setup, Strands Decider 2B answers a single question in about 70ms on the GPU. With request batching, one instance handled about 50 single-question requests per second, with a median latency under 150 ms, which was about 3.5 times the throughput of the default engine.
At this point, you can experiment with your own classification tasks, and can give your agents in your dev environments, and developers access to this model to securely test and experiment with its capabilities.
This walkthrough spins up real AWS resources. Be sure to continue on to the clean up section in the reference notebook to ensure that you don't keep incurring costs.
Conclusion
Decision models like Strands Decider 2B represent an exciting evolution in AI applications and architectures. For many routine decisions, decision models cut cost and latency with little loss in quality, which shifts the economics of agentic applications and opens new avenues for innovation.
In the next post, we will explore one such use case these models are able to unlock - prompt routing.
Code
The full code walkthrough can be found here - https://github.com/jonakola/strands-decider-sagemaker
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article