Quick Testing Jev - LLM, a decision model, or your own trained classifier
A quick evidence-based comparison of Jev, LLMs, and custom classifiers for text classification. Using a spam dataset, it compares accuracy, precision, recall, costs, and behavior under distribution shift, then provides source-aligned replication steps for Jev through OpenRouter, LLMs on Bedrock, and TF-IDF/DistilBERT jobs on SageMaker.
Series: Testing Decision Models (2 articles)
- 1Quick Testing Jev - LLM, a decision model, or your own trained classifier This article
If you have a routing or classification step in your pipeline (is this ticket urgent, which agent should handle this, is this content safe to show), there are three fundamentally different ways to build it. You can reuse a general-purpose small LLM and prompt it to output a label, or you can call a purpose-built decision model, a new category represented right now by TypeSafe's Jev, or you can train a small classifier on your own labeled data and host it yourself. This post walks through the difference, what each costs, and how to test all three against your own data instead of a vendor's benchmark.
The three approaches
A small general-purpose LLM doing the classification
In this case, you would send the LLM an input plus a prompt asking for a label. It will generate that label as text. Claude Haiku and OpenAI's GPT-5.6 Luna are both examples, but the category is any small, cheap, fast model repurposed for a task it wasn't built for. The appeal is that it already understands language and context. It was trained on everything, so it needs zero labeled examples from you and can reason through inputs it's never seen phrased that way before. This is a great example of asking a generalist to do a specialist's job. Some of these models (Luna included) spend part of their output budget on internal reasoning before emitting the actual label, which adds a cost and latency wrinkle that's easy to miss if you only test with a demo prompt.
A purpose-built decision model
Jev, from TypeSafe, is the first model in this category. Instead of generating text, you send it a block of state (text, JSON, whatever) plus a typed question, either a choice from a fixed set, a score on a scale, or a yes/no probability, and it returns that structured decision directly. TypeSafe calls it a "System One" model. The pitch is that classification and routing don't need a full language model's generative capability. They need judgment, so a model trained specifically to output calibrated decisions should beat both a repurposed LLM and a from-scratch classifier at the thing it's actually built for.
A classifier you train yourself
TF-IDF plus logistic regression, or a fine-tuned small encoder like DistilBERT, trained on your own labeled data is the classification approach used with my testing. You can run it as a one-off batch job or deploy it to a real-time endpoint with SageMaker Processing and Training jobs. This is the oldest of the three approaches and the only one where you're not depending on someone else's model at all. It needs labeled data upfront and retraining when your task drifts, but everything about its behavior is yours to inspect and control.
The question this article is trying to test is "Does a model purpose-built for decisions (Jev) actually beat a repurposed generalist (a small LLM) at classification, and does either beat the boring option of just training your own model on your own data?"
What "training your own classifier" actually means
A classic classifier is a small, dedicated program that learns to sort things into categories from examples, rather than being told the rules. You give it a pile of labeled examples (this email is spam, this one isn't) and mathematically it works out by patterns separated into the two groups. Once it's trained, you hand it something new and it guesses a label based on those patterns.
The specific version we use here is TF-IDF plus logistic regression which is a pairing that's been a workhorse for text classification for over two decades. We can break this into two halves.
TF-IDF (term frequency-inverse document frequency) turns text into numbers a program can compare. It counts how often each word appears in a given message, then discounts words that show up everywhere (like "the" or "and", which tell you nothing) and boosts words that are rare across the whole dataset but common in this one message (like "unclaimed prize" or "wire transfer", which tell you a lot). The output is a big list of numbers, one per word, that captures which words matter for this particular piece of text.
Logistic regression is the part where you can feed it that list of numbers for a bunch of labeled examples, and it learns a weight for each word. Each weight captures whether the word pushes toward spam or away from it, and by how much. Add up the weighted push from every word in a new message, and it outputs a probability (say, 87% likely spam).
This is why the classifier's cost line in the results table below is a fraction of a cent. Once trained, running it costs whatever a few milliseconds of basic arithmetic costs on a cheap server. The Pricing section below puts real numbers on that gap against Jev and a small LLM per call. It's also why it needs labeled data upfront and can't handle a case it's never seen the pattern for. Unlike an LLM, it has no broader understanding to fall back on. It only knows what the training examples taught it.
Where each one wins and loses
The LLM approach wins on flexibility. It needs no training data, handles novel phrasing and edge cases reasonably well because it's a full language model with broad world knowledge, and it's usually already in your stack. The cost is that you're paying for generation (input and output tokens, plus latency) to do something that doesn't need generation, and there's some chance of an off-schema or malformed response if your prompt isn't tight.
Jev's advantage, based on independent testing, is distribution shift, where it retains recall on patterns that were not in the data it was calibrated with. One independent reproduction (a spam/phishing classifier eval) found Jev's zero-shot recall on newly invented phishing patterns at 95%, against 75% for a classifier trained before those patterns existed. TypeSafe's own disclosed benchmark numbers show 67.8% accuracy for Jev against 74.1% for the best comparator LLM on the same tasks (AInvest ). A separate independent benchmark across four datasets, including a phishing email set, found trained local classifiers winning on raw accuracy in most cases, with Jev's advantage showing up in zero-shot flexibility rather than accuracy (MindStudio ), meaning if your data isn't drifting, the boring option might just be better and cheaper than the purpose-built one.
A classifier you train yourself keeps costs low and makes its behavior inspectable once it's built. You own the model, its weights, and its exact decision boundary. It is the cheapest and fastest of the three per call, but it learns through labeled data and degrades quietly when your input distribution shifts, since nothing tells you the world moved out from under what it learned.
"Trained classifier" here covers both the TF-IDF plus logistic regression approach and a small encoder like DistilBERT fine-tuned on your labeled data. A fine-tuned open-weight LLM (Llama, Mistral, and similar) starts with broader pretraining than either of those approaches. It is larger and more expensive to host and fine-tune.
This article sticks to the smaller, cheaper end of that spectrum because it's the more common starting point for a routing or classification step.
Pricing
A small LLM like Haiku 4.5 on Bedrock runs $1 per million input tokens and $5 per million output tokens, with cached input reads around $0.10-$0.11 per million. GPT-5.6 Luna, also on Bedrock, is roughly 5x cheaper at $0.20 per million input and $1.20 per million output. Current rates for both are on the Amazon Bedrock pricing page .
For a short classification prompt returning one label, almost all your cost is input tokens on a normal generation model, but on a reasoning-family model like Luna, output cost can run higher than you'd expect from a one-word answer, since some of that output budget goes to reasoning tokens the model spends before it commits to a label.
At the September 29, 2026 availability check, OpenRouter's active typesafe/jev-1.13 endpoint reported a prompt price of $0.042 per million tokens and a completion price of zero. The saved test used each response's usage.cost field and totaled $0.064 across 2,123 successful calls. Pricing and endpoint availability can change, so check the Jev endpoint detail before budgeting or rerunning the test.
Your own classifier's cost is mostly upfront by labeling, plus training time, which for TF-IDF/logistic regression is seconds on a laptop-sized job. A SageMaker real-time endpoint is a separate production choice and is billed per instance-hour for whatever instance size you pick. AWS's own worked example puts a small production endpoint around $306/month, though your real number depends on instance type, region, and whether you use SageMaker Savings Plans.
This test did not deploy an endpoint, so its SageMaker figures below are one-off job costs rather than measured endpoint-inference costs. Use the AWS Pricing Calculator with your actual instance choice for a number you can budget against rather than relying on this estimate.
What I found
I ran this test on a public, dated spam corpus with 5,052 ham and 2,399 spam messages with real Date: headers. I split it into a training set, an in-distribution test set, and a later-dated distribution-shift test set. I tested five systems which include a TF-IDF plus logistic regression classifier, a DistilBERT fine-tune, Jev, Claude Haiku 4.5, and GPT-5.6 Luna.
The Jev in-distribution run had two API errors, so its in-distribution metrics and API cost cover 1,244 successful calls rather than 1,246. Every other row covers all 1,246 in-distribution messages and all 879 shift-slice messages.
Reading the metrics: what precision, recall, and F1 actually mean
The results table below leans on four numbers per system per test slice, using spam detection as the concrete case.
The results table below leans on four numbers per system per test slice, using spam detection as the concrete case.
- Accuracy is the simplest and the easiest to misread: what fraction of all predictions were correct, both spam and non-spam combined. The catch is that accuracy can look great while a system is useless. If 95% of your messages aren't spam, a system that lazily labels everything "not spam" scores 95% accuracy while catching zero actual spam. The imbalance in the shift slice in this test (40 spam out of 879 messages) needs the next two metrics to say anything real.
- Precision answers: of everything this system flagged as spam, how much of it was actually spam? Low precision means a lot of false alarms, real messages wrongly caught. This metric tells you how much your users will get annoyed by things wrongly sent to spam.
Recall answers that mirror question, "of all the actual spam that existed, how much did this system catch?" Low recall means real spam is slipping through. This metric tells you how much junk actually reaches the inbox. - F1 is a single number that balances precision and recall together (specifically, their harmonic mean), useful when you want one score to compare systems by instead of eyeballing two numbers that can pull in opposite directions. A system with 99% precision and 10% recall (catches almost nothing, but never wrong when it does) and a system with 40% precision and 95% recall (catches almost everything, but cries wolf constantly) can land on very different F1 scores despite neither number alone telling the whole story.
- Precision and recall usually trade off against each other: a system that flags anything remotely suspicious catches more real spam (higher recall) but also flags more clean messages by mistake (lower precision), and a cautious system that only flags the obvious cases does the opposite. Which one matters more depends on your actual cost of being wrong. A spam filter that's too aggressive costs you real emails and a content moderation filter that's too lenient costs you exposure to bad content. Neither number is "the" right one to optimize. It depends on which mistake hurts more in your specific pipeline.
| System | In-dist accuracy | In-dist precision | In-dist recall | In-dist F1 | Shift accuracy | Shift precision | Shift recall | Shift F1 | Estimated one-off job cost | API cost in this test |
|---|---|---|---|---|---|---|---|---|---|---|
| Classifier, SageMaker Processing job (TF-IDF + LR) | 99.04% | 99.80% | 97.89% | 98.84% | 98.52% | 81.40% | 87.50% | 84.34% | ~$0.006 | — |
| DistilBERT fine-tune, SageMaker Training job | 99.76% | 99.81% | 99.62% | 99.71% | 97.84% | 67.80% | 100.00% | 80.81% | ~$0.088 | — |
| Jev | 98.71% | 99.41% | 97.50% | 98.45% | 98.98% | 86.05% | 92.50% | 89.16% | — | $0.064 (2,123 successful calls) |
| GPT-5.6 Luna | 98.39% | 97.90% | 98.27% | 98.08% | 97.72% | 67.24% | 97.50% | 79.59% | — | $0.208 (2,125 calls) |
| Haiku 4.5 | 97.67% | 98.62% | 95.78% | 97.18% | 97.95% | 72.00% | 90.00% | 80.00% | — | $1.006 (2,125 calls) |
Every prediction a classifier makes lands in one of four buckets. It correctly flags real spam (a true positive), correctly leaves a real non-spam message alone (a true negative), wrongly flags a clean message as spam (a false positive, the annoying kind of mistake, a real email lost to your spam folder), or wrongly lets real spam through (a false negative, the dangerous kind, spam that reaches the inbox). Every metric above is just a different way of counting those four buckets.
The custom-image Processing job completed in 177.9 seconds on SageMaker with a ml.m5.large. The DistilBERT Training job recorded 430 billable seconds on ml.g4dn.xlarge. I did not measure real-time endpoint cost or endpoint latency. I also ran the TF-IDF classifier locally with the same code and data, and it reproduced the SageMaker custom-image result exactly with the same accuracy, F1, and confusion matrix on both slices.
Key findings
Jev had the highest precision on the shifted slice. Jev reached 86.05% precision on the later-dated data. The custom-image TF-IDF classifier reached 81.40%, Haiku reached 72.00%, Luna reached 67.24%, and the DistilBERT fine-tune reached 67.80%. Jev still caught 37 of the 40 spam messages, for 92.50% recall.
The classic classifier was stronger on the in-distribution slice. The SageMaker TF-IDF classifier reached 99.04% accuracy and 99.80% precision, compared with Jev's 98.71% accuracy and 99.41% precision. Its one-off Processing job cost about $0.006 in this test. That is not a production endpoint comparison, because no endpoint was deployed, but it shows what the batch test itself cost.
Fine-tuning maximized recall but raised more false alarms after the date split. DistilBERT caught all 40 shifted spam messages, but its 67.80% precision came from 19 false positives. The TF-IDF classifier had 8 false positives and Jev had 6 on the same slice.
Is Jev worth it, and for what
Jev was a strong zero-shot option in this test. It cost $0.064 for 2,123 successful API calls and had the best shifted-slice precision, 86.05%. Its precision lead over the tested SageMaker classifier was 4.65 percentage points on the later-dated data.
For a stable task with labeled data, the TF-IDF classifier produced better in-distribution accuracy and precision in this run, and the one-off batch job was cheaper. For a task where labels are scarce or the input changes over time, Jev's stronger shifted-slice precision makes the added API dependency easier to justify. Production cost still depends on the endpoint, batch, or API deployment model you choose.
There's now a Part 2 . This test used one spam task with two possible answers, so I widened it to eleven models on 1,000 DecisionBench cases across software tool routing and entity alignment, including Jev, GPT-6 Luna on Amazon Bedrock, GLM-5.3-Flash, and open decision models hosted on SageMaker endpoints, and scored them on accuracy, latency, and cost per 1,000 decisions. The accuracy race is close at the top, so cost and latency end up deciding it. I also fine-tuned the weakest model and moved it from 54.8% to 70.2%. Read it here: Testing decision models, Part 2 .
Replicate this test yourself
Don't take any vendor's number as ground truth, including this article's. Here's the exact setup to run all three approaches against your own labeled data.
Prerequisites
- An AWS account with SageMaker access and an execution role that can read and write S3 objects and publish CloudWatch Logs.
- Bedrock model access enabled for Claude Haiku 4.5 in your account's console under Bedrock > Model access.
- An OpenRouter account and an OPENROUTER_API_KEY environment variable for Jev.
docker or finch to build the custom container image. - Python 3.12, which is the launcher version used for this test, with boto3, sagemaker<3, requests, and scikit-learn. Install python-dotenv only if you prefer to load OPENROUTER_API_KEY from a local .env file.
- A labeled text-classification dataset with timestamps. This walkthrough uses the public Apache SpamAssassin corpus, which includes genuine Date: headers.
- Step 1: Build the dataset with a temporal split
Pull the dataset and split it three ways: a training set, an in-distribution test set held out from the same time window, and a later-dated distribution-shift test set. Use human labels throughout, never a model's own output as ground truth.
The tested scripts expect this layout:
1
2
3
4
data/
├── train.jsonl
├── in_distribution_test.jsonl
└── distribution_shift_test.jsonlEach JSONL record needs text and label fields. The test also carried an id field for error reporting. Labels were exactly spam and not_spam:
1
{"id": "message-001", "text": "Example email body", "label": "spam"}Replication path A: Train your own classifier on SageMaker
Amazon SageMaker runs machine-learning code on managed infrastructure. I used a Processing job for the TF-IDF classifier and a Training job for DistilBERT. Both jobs stopped after scoring the dataset. Neither result came from a persistent endpoint.
Amazon SageMaker runs machine-learning code on managed infrastructure. I used a Processing job for the TF-IDF classifier and a Training job for DistilBERT. Both jobs stopped after scoring the dataset. Neither result came from a persistent endpoint.
- Build a custom container image. SageMaker's built-in scikit-learn container ships an older library version (1.2.x) on Python 3.9.
1
2
3
4
5
6
7
# Dockerfile.sklearn191
FROM python:3.12-slim
RUN pip install --no-cache-dir scikit-learn==1.9.1
WORKDIR /opt/ml/processingcode block content# Dockerfile.sklearn191
FROM python:3.12-slim
RUN pip install --no-cache-dir scikit-learn==1.9.1
WORKDIR /opt/ml/processingNo ENTRYPOINT needed. The SageMaker SDK's ScriptProcessor sets the container's entrypoint itself from the script you hand it.
- Build, push, and launch:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
export AWS_REGION=us-east-1
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export IMAGE_REPOSITORY=jev-classifier-sklearn191
export IMAGE_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${IMAGE_REPOSITORY}:latest"
# Run this first only if Finch is installed but its VM is not already running.
finch vm start
# One-time repository creation.
aws ecr create-repository \
--repository-name "$IMAGE_REPOSITORY" \
--region "$AWS_REGION"
# The SageMaker instance used in the test is x86_64.
finch build --platform linux/amd64 \
-f Dockerfile.sklearn191 \
-t "${IMAGE_REPOSITORY}:latest" .
aws ecr get-login-password --region "$AWS_REGION" \
| finch login --username AWS --password-stdin \
"${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
finch tag "${IMAGE_REPOSITORY}:latest" "$IMAGE_URI"
finch push "$IMAGE_URI"Before running the launcher, pass the image URI and your SageMaker execution role without putting either value in source control:
1
2
export CLASSIFIER_IMAGE_URI="$IMAGE_URI"
export SAGEMAKER_EXECUTION_ROLE_ARN="arn:aws:iam::<account-id>:role/<execution-role>"1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# run_sagemaker_processing_custom_image.py
import os
import shutil
import tempfile
from pathlib import Path
import boto3
import sagemaker
from sagemaker.processing import ProcessingInput, ProcessingOutput, ScriptProcessor
REGION = "us-east-1"
PROFILE = os.getenv("AWS_PROFILE")
ROLE_ARN = os.environ["SAGEMAKER_EXECUTION_ROLE_ARN"]
IMAGE_URI = os.environ["CLASSIFIER_IMAGE_URI"]
BASE_DIR = Path(__file__).parent
DATA_DIR = BASE_DIR / "data"
S3_PREFIX = "jev-classifier-test-v191-custom"
FILES = ["train.jsonl", "in_distribution_test.jsonl", "distribution_shift_test.jsonl"]
session_args = {"region_name": REGION}
if PROFILE:
session_args["profile_name"] = PROFILE
boto_session = boto3.Session(**session_args)
sm_session = sagemaker.Session(boto_session=boto_session)
bucket = sm_session.default_bucket()
stage_dir = Path(tempfile.mkdtemp(prefix="jev_sm_stage_"))
for name in FILES:
shutil.copy(DATA_DIR / name, stage_dir / name)
processor = ScriptProcessor(
image_uri=IMAGE_URI,
command=["python3"],
role=ROLE_ARN,
instance_type="ml.m5.large",
instance_count=1,
base_job_name="jev-classifier-sklearn191-custom",
sagemaker_session=sm_session,
)
processor.run(
code=str(BASE_DIR / "sagemaker_processing_script_custom.py"),
inputs=[ProcessingInput(source=str(stage_dir), destination="/opt/ml/processing/input")],
outputs=[ProcessingOutput(
source="/opt/ml/processing/output",
destination=f"s3://{bucket}/{S3_PREFIX}/output",
)],
wait=True,
logs=True,
)
boto_session.client("s3").download_file(
bucket,
f"{S3_PREFIX}/output/results_classifier_sagemaker.json",
str(BASE_DIR / "results_classifier_sagemaker_custom_v191.json"),
)The container-side script reads the three staged files from /opt/ml/processing/input and writes results_classifier_sagemaker.json to /opt/ml/processing/output. Its tested model configuration was:
1
2
3
4
5
6
7
8
9
10
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
vectorizer = TfidfVectorizer(
max_features=20_000,
ngram_range=(1, 2),
min_df=2,
stop_words="english",
)
clf = LogisticRegression(max_iter=1000, C=1.0)docker works identically if that's what you have running. The completed Processing job took 177.9 seconds after SageMaker created it. Image build and push time varies by local machine and registry state, so it is not included in that measurement.
- Optional: fine-tune a stronger baseline. If the TF-IDF baseline isn't strong enough, fine-tune a small encoder (DistilBERT-class) via a SageMaker Training job instead of a Processing job:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import shutil
import tempfile
from pathlib import Path
from sagemaker.huggingface import HuggingFace
# BASE_DIR, DATA_DIR, boto_session, sm_session, and bucket come from the
# launcher setup above.
stage_root = Path(tempfile.mkdtemp(prefix="jev_distilbert_stage_"))
train_stage = stage_root / "train"
test_stage = stage_root / "test"
train_stage.mkdir()
test_stage.mkdir()
shutil.copy(DATA_DIR / "train.jsonl", train_stage / "train.jsonl")
shutil.copy(DATA_DIR / "in_distribution_test.jsonl", test_stage / "in_distribution_test.jsonl")
shutil.copy(DATA_DIR / "distribution_shift_test.jsonl", test_stage / "distribution_shift_test.jsonl")
train_s3_uri = sm_session.upload_data(str(train_stage), bucket=bucket, key_prefix="jev-distilbert-test/train")
test_s3_uri = sm_session.upload_data(str(test_stage), bucket=bucket, key_prefix="jev-distilbert-test/test")
code_stage_dir = Path(tempfile.mkdtemp(prefix="jev_distilbert_code_"))
shutil.copy(
BASE_DIR / "distilbert_training_script.py",
code_stage_dir / "distilbert_training_script.py",
)
estimator = HuggingFace(
entry_point="distilbert_training_script.py",
source_dir=str(code_stage_dir),
role=ROLE_ARN,
instance_type="ml.g4dn.xlarge",
instance_count=1,
transformers_version="4.56.2",
pytorch_version="2.8.0",
py_version="py312",
base_job_name="jev-distilbert",
sagemaker_session=sm_session,
max_run=1800,
)
estimator.fit(inputs={"train": train_s3_uri, "test": test_s3_uri}, wait=True, logs=True)Training jobs mount each input as a named channel (/opt/ml/input/data/train/, /opt/ml/input/data/test/), unlike a Processing job's single flat input path. The tested training script used distilbert-base-uncased, three epochs, batch size 16, learning rate 2e-5, weight decay 0.01, maximum token length 256, and FP16 when CUDA was available. The completed job recorded 430 billable seconds on ml.g4dn.xlarge, about $0.088 at the on-demand rate used for this test.
Replication path B: Call Jev through OpenRouter
Jev is hosted by TypeSafe and the reported test used OpenRouter's Decisions API. Set OPENROUTER_API_KEY in your environment before running this code.
Jev is hosted by TypeSafe and the reported test used OpenRouter's Decisions API. Set OPENROUTER_API_KEY in your environment before running this code.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import os
import requests
DECISIONS_URL = "https://openrouter.ai/api/alpha/decisions"
MODEL_ID = "typesafe/jev-1.13"
QUESTION_KEY = "spam_check"
CRITERIA = {
"spam": "Unsolicited, deceptive, or phishing content designed to manipulate the recipient.",
"not_spam": "Legitimate correspondence relevant to the recipient's actual affairs.",
}
def classify_jev(email_text: str) -> dict:
response = requests.post(
DECISIONS_URL,
headers={
"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}",
"Content-Type": "application/json",
},
json={
"model": MODEL_ID,
"state": {"text": email_text},
"questions": {
QUESTION_KEY: {
"type": "choice",
"instructions": "Is this email spam or legitimate correspondence?",
"criteria": CRITERIA,
}
},
},
timeout=30,
)
response.raise_for_status()
result = response.json()
answer = result["answers"][QUESTION_KEY]
return {
"label": answer["choice"],
"probabilities": answer["probabilities"],
"confidence": answer["confidence"],
"cost_usd": result.get("usage", {}).get("cost", 0.0),
}This is the request and response shape used for the Jev results. The test used state, a keyed questions object, criteria for each choice, and answers["spam_check"] in the response. It did not use the older input/decision shape or batch multiple emails in one request.
As of September 29, 2026, the public Jev endpoint detail reports one active TypeSafe endpoint for typesafe/jev-1.13, with status: 0, a prompt price of 0.000000042 dollars per token ($0.042 per million), and a completion price of zero. The default /api/v1/models response is text-output focused and omits this decisions-output model.
Replication path C: Call Claude Haiku on Bedrock
The reported Haiku run used Bedrock's native Converse API, the cross-region inference profile below, a one-word prompt, a 3,000-character input cap, and output normalization. It did not use toolConfig.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import os
import boto3
REGION = "us-east-1"
MODEL_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0"
PROMPT_TEMPLATE = """You are a spam classifier. Given the email below, respond with EXACTLY one word: either "spam" or "not_spam". No explanation, no punctuation, just the single word.
Email:
{text}
Classification:"""
session_args = {"region_name": REGION}
if os.getenv("AWS_PROFILE"):
session_args["profile_name"] = os.environ["AWS_PROFILE"]
client = boto3.Session(**session_args).client("bedrock-runtime")
def classify_haiku(email_text: str) -> str:
response = client.converse(
modelId=MODEL_ID,
messages=[{
"role": "user",
"content": [{"text": PROMPT_TEMPLATE.format(text=email_text[:3000])}],
}],
inferenceConfig={"maxTokens": 10, "temperature": 0},
)
raw = response["output"]["message"]["content"][0]["text"].strip().lower()
if "not" in raw or "ham" in raw:
return "not_spam"
if "spam" in raw:
return "spam"
return raw # record unrecognized output separately during scoringIf that inference profile is not enabled in your region, use the equivalent enabled Haiku 4.5 profile for your account. Keep the label contract and prompt format unchanged if you want comparable results.
Key difference from Jev is that Haiku generates text, so the test normalized the visible text before scoring. Jev returns a typed choice and probability distribution directly.
Score everything on the same held-out data
Compute the same two held-out slices for every system. Convert spam to 1 and not_spam to 0, retain every confusion matrix, and derive precision, recall, F1, and accuracy from it. The test also recorded Brier score for systems that returned probabilities.
Compute the same two held-out slices for every system. Convert spam to 1 and not_spam to 0, retain every confusion matrix, and derive precision, recall, F1, and accuracy from it. The test also recorded Brier score for systems that returned probabilities.
Do not silently treat failed API calls as correct predictions. The Jev run had two in-distribution failures, which were counted separately and excluded from its metrics and API-cost total. The Haiku and Luna runs completed all 2,125 calls. The SageMaker rows were batch jobs, so their job time is not comparable to API latency or a deployed endpoint's p50/p95 latency.
Series: Testing Decision Models (2 articles)
- 1Quick Testing Jev - LLM, a decision model, or your own trained classifier This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article