Meet JuL - a Jev' like System-One Decision Inside a Lambda
Run a Jev like System-One Decision Inside a Lambda
Most of the AI you use generates text: you send a prompt and it writes back a paragraph.
Last week a different shape of model appeared: Jev, from TypeSafe AI, calls it a "System One" model.
Last week a different shape of model appeared: Jev, from TypeSafe AI, calls it a "System One" model.
The principle is that you hand it some state and a list of typed questions, and it hands back typed decisions: a yes/no probability, a pick from a fixed set, or a score on a scale you defined. All the questions answered in one pass, without any token generated, in roughly a few hundred ms.
Meet JuL
JuL stands for "Juste use Less", and it is a drop-in replacement for Jev: it has the same API as Jev, TypeSafe's hosted System One model.
And it can run in an AWS Lambda, with the model inside the function's own zip, no GPU. The questions are fixed at build time and the event only carries the text.
The one model to deploy is
jul-decision-e5-small (ONNX, 8-bit), measured on arm64 in eu-west-1:- 17 ms per message at 1,769 MB
- $0.59 per million calls
- 2.4 s cold start
- 225 MB package, model included, under Lambda's 250 MB limit
What it is for
- One function decides one precise thing, always the same, on every event (ticket routing, "is this angry", "does this need a human").
- Not a general-purpose endpoint. For open-ended questions, use JuL's larger models.
- Zero-shot it scores 0.557 on the Jev bench.
- Tuned on a few hundred labeled examples it reaches 0.780.
Build it, on your machine
- Install and add the model:
1
2
pip install "jul[onnx,tune,yaml,calibrate]"
jul models add jul-decision-e5-small --repo usejul/jul-decision-e5-small-onnx --backend onnx
- (Optional) Write
questions.yamlwith the fixed questions (Choice, Noul, Score):
1
2
3
4
5
6
7
8
9
10
team:
type: choice
instructions: Which team should handle this ticket?
criteria:
billing: payments, invoices, refunds
tech: bugs, errors, crashes
sales: plans, pricing, upgrades
angry:
type: noul
instructions: Is the customer angry?
- (Optional) Label a few real messages in
labeled.jsonl, one message per line with the answer to each question, then tune:
1
2
{"state": "I was charged twice for order 4411, fix it now", "answers": {"team": "billing", "angry": true}}
{"state": "the export button does nothing", "answers": {"team": "tech", "angry": false}}
1
2
jul autotune tickets --questions questions.yaml --labeled labeled.jsonl --features hybrid \
--model jul-decision-e5-small --backend onnx
- (Optional) Pack and check which model the bundle needs:
1
2
jul pack bundle/ --questions questions.yaml --context tickets \
--model jul-decision-e5-small --backend onnx
- Choice and tuned heads read the
vectormodel, untuned yes/no and score read thecrossmodel. One model fits 250 MB, both do not. If it saysvector + cross, tune the yes/no questions or split into two functions. - Build the package:
1
2
3
4
5
6
7
8
9
10
11
mkdir -p build
PLAT="--platform manylinux2014_aarch64 --platform manylinux_2_28_aarch64 --implementation cp --python-version 3.12 --only-binary=:all:"
pip install -q --target build $PLAT "onnxruntime>=1.18" "jinja2>=3.1" "numpy>=1.24"
pip install -q --target build $PLAT --no-deps "tokenizers>=0.20"
pip install -q --target build --no-deps jul
cp -r bundle build/bundle
huggingface-cli download usejul/jul-decision-e5-small-onnx --local-dir build/model
du -sh build # must stay under 250 MB
1
2
3
4
5
6
lambda/
handler.py
requirements.txt # onnxruntime>=1.18, jinja2>=3.1, numpy>=1.24, tokenizers>=0.20, jul
bundle/ # from jul pack
model/ # huggingface-cli download usejul/jul-decision-e5-small-onnx
model-cross/ # only if jul pack said "vector + cross"
- Write
lambda/handler.py. It loads the bundle once at init and warms it, so the first real invocation does not pay for warm-up.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import os, time
from jul import Bundle
HERE = os.path.dirname(__file__)
_cross = os.path.join(HERE, "model-cross")
bundle = Bundle.load(os.path.join(HERE, "bundle"), backend="onnx",
model=os.path.join(HERE, "model"),
cross_model=_cross if os.path.isdir(_cross) else None)
bundle.system_one("warm-up") # at init, not on the first call
def handler(event, context):
messages = event.get("messages") or [event["message"]]
t = time.perf_counter()
responses = bundle.system_one_batch(messages)
return {"results": [{n: a.as_dict() for n, a in r.answers.items()} for r in responses],
"ms": round((time.perf_counter() - t) * 1000, 1)}
Deploy
model=andcross_model=in the handler point inside the package, so nothing is downloaded at run time. That is what keeps the cold start at 2.4 s.- With CDK, the function is a single asset from the
build/directory.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import * as path from "node:path";
import { App, CfnOutput, Duration, RemovalPolicy, Stack } from "aws-cdk-lib";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as logs from "aws-cdk-lib/aws-logs";
const BUILD = path.join(__dirname, "..", "build");
const MEMORY_MB = 1769; // one full vCPU, best price per call
const app = new App();
const stack = new Stack(app, "JulTickets", {
env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
});
const fn = new lambda.Function(stack, "Tickets", {
runtime: lambda.Runtime.PYTHON_3_12,
architecture: lambda.Architecture.ARM_64,
handler: "handler.handler",
code: lambda.Code.fromAsset(BUILD),
memorySize: MEMORY_MB,
timeout: Duration.seconds(30),
environment: { JUL_ONNX_THREADS: String(Math.max(1, Math.floor(MEMORY_MB / 1769))) },
logGroup: new logs.LogGroup(stack, "TicketsLogs", {
retention: logs.RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
}),
});1
2
3
4
npx cdk deploy
aws lambda invoke --function-name <FunctionName> --cli-binary-format raw-in-base64-out \
--payload '{"message": "I was charged twice, fix this now"}' /dev/stdout
Traps worth knowing
- 1,769 MB is one full vCPU and the best price per call.
- 3,538 MB does not cut latency and raises the bill.
- The model reads at most 512 positions.
- The end of a long message goes first, so it keeps its last paragraph.
- Pack on the backend you deploy on (ONNX).
- SnapStart buys nothing at this size as the 2.4 s is the model loading from the package.
- Batch several messages per event (
"messages": [...]) to push the per-message price down, worth it when the source is a queue.
A model in a Lambda beats a hosted API's per-call bill and a GPU server's idle cost when the decision is fixed and you have labels to tune on.
🗳️ Associated poll : https://builder.aws.com/poll/3KB0WDrnMayP1HCToGmZiWMYajP_po
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article