Jev vs BERT on Kyoto-ben Intent Classification, Measured
I ran Jev and fine-tuned BERT on the same 60 Kyoto-dialect phrases and compared accuracy, confidence intervals, calibration, speed, and cost.
TypeSafe AI announced Jev on September 15, 2026. Jev is what TypeSafe calls a System One model: it doesn't generate text, and returns probabilities over options you define. Its listed price is $0.042 per million input tokens, with free output.
Write-ups of Jev sometimes address how it differs from classifiers like BERT. I hadn't seen the two run on the same inputs, so I ran them. In an earlier article I built a web app that uses Jev to classify what a Kyoto-ben (Kyoto dialect) phrase really means into six categories. This time I kept those six categories and gave the same phrases to Jev and to BERT models I fine-tuned myself.
I'm not going to declare a winner. This article reports the numbers, the confidence intervals, whether the differences are significant, and the conditions behind them.
Two different kinds of tool
Jev takes the options and a description of each option in the request itself. In my earlier app, the six categories were defined that way, as short descriptions (called criteria) sent with every call. TypeSafe's material describes no training data requirement for this.
BERT is fine-tuned by you. As is generally known, that needs labeled training data, and the set of output categories is fixed at training time. Changing a category means fixing the data and training again.
So this comparison is about accuracy on one problem. It says nothing about ease of operation.
Comparisons that already exist
I checked what was public as of September 21, 2026.
- OpenSystem1-classifier (GitHub, maustin10) compares Jev with a zero-shot ModernBERT classifier. On a 30-case tool-routing set, Jev got 96.7% (29/30) and ModernBERT 76.7% (23/30). The author states that this is a BFCL-derived routing benchmark, not an official BFCL leaderboard score.
- Supa Journal (in Japanese) mainly compares Jev with four lightweight LLMs. BERT-family models are not part of the comparison, and it only cites an existing experiment.
- explainx.ai states that no controlled head-to-head benchmark of Jev against XGBoost or a fine-tuned BERT on identical tasks has been published by TypeSafe, an independent researcher, or explainx.ai.
- Verdict-open-jev (GitHub, Heman10x-NGU) compares Jev with a ModernBERT-based encoder (GLiClass backbone). Per its README, on TypeSafe's public evaluation of 337 cases, Jev scores 90.80% and the encoder 48.07%. On a four-workflow evaluation, Jev scores 68.00% and a TF-IDF + LogReg baseline 66.10%. The comparison is mainly against TypeSafe's published figures, the evaluations use datasets such as Banking77 and CLINC150, and the README does not mention Japanese. It is also a different task from my six categories.
I did not find a measurement that compares against fine-tuned BERT, in Japanese, on the same six categories. I can't rule out that I missed something.
Experiment design
Data
The six categories are social nicety, indirect refusal, literal compliment, warning, literal (meaning what it says), and sarcastic criticism. An example: "Bubuzuke demo doudosu?" ("Would you like some ochazuke?") is the famous anecdote for hinting that a guest should leave. How often anyone actually says it is debated.
I generated the phrases with Claude Sonnet 4.6 on Amazon Bedrock, at temperature 1.0, ten phrases per call.
- Training: 240 phrases, split 210 for training and 30 for validation (35 and 5 per category)
- Test: 60 phrases (10 per category), generated with a separate prompt and a separate list of scenes and relationships, then frozen after I checked every one by eye
- Labels for training and validation are exactly as Claude generated them, and I have not reviewed them
Leakage checks
Jev's criteria contain examples. At least 8 of the 14 sample phrases from my earlier article are in them, so any of those appearing in the test set would favor Jev.
I fixed a reference set (the earlier sample phrases, the earlier 10 generalization phrases, and the quoted examples inside the criteria) and checked the generated phrases against it. The checks covered exact matches after normalization, character 2-gram and 3-gram Jaccard similarity, edit-distance ratio at a 0.8 threshold, and containment. No issues came up. In the reverse direction, where a candidate is a substring of a reference, one test phrase matched: "Yoku okoshiyashita" (roughly "thank you for coming"), a short greeting contained in one of the earlier generalization phrases. It stays in the main results, and I report a sensitivity analysis without it below.
BERT training
I used two models: tohoku-nlp/bert-base-japanese-v3 and sbintuitions/modernbert-ja-130m. Training sizes were N=10, 25, 50, 100, and 200, with 5 seeds each. The learning rate (2e-5 or 5e-5) was picked once on the validation set at N=210 with seed 0, and both models got 5e-5. Everything ran on an M1 Mac using MPS.
Jev calls
I sent the same criteria as in the earlier app, unchanged. Jev got the 60 test phrases and the 10 generalization phrases, three repetitions each. All statistics use the 60 test phrases only. The generalization phrases are for reference.
Results
Accuracy
Confidence intervals are 95% bootstrap intervals from 10,000 resamples of the 60 test phrases, using the same resamples for every method. Jev is the mean of 3 repetitions and BERT is the mean of 5 seeds.
| Method | Accuracy | Macro-F1 | 95% CI |
|---|---|---|---|
| Jev (3-run mean) | 0.806 | 0.804 | [0.706, 0.900] |
| BERT (Tohoku) N=10 | 0.310 | 0.263 | [0.257, 0.367] |
| BERT (Tohoku) N=25 | 0.353 | 0.322 | [0.297, 0.410] |
| BERT (Tohoku) N=50 | 0.457 | 0.445 | [0.380, 0.530] |
| BERT (Tohoku) N=100 | 0.613 | 0.598 | [0.537, 0.687] |
| BERT (Tohoku) N=200 | 0.683 | 0.677 | [0.590, 0.777] |
| ModernBERT N=10 | 0.203 | 0.137 | [0.167, 0.240] |
| ModernBERT N=25 | 0.313 | 0.278 | [0.267, 0.360] |
| ModernBERT N=50 | 0.433 | 0.415 | [0.367, 0.500] |
| ModernBERT N=100 | 0.543 | 0.533 | [0.463, 0.620] |
| ModernBERT N=200 | 0.660 | 0.652 | [0.587, 0.730] |
Jev's three runs scored 0.800, 0.817, and 0.800. The label was identical across all three runs for 98.3% of phrases.
BERT's accuracy rises with N in both models, and it is still rising at N=200, the largest size I tried. At small N the seed matters a lot (the standard deviation for Tohoku at N=10 is 0.134).
Is the difference significant?
I compared Jev with each BERT at N=200 using a paired bootstrap on the same resamples. As a reference, I also ran McNemar's test on majority-vote correctness per phrase.
| Comparison | Difference (Jev minus BERT) | 95% CI | Bootstrap p | Significant at 0.05 | McNemar (Jev-only correct / BERT-only correct) |
|---|---|---|---|---|---|
| Jev vs Tohoku N=200 | +0.122 | [-0.006, 0.247] | 0.0614 | No | 13/7, p=0.263 |
| Jev vs ModernBERT N=200 | +0.146 | [0.039, 0.250] | 0.0082 | Yes | 12/7, p=0.359 |
For ModernBERT the two tests disagree: significant under the bootstrap, not under McNemar. They look at different things. The bootstrap resamples per-phrase accuracy over the test phrases, so it only captures variation from which phrases were drawn, and BERT's seed-to-seed variation is not in the interval. McNemar collapses each phrase to right or wrong by majority vote, which throws away information like "correct in only 2 of 5 seeds." With 60 phrases, the choice of test changes the conclusion, so I don't call a winner from either one alone.
Sensitivity analysis
Dropping the one phrase that matched in the reverse-containment check leaves 59 phrases. The numbers barely move: Jev 0.802 ± 0.010, Tohoku N=200 0.688 ± 0.015, ModernBERT N=200 0.661 ± 0.051.
Where the errors are
Per-category F1 puts Jev's "literal" category at 1.00. BERT's is 0.33 for Tohoku and 0.53 for ModernBERT at N=200, the lowest for both.
Jev's errors cluster near category boundaries. Summed over 3 runs (30 attempts per category):
- literal compliment answered as sarcastic criticism: 11 of 30
- warning answered as sarcastic criticism: 6 of 30
BERT makes few errors between those boundary categories. For Tohoku at N=200 (5-seed mean), it confuses "literal" with others: only 3.0 of 10 "literal" phrases were correct, with 3.2 labeled as warning and 2.6 as indirect refusal. ModernBERT at N=200 got 5.8 of 10 "literal" phrases right, with an F1 of 0.53. The errors land in different places for Jev and BERT. I did not investigate why Jev leans this way.
Calibration, speed, and cost
Calibration (ECE, 5 equal-width bins): Jev is 0.097 using its maximum probability and 0.150 using its confidence field. BERT at N=200 is 0.078 for Tohoku and 0.196 for ModernBERT. With 60 phrases in 5 bins, treat this as a rough guide.
Speed is not a like-for-like comparison. BERT ran batch-size-1 inference locally, at 17.56 ms for Tohoku on MPS. Jev is a network API call, with a median of 512 ms and p95 of 608 ms over 180 calls, and I have no view into Jev's internals.
Cost:
- Jev: input-only billing, about 1,224 input tokens per call, roughly $0.000051 per call, or $0.0031 for one pass over the 60 phrases
- BERT: training time for Tohoku at N=200 was 266 seconds summed over 5 seeds, and 1,202 seconds across all conditions. ModernBERT took 997 seconds across all conditions
Ten unseen phrases
Jev also ran on the 10 generalization phrases from the earlier article, and scored 0.7 in all three runs. I did not collect BERT predictions for these 10. I considered adding them and decided against it, so there is no comparison here.
Caveats
- The 60 test phrases (10 per category) are a small set, and the confidence intervals are wide
- "Correct" means agreeing with labels I checked by eye. It does not show that the label is right as Kyoto-ben usage
- Training and validation labels are as Claude generated them and unreviewed
- Test and training data both came from the same Claude model family, so their style is close, which may favor BERT (this is my guess)
- Jev's criteria contain at least 8 of the 14 sample phrases as examples, and none of them are in the test set
- This is a single experiment, and a different phrase set or category definitions could give different numbers
- BERT ran locally on MPS and Jev ran through its API, so conditions differ a lot, and I haven't verified Jev's internals
- MPS includes non-deterministic operations, so reruns with fixed seeds are only approximately reproducible
What the numbers say about choosing
Only what this experiment supports:
- With little training data, BERT scored 0.2 to 0.46 at N=10 to 50, well below Jev's 0.806
- With 200 training phrases, Tohoku BERT's gap to Jev is +0.122 and not significant, and ModernBERT's result depends on which test you use. I did not try larger N
- To change a label definition, Jev needs edited criteria and a resend, while BERT needs corrected data and retraining
- To keep everything on the device, BERT ran locally in about 17 ms with no network
- Near boundary categories such as warning and literal compliment versus sarcastic criticism, Jev showed a tendency to confuse them
For a real use case, running the same check on your own data is the reliable way to find out.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article