Jev

What it is

Jev takes a document and a list of questions, and answers each one with a value drawn from a type you declare up front. Three primitives cover everything: choice picks one option from a closed set, score returns a bounded number, noul returns a yes/no. Every choice comes back with a probability across the whole option set plus a confidence value, which is the part that makes threshold-setting possible at all.

The interface is one endpoint, POST /v1/systemone. There is no prompt to tune and no output to parse. That constraint is the product.

What we measured

We ran Jev 1.13.0 against the frozen test split of LocalLLaMA/typed-decisions on 2026-09-20: 400 cases, 2,000 individual decisions, questions passed through verbatim with no prompt rewriting. All 400 requests succeeded, the run took 34 seconds, and input cost was $0.0159.

Metric	Value	95% CI (2,000-sample bootstrap)
Accuracy	0.740	[0.721, 0.759]
Brier score, against the gold distribution	0.148	[0.139, 0.156]
ECE, 10 bins	0.045	[0.037, 0.066]
Score MAE	0.389	not computed
Latency	P50 687 ms / P95 777 ms	client-side, network included, 5 decisions per request

Broken out by primitive: choice 0.738 (n=600), score 0.705 (n=800), noul 0.787 (n=600). By workflow: agent trace observability 0.640, customer service 0.788, invoice processing 0.786, security incidents 0.744.

Confidence is worth trusting, so thresholds work

An ECE of 0.045 means the confidence number is close to the truth it claims. That is the precondition for routing on it. Abstain below a threshold and the accuracy on what remains moves like this:

Confidence threshold	Coverage	Accuracy within that coverage
None	100%	0.740
≥ 0.7	57.6%	0.864
≥ 0.9	24.5%	0.937

This table, not the headline accuracy, is what answers “how much of this queue can we automate.”

One number we could not reproduce

The laya-typed-decisions model card cites Jev figures to compare itself against. Most of them land where we landed. Brier matches to three decimal places, which tells us both sides are computing it the same way, against the gold distribution rather than one-hot labels.

Metric	Cited on the Laya card	Our measurement
Accuracy	0.727	0.740	inside our CI
Brier	0.148	0.148	exact match
ECE	0.144	0.045	outside our CI

We are not claiming the card is wrong. The most likely explanation is a different binning scheme or a different ECE definition, neither of which is published. What we can say is that following the public description does not reproduce 0.144, and that the gap matters: calibration is the entire selling point of this class of model.

The 0.740 is two different numbers stacked

The gold labels in this dataset are a teacher-model ensemble’s distribution, not human ground truth. The dataset ships a label_agreement.argmax_agree flag marking whether the teachers converged. On 40.6% of decisions they did not.

Subset	n	Accuracy	95% CI
Teachers agreed	1,188	0.841	[0.821, 0.862]
Teachers disagreed	812	0.591	[0.559, 0.626]

Any single headline accuracy on this dataset averages a task with a defensible answer against a task without one. We report both halves.

What we have not measured

Multilingual input, long-context behaviour anywhere near the 32K limit, throughput under concurrency, and cost at production volume. The vendor’s own speed and price comparisons against general LLMs are theirs, not ours, and we have not attempted to reproduce them.