Measured on this exact workflow
How to gate invoice approval with a decision model
Approval gating is a yes or no with an asymmetric cost, which makes it the scenario where the confidence threshold matters most and where a calibrated model earns its keep.
The decision, typed
Does this invoice need a human to approve it, or does it match the purchase order closely enough to pass? As a typed decision this is a noul: a yes or no, over needs human · passes. Usually decided in the same call: How confident the line-item match is, scored rather than chosen.
What we measured on this workflow
From our own run of Jev against 500 decisions in this workflow, on a frozen split. Method and confidence intervals are on the model page.
| Subset | Decisions | Accuracy |
|---|---|---|
| All of it | 500 | 0.786 |
| Graders agreed | 340 | 0.868 |
| Graders disagreed | 160 | 0.613 |
The graders behind this benchmark disagreed with each other on 32.0% of these decisions. That gap is the single most useful thing on this page: it tells you how much of the task has a defensible answer at all before you ask any model to do it.
What you could automate
| Confidence threshold | Coverage | Accuracy within it |
|---|---|---|
| 0.5 and above | 85.8% | 0.83 |
| 0.7 and above | 63.4% | 0.924 |
| 0.9 and above | 15.2% | 1 |
How to actually set this up
Write down the option set before you pick a model
If you cannot enumerate the valid answers, this is not a typed decision yet and no model in the directory will help. The list is an argument to the call, so it can change later, but it has to exist now.
Freeze a test set from decisions you already made
Historical cases plus what was actually decided. Hold it out and never tune against it. Where your own records disagree with each other, keep the disagreement rather than cleaning it away. On this workflow that is 32.0% of cases.
Measure accuracy at coverage, not accuracy
A single accuracy figure cannot tell you what to automate. Sort by the model’s confidence, pick thresholds, and read off what share of the queue clears your error budget at each one.
Check the calibration before you trust the threshold
If the model says 0.9 on a hundred cases, roughly ninety should be right. When that does not hold, every threshold you set on top of it is wrong in a way accuracy will not reveal.
Staff the escalation path before you ship the automation
When a wrong approval is unrecoverable money. Set the threshold so the model only ever auto-passes, never auto-rejects, and let a person handle everything it is unsure about.
Models worth shortlisting
Commercially licensed, open weights, small enough to run where the decision happens. None has an independent benchmark, so this is a shortlist rather than a ranking.
laya
The most-liked open-weight decision model of the post-Jev wave, a 421M ModernBERT-large encoder. The base release of a three-model family.
JEV-27B-VL
Qwen3.8-27B fine-tuned to return typed decisions, 28B parameters.
JEV-35B
Qwen3.5-35B-A3B fine-tuned to return typed decisions, 36B parameters.
JEV-27B
Qwen3.8-27B fine-tuned to return typed decisions, 27B parameters.
Narrow it further: licence: commercial use · format: onnx
People already doing this
Open-source projects in the developer tooling cluster. Working code to read before you build your own.
jevgrep 3.1k ★
Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
skills 2.9k ★
Agent skills for building with TypeSafe's System One API
hermes-jev-skills 1.3k ★
Jev-powered model routing, memory, compaction, skill selection, computer and browser use for Hermes agents (also Claude Code and Codex)
jev-browser-use 1.2k ★
5–10x faster browser operations: Jev clicks, Codex thinks and verifies. Built at EZCollegeApp.
momo-code 1.1k ★
MOMO CODE — RSI *JEV*Pi --AI coding agent that evolves with you
foreman 804 ★
An agent supervisor and software factory foreman powered by decision models
Other decisions people automate
Which queue does this ticket belong in, and can it be routed without a person looking?
How severe is this alert, and does it need to wake someone up?
Did the agent actually complete the step it claims, and was the tool call the right one?
Where do I draw the line between automate and escalate?
Out of the tools this agent has, which one should it call next?
Which of these retrieved chunks are actually worth putting in the prompt?
Before you build it
What people ask at this point
The questions that come up once the decision is written down and the option set exists.
Do I need a decision model for this, or will a general LLM do?
A general model behind constrained decoding produces a typed answer too. What it does not reliably give you is a calibrated probability across the option set, and without that you cannot set a threshold. If you are going to send everything to a human anyway, you do not need the calibration and a general model is fine. The longer answer is here.
How much historical data do I need before this is worth trying?
Enough cases with recorded outcomes that you can hold out a test set and still have something left. A few thousand is comfortable. What matters more than volume is that the recorded outcome is what actually happened, not what a workflow defaulted to.
What if my own team disagrees about the right answer?
Then you are in the normal case. On this exact workflow, the graders behind the public benchmark disagreed with each other on 32.0% of decisions. Score the agreed and contested subsets separately, because a single accuracy number averages a task that has a defensible answer with one that does not.
Which model should I start with?
Licence first, then size. Apache-2.0 or MIT, small enough to run where the decision happens, and shipping in a format you can actually deploy. The commercially licensed models are here. None of them has an independent benchmark, so treat the first pick as a candidate rather than an answer.
Find out what your data actually supports
We turn your historical decisions into a frozen test set, then tell you which model, which threshold, and how much of it you can safely automate.