Not yet measured by us
How to pick an agent's next tool with a decision model
Tool selection is the purest version of the problem: a closed set supplied per call, decided hundreds of times per session, where a generated reasoning trace per step is not affordable.
The decision, typed
Out of the tools this agent has, which one should it call next? As a typed decision this is a choice — one option from a closed set you supply per call, over the tools registered for this agent.
How to actually set this up
Write down the option set before you pick a model
If you cannot enumerate the valid answers, this is not a typed decision yet and no model in the directory will help. The list is an argument to the call, so it can change later, but it has to exist now.
Freeze a test set from decisions you already made
Historical cases plus what was actually decided. Hold it out and never tune against it. Where your own records disagree with each other, keep the disagreement rather than cleaning it away.
Measure accuracy at coverage, not accuracy
A single accuracy figure cannot tell you what to automate. Sort by the model’s confidence, pick thresholds, and read off what share of the queue clears your error budget at each one.
Check the calibration before you trust the threshold
If the model says 0.9 on a hundred cases, roughly ninety should be right. When that does not hold, every threshold you set on top of it is wrong in a way accuracy will not reveal.
Staff the escalation path before you ship the automation
When the agent genuinely needs to reason about why before it acts, and a person will read that reasoning. A decision model returns a label, not an argument.
Models worth shortlisting
Commercially licensed, open weights, small enough to run where the decision happens. None has an independent benchmark, so this is a shortlist rather than a ranking.
laya
The most-liked open-weight decision model of the post-Jev wave, a 421M ModernBERT-large encoder. The base release of a three-model family.
laya-multilingual
The only multilingual decision model we have found, a 322M encoder on mmBERT. Also the one our English-only test set cannot say anything useful about.
decider-2b
The most-downloaded open-weight decision model in the post-Jev wave. A Qwen3.5-2B decoder fine-tuned to emit typed decisions, and the largest member of a three-model family.
laya-typed-decisions
The Laya variant tuned on the typed-decisions benchmark, and the only open model whose card publishes head-to-head numbers against Jev. One of those numbers is the reason we started measuring.
Narrow it further: size: under 500m · format: gguf
People already doing this
Open-source projects in the agents cluster. We have not run any of them; this is working code to read, not a recommendation.
mobile-jev 272 ★
▶ Watch the demo — Jev opens Uber, enters a route from San Francisco Airport to the Golden Gate Bridge, and reaches paym
jev-codex-router 86 ★
Per-turn model & reasoning routing for Codex, driven by Jev (TypeSafe System One): picks the model, thinking depth and speed mode for every turn.
jev-recruiter 40 ★
A Jev powered LinkedIn recruiting agent. Watch it browse relevant profiles, save links, and review evidence against your hiring brief.
Jevbridge 31 ★
ACP and MCP adapter that bridges TypeSafe Jev with any LLM — computer use and typed decisions alongside Codex, Claude, Grok, and OpenCode.
pi-jev-router 14 ★
A minimal Pareto-optimal OpenRouter model router for pi, based on Jev
jev-mcp 14 ★
MCP server for TypeSafe Jev: typed classify, score, check, match and screen for any agent, with confidence on every answer
Other decisions people automate
Which queue does this ticket belong in, and can it be routed without a person looking?
Does this invoice need a human to approve it, or does it match the purchase order closely enough to pass?
How severe is this alert, and does it need to wake someone up?
Did the agent actually complete the step it claims, and was the tool call the right one?
Where do I draw the line between automate and escalate?
Which of these retrieved chunks are actually worth putting in the prompt?
Before you build it
What people ask at this point
The questions that come up once the decision is written down and the option set exists.
Do I need a decision model for this, or will a general LLM do?
A general model behind constrained decoding produces a typed answer too. What it does not reliably give you is a calibrated probability across the option set, and without that you cannot set a threshold. If you are going to send everything to a human anyway, you do not need the calibration and a general model is fine. The longer answer is here.
How much historical data do I need before this is worth trying?
Enough cases with recorded outcomes that you can hold out a test set and still have something left. A few thousand is comfortable. What matters more than volume is that the recorded outcome is what actually happened, not what a workflow defaulted to.
What if my own team disagrees about the right answer?
Score the cases your team agrees on separately from the ones they do not. A single accuracy figure across both hides which kind of error you are actually making, and the contested subset is usually where the model looks worst.
Which model should I start with?
Licence first, then size. Apache-2.0 or MIT, small enough to run where the decision happens, and shipping in a format you can actually deploy. The commercially licensed models are here. None of them has an independent benchmark, so treat the first pick as a candidate rather than an answer.
Find out what your data actually supports
We turn your historical decisions into a frozen test set, then tell you which model, which threshold, and how much of it you can safely automate.