Independent research on decision models
Decision models, checked in public
500 models in the directory, 344 open-source projects built on them, 158 head-to-head comparisons, and a full benchmark run with the working shown.
500
decision models tracked
counts re-read from Hugging Face every Monday
2,000
frozen decisions in our benchmark run
accuracy, calibration and coverage, with intervals
344
open-source projects built on them
sorted into 12 clusters by what they do
158
head-to-head comparisons
licence, architecture, size and evidence
Where to start
Six ways into the same data, depending on the decision in front of you.
Pick a model
Filter by format, size, licence and language. Every card carries its base model, family and evidence grade.
158 pairsCompare two
Side by side on repository metadata, with the rows that actually change the choice marked. Licence usually decides it first.
By licence, format and sizeReplace a hosted call
Open-weight candidates to run instead of the hosted API, grouped by the reason you would switch.
344 projectsSee what people build
Coding agents, browser extensions, trading systems. What each project decides, which model it calls, and when it was last pushed.
16 serversRun one yourself
Servers that speak the same API in front of weights you choose, plus runtime guides for GGUF, MLX, ONNX, Core ML.
12 walkthroughsAutomate a decision
Ticket routing, invoice approval, incident triage and more. 4 of the walkthroughs carry numbers from our own run.
Why one number is hiding two
The benchmark everyone in this category reports against ships a flag on every decision saying whether the teacher models that produced the label agreed with each other. Split on it and a single accuracy figure of 0.740 comes apart into 0.841 where the graders converged and 0.591 where they did not.
Those are not the same task. On the first, being wrong means being wrong. On the second there is no objective answer to be right about, and a model that scores well there is most plausibly fitting the teacher ensemble’s noise. Which is also why claims about clearing the teacher ceiling deserve a second look: the only way past it runs through the contested 40%.
Nobody else publishes this split. It takes one line of code and it changes how you read every number in the category.
How we grade what we publish
Every entry carries an evidence grade. Benchmarked means we ran the model on a frozen split ourselves and published accuracy, Brier score and calibration with bootstrap intervals. Community card means the facts are read from the model’s own repository, with the date they were read.
When a published figure does not reproduce, the page says so. Another model’s card cites a calibration error of 0.144 for Jev. We measured 0.045, and the full run shows how.
The directory and the project list are re-read every Monday. New models and projects join the week they appear, and every star, download and like count carries the date it was read.
What we sell
The public run builds a shortlist. It cannot tell you whether a model clears your error budget, because it has never seen your traffic. A private benchmark turns your own historical decisions into a frozen test set and returns the model, the threshold and the coverage your budget actually supports.
Your decisions are the only test set that counts
We freeze your historical cases into a set nobody gets to tune against, run the candidates through one interface, and hand back the threshold your error budget supports.