What we found running the public test set
40.6% of the benchmark has no right answer
Decision model leaderboards are scored against labels produced by a teacher model ensemble, not by people. On four decisions in ten, those teachers disagree with each other. Every published score is averaging a task that has a defensible answer against one that does not.
40.6%
of the public test set has no teacher consensus
812 of 2,000 decisions, flagged by the dataset itself
0.841 / 0.591
accuracy on the agreed and the contested halves
the same model, the same run, two different tasks
0.045
measured calibration error, against 0.144 cited elsewhere
we could not reproduce the higher figure from public information
Why one number is hiding two
The benchmark everyone in this category reports against ships a flag on every decision saying whether the teacher models that produced the label agreed with each other. Split on it and a single accuracy figure of 0.740 comes apart into 0.841 where the graders converged and 0.591 where they did not.
Those are not the same task. On the first, being wrong means being wrong. On the second there is no objective answer to be right about, and a model that scores well there is most plausibly fitting the teacher ensemble’s noise. Which is also why claims about clearing the teacher ceiling deserve a second look: the only way past it runs through the contested 40%.
Nobody else publishes this split. It takes one line of code and it changes how you read every number in the category.
What we have measured so far
One model, properly. Nine more tracked and explicitly marked as unmeasured, because pretending otherwise would make the first number worthless too.
Jev, in full
2,000 decisions on a frozen split. Accuracy, Brier and calibration with bootstrap confidence intervals, the risk and coverage curve, and the one cited figure we could not reproduce.
The model directory
Every decision model we could verify, with base and licence on each entry, graded on whether anyone outside the lab that built it has checked the numbers. One has.
Calibration is what makes a score usable
Accuracy is one number about a population. The decision in front of you is per case: this ticket, automate or escalate. Those only connect if the confidence the model reports tracks how often it is actually right.
When it does, one model becomes several products. Automate everything at 0.740 and accept a 26% error rate, or automate the most confident quarter at 0.937 and route the rest to a person. Which row you pick is a question about your error budget, and no public leaderboard knows what that is.
| Confidence threshold | Coverage | Accuracy within that coverage |
|---|---|---|
| None | 100% | 0.740 |
| 0.7 and above | 57.6% | 0.864 |
| 0.9 and above | 24.5% | 0.937 |
What we sell
The public run builds a shortlist. It cannot tell you whether a model clears your error budget, because it has never seen your traffic. A private benchmark turns your own historical decisions into a frozen test set and returns the model, the threshold and the coverage your budget actually supports.
Your decisions are the only test set that counts
We freeze your historical cases into a set nobody gets to tune against, run the candidates through one interface, and hand back the threshold your error budget supports.