laya-typed-decisions

What it is

The Laya checkpoint fine-tuned on LocalLLaMA/typed-decisions, the community benchmark set that the whole post-Jev wave converged on. Same 421M ModernBERT-large base as the general release.

Why this page exists

Its model card is, as far as we can find, the only open-weight card in this ecosystem that puts its own numbers next to Jev’s. The card itself notes that the comparison is not a same-conditions run. We took it at its word and ran Jev ourselves on the same frozen split.

Most of the cited figures held up. Accuracy landed inside our confidence interval and Brier matched to three decimal places, which is strong evidence both sides compute it identically, against the gold distribution rather than one-hot labels. Expected calibration error did not: the card cites 0.144 for Jev where we measured 0.045, well outside our interval.

The card uses that Jev figure to argue its own ECE of 0.213 is only slightly worse. Against 0.045 it would be roughly 4.7 times worse, not 1.5. We are not saying the card is wrong. Binning schemes and ECE definitions differ and neither is published. We are saying the comparison cannot be checked from public information, and that on a model class whose entire pitch is calibrated confidence, that is the number you least want to leave uncheckable.

The teacher ceiling

The card also positions its accuracy as clearing a teacher ceiling around 0.735. Worth knowing what that ceiling is made of: the gold labels are a teacher ensemble’s distribution, and on 40.6% of decisions in the test split the teachers do not agree with each other. Beating a teacher ensemble on labels that same ensemble produced is most plausibly explained by fitting its noise on the contested subset, where there is no objective answer to be right about.

Status

Not yet benchmarked by us. It is first among the Laya family when the open-weight batch runs, because it is the one with published claims to check.