12 decisions · 4 with our own measurements
How to automate a decision
Each of these starts the same way: write the decision down as a closed set of answers. If you cannot, no model on this site will help yet.
Measured on our own run
These four are workflows in the public benchmark, so we have real numbers for them: accuracy, how much of the queue clears each confidence threshold, and how often the graders themselves disagreed.
Which queue does this ticket belong in, and can it be routed without a person looking?
Ticket routing is the cleanest fit for a decision model in the whole category: the queues are enumerable, the volume is high, and being wrong costs a re-route rather than a disaster.
Does this invoice need a human to approve it, or does it match the purchase order closely enough to pass?
Approval gating is a yes or no with an asymmetric cost, which makes it the scenario where the confidence threshold matters most and where a calibrated model earns its keep.
How severe is this alert, and does it need to wake someone up?
Alert triage is a scoring problem wearing a routing problem's clothes. The hard part is not classifying the alert, it is knowing when the model does not know.
Did the agent actually complete the step it claims, and was the tool call the right one?
This is the scenario everyone wants and the one our own measurement performed worst on. Worth reading before you build on it.
The number to look at
Not the accuracy. The share of decisions where the graders disagreed with each other, which runs from 32% to 50% across those four. That is how much of each task has no defensible answer before any model touches it, and it caps what automation can ever mean there.
Not measured by us
Common decisions with no public benchmark behind them. The method on each page is the same; what is missing is anyone’s numbers, including ours.
Where do I draw the line between automate and escalate?
This is the only question that turns a benchmark score into a deployment. It has a method, and the method does not start with the model.
Out of the tools this agent has, which one should it call next?
Tool selection is the purest version of the problem: a closed set supplied per call, decided hundreds of times per session, where a generated reasoning trace per step is not affordable.
Which of these retrieved chunks are actually worth putting in the prompt?
Retrieval gives you candidates ranked by similarity. Similarity is not relevance, and the gap between them is a typed decision nobody wants to spend a full model call on.
Does this post violate the policy, and which part of it?
Moderation is where calibration stops being a technical nicety. A confidence number you cannot trust means either over-removal or under-enforcement, and both are visible.
Does this diff need a human reviewer, and which checks apply?
The densest cluster of real projects in this ecosystem is developer tooling, and most of them are doing some version of this.
Which element should the agent click, and what operation applies?
Browser automation makes a decision per step with a human watching. Latency is not an engineering detail here, it is the product.
Is this lead worth a salesperson's time, and which tier does it belong in?
Lead scoring has been a classifier problem for twenty years. What changes is that the tier definitions can now move per call without a retraining cycle.
Should this case be handled automatically, or does it need a person?
Every scenario on this list eventually reduces to this one. It is also the decision where the model's confidence is the input, not the output.
Find out what your data actually supports
We turn your historical decisions into a frozen test set, then tell you which model, which threshold, and how much of it you can safely automate.