Not yet measured by us

How to moderate content with a decision model

Moderation is where calibration stops being a technical nicety. A confidence number you cannot trust means either over-removal or under-enforcement, and both are visible.

The decision, typed

Does this post violate the policy, and which part of it? As a typed decision this is a choice — one option from a closed set you supply per call, over the policy categories you enforce.

How to actually set this up

  1. Write down the option set before you pick a model

    If you cannot enumerate the valid answers, this is not a typed decision yet and no model in the directory will help. The list is an argument to the call, so it can change later, but it has to exist now.

  2. Freeze a test set from decisions you already made

    Historical cases plus what was actually decided. Hold it out and never tune against it. Where your own records disagree with each other, keep the disagreement rather than cleaning it away.

  3. Measure accuracy at coverage, not accuracy

    A single accuracy figure cannot tell you what to automate. Sort by the model’s confidence, pick thresholds, and read off what share of the queue clears your error budget at each one.

  4. Check the calibration before you trust the threshold

    If the model says 0.9 on a hundred cases, roughly ninety should be right. When that does not hold, every threshold you set on top of it is wrong in a way accuracy will not reveal.

  5. Staff the escalation path before you ship the automation

    When the policy is genuinely contested inside your own team. If your reviewers disagree with each other, a model trained on their labels will inherit that disagreement and hide it behind a number.

Before you build it

What people ask at this point

The questions that come up once the decision is written down and the option set exists.

Do I need a decision model for this, or will a general LLM do?

A general model behind constrained decoding produces a typed answer too. What it does not reliably give you is a calibrated probability across the option set, and without that you cannot set a threshold. If you are going to send everything to a human anyway, you do not need the calibration and a general model is fine. The longer answer is here.

How much historical data do I need before this is worth trying?

Enough cases with recorded outcomes that you can hold out a test set and still have something left. A few thousand is comfortable. What matters more than volume is that the recorded outcome is what actually happened, not what a workflow defaulted to.

What if my own team disagrees about the right answer?

Score the cases your team agrees on separately from the ones they do not. A single accuracy figure across both hides which kind of error you are actually making, and the contested subset is usually where the model looks worst.

Which model should I start with?

Licence first, then size. Apache-2.0 or MIT, small enough to run where the decision happens, and shipping in a format you can actually deploy. The commercially licensed models are here. None of them has an independent benchmark, so treat the first pick as a candidate rather than an answer.

Find out what your data actually supports

We turn your historical decisions into a frozen test set, then tell you which model, which threshold, and how much of it you can safely automate.