A queue that respects the score.
Flags ordered by calibrated confidence, so the reviewer's first hour is spent where the risk is highest. An unordered queue wastes the ranking the model provides.
Capability is measured.
Authority is earned.
AI model supervision places a specialised model between AI-generated work and the people who rely on it, with each check granted only the authority its evidence supports.
READ THE DEFINITION ↓REVIEW · FLAG · REFUSE
01 / DEFINITION
The short answer first.What is AI model supervision?
AI model supervision is the practice of running a second, specialised model over the output of a primary AI system to identify defective work before a person relies on it. The supervising model reports and flags; the authority it holds over each check is set by measured evidence rather than by its overall score.
Supervision is a deployment pattern, not a metric. Evaluation asks how good a model is before release; supervision asks what happens to each individual piece of work after release, when nobody has time to check all of it. The two are connected: a supervision model is only worth deploying if its own evaluation shows it can order defective work above sound work reliably enough for a threshold to exist.
02 / WHY THE PATTERN EXISTS
The problem review capacity creates.When should a model
trust another model?
Generation is cheap and review is not. Output volume grows faster than the expert attention available to check it, and the gap is where defects reach the client.
Once an organisation produces more AI-assisted work than it can review, it has three options: review a random sample and accept the rest unchecked, review everything and lose the productivity gain, or order the work so that scarce expert attention goes to the items most likely to be wrong. The third requires a score that ranks reliably, and that is what a supervision model provides.
The ordering property matters more than the accuracy. A supervisor that catches 96 of 100 defective passages when each is placed next to a sound one is useful even if it is occasionally wrong, because it concentrates review where the risk is. A supervisor that ranks close to random is worse than useless: it generates work while providing no information about where to look.
03 / AUTHORITY, NOT CAPABILITY
The distinction that defines the pattern.| Question | Answered by | Changes when |
|---|---|---|
| What can the model do? | Measurement on held-out data | The model, prompt or data changes |
| What is it allowed to do? | A decision by the organisation that carries the risk | The measured evidence or the accepted failure cost changes |
| What happens when it is wrong? | The surrounding process design | The review workflow changes |
Strong benchmark performance is one part of responsible deployment. It is not the same as permission to act. MKC2 illustrates the separation directly: of 45 checks, 25 carry report-only authority and one is refused outright for insufficient ranking quality, and the model as a whole operates in review-only mode — it reports and flags, it does not block or act on its own.
That ceiling is deliberate. It is set at the level where the evidence is strong and the failure mode is recoverable: a wrong flag costs a reviewer some time, while a wrong autonomous action costs something that cannot always be undone. Where an AI system does take actions, the equivalent controls belong to the agent layer, described in AI agent governance.
04 / DESIGNING THE LOOP
What has to exist around the model.A disposition record.
Every flag needs an outcome: correct, incorrect, unclear. Without recorded dispositions there is no way to measure the supervisor on live work, only on benchmarks.
A route for disagreement.
Reviewers who can only accept or dismiss a flag cannot report that the check itself is wrong. That signal is the main source of improvement for the question bank.
Point two is the honest limitation of any pre-deployment supervision claim, including Privilege AI's own. MKC2's published figures are measured on held-out evaluation data. No attorney dispositions have been recorded, so the record does not show that the model agrees with a lawyer on live work. Building the disposition loop is how that evidence eventually gets made, and it is worth saying plainly that it does not exist yet rather than implying it does.
05 / QUESTIONS
Asked when deciding how much authority a model should hold.How is supervision different from evaluation?
+
Evaluation is periodic and measures the model. Supervision is continuous and measures the work. An evaluation tells you a system is 93% accurate; supervision tells you which of today's documents to read first. Both are needed, and supervision depends on evaluation to establish that its own scores mean something.
Why run a supervision model in review-only mode?
+
Because the failure modes are asymmetric. A missed flag leaves the status quo, in which a human reviews the work; a wrong autonomous action changes a document, a filing or a client outcome. Review-only keeps the model's benefit while keeping the irreversible decisions with a person.
Does supervision require a specialised model?
+
For narrow, expert domains the evidence favours one. A stock model's ranking quality on MKC2's task is 0.7281 against 0.9623 for the fine-tuned version, on the same held-out items. For broader tasks a general model with a well-designed rubric may be sufficient; the way to find out is to measure both.
Can a supervision model review AI agents as well as text?
+
Yes, and that is a distinct task: reviewing a proposed action requires reasoning about consequences rather than about prose quality. MKC2 is built to evaluate AI-generated work and the actions of AI agents. See AI agent evaluation.
What does a supervision model cost to run?
+
It adds an inference pass over every reviewed item, which for an 8B-class model on local hardware is modest and predictable. The comparison that matters is against expert review time, not against zero. Deployment considerations are covered in on-premise LLM deployment.
Put the work
under review.
Privilege AI develops supervision models that rank defective work above sound work, and states the authority each check has earned.