Define the decision.
Write down what changes if the model passes. An evaluation with no attached decision produces a number nobody acts on.
Every model claim
is a measurement.
AI model evaluation decides what a system is allowed to do. Privilege AI builds evaluation sets, runs held-out comparisons and reports what the numbers can and cannot support.
READ THE DEFINITION ↓MEASUREMENT · CALIBRATION · AUTHORITY
01 / DEFINITION
The short answer first.What is AI model evaluation?
AI model evaluation is the measurement of what a model does on data it has not seen, using metrics chosen before the run. A complete evaluation reports accuracy, ranking quality, calibration and failure behaviour together, because a model can be accurate on average and still be unusable wherever its confidence is wrong.
Evaluation is not a single number. The word is used for three different activities that have different consumers: development evaluation, which guides training decisions; acceptance evaluation, which decides whether a model ships; and operational evaluation, which decides whether a deployed model still behaves as it did at acceptance. Confusing them is the most common structural error in an AI programme, and it usually shows up as a development metric being quoted in a governance meeting.
The practical distinction is who is allowed to see the data. Development sets are seen repeatedly and degrade with every look. Acceptance sets are seen once, ideally by someone who did not build the model. Operational measurement never has ground truth at the moment of use, so it depends on sampling, delayed labels and proxy signals instead.
02 / WHAT TO MEASURE
Four families, not one score.Accuracy is the
least interesting number.
Most evaluation reports stop at accuracy. Accuracy alone cannot tell you whether a model's confidence is usable, which is the property that determines whether anything can be automated.
| Family | Question it answers | Typical measure | What it decides |
|---|---|---|---|
| Correctness | How often is the output right? | Accuracy, exact match, F1 | Whether the model is competent at all |
| Ranking quality | Does the model score bad work above good work? | ROC AUC, pairwise ranking | Whether a threshold can exist |
| Calibration | Does a confidence of 0.8 mean 80%? | Brier score, expected calibration error | Whether the score can be shown to a human |
| Failure behaviour | What happens when it is wrong? | Error taxonomy, worst-slice analysis | How much authority the model may hold |
Ranking quality deserves more attention than it usually gets. If a model cannot order defective work above sound work, no threshold exists that separates the two, and every downstream decision built on its score is arbitrary. This is measurable independently of accuracy: a model can be right 90% of the time and still rank almost randomly on the 10% that matter.
Privilege AI reports these families separately for every evaluated model, including per-question breakdowns, so a single strong average cannot conceal a check that does not work. Further detail on each measure is set out in LLM evaluation metrics, and the confidence question is treated on its own in AI model calibration.
03 / METHOD
How an evaluation is actually run.Build the held-out set.
Sample from the same distribution as production, hold it out from every training and prompt-iteration loop, and keep its provenance recorded.
Re-score the baselines.
Run the incumbent and the stock base model in the same pass, on the same items, with the same parser. Numbers from different runs are not comparable.
Report the refusals.
Publish the checks that failed to reach the bar alongside those that passed, and switch the failing ones off rather than shipping them quietly.
Step three is where most published comparisons break. A baseline quoted from a vendor card, a paper or a previous quarter was produced with a different prompt, a different output parser and often a different sample. Re-scoring the incumbent in the same run is cheap, and it is the only version of the comparison that survives scrutiny.
Step four is a cultural problem rather than a technical one. An evaluation suite that has never rejected one of its own checks is not being read carefully enough.
04 / FIRST-PARTY EVIDENCE
What this looks like when it is done.A worked example:
MKC2.
MKC2 is a supervision model Privilege AI fine-tuned from Qwen3-8B to review legal drafting and the actions of legal AI agents. Its calibration record, dated 21 September 2026, is built on 4,403 held-out items with 45 separate checks applied to each, and the model it replaces was re-scored in the same run on the same material.
Ranking quality
Defective work ranked above sound work.
Stock base model: 0.7281.
Answers correct
Against 83.09% for the stock base model,
scored on the same material.
Checks refused
One check could not be shown to work,
so it is switched off, not shipped.
Benchmark results on held-out data, not live-matter outcomes. No attorney dispositions have been recorded, so none of these figures show that MKC2 agrees with a lawyer on live work. MKC2 runs in review-only mode: it reports and flags, it does not act. Full record on the MKC2 page.
05 / COMMON MISTAKES
Where evaluations quietly stop being valid.The test set has been seen.
Prompt iteration against a held-out set is training by another name. After a few dozen passes the set measures memorisation of your own fixes.
One average hides everything.
A single headline figure across many question types conceals the subset where the model is at chance. Per-check reporting is the fix.
The baseline is borrowed.
Comparing your fine-tune against a number from someone else's run compares two prompts and two parsers, not two models.
A fourth mistake is subtler: evaluating a model at a capability it will never be given. Measuring autonomous decision quality for a model that will only ever produce flags for a human reviewer wastes the evaluation budget and overstates the risk. Measure the authority the model is actually going to hold, which is the argument set out in AI model supervision.
06 / QUESTIONS
Asked by teams standing up an evaluation programme.How is AI model evaluation different from model testing?
+
Evaluation measures behaviour on unseen data to produce a defensible estimate of quality. Testing is the engineering discipline of running that measurement repeatedly and automatically so a regression is caught before release. Evaluation answers how good the model is; AI model testing answers whether today's build is worse than yesterday's.
How large does an evaluation set need to be?
+
Large enough that the smallest slice you intend to report on still has enough items to be meaningful, which is usually a stronger constraint than the overall total. A 4,000-item set with 45 checks per item gives a solid overall figure, but a check that only applies to 60 of those items carries a wide confidence interval and should be reported with that caveat rather than averaged away.
Can a model evaluate another model?
+
Yes, and it is now the practical option for open-ended work where no reference answer exists. It is also where most evaluation programmes introduce silent bias, because a general-purpose judge inherits the preferences of its own training. LLM-as-a-judge sets out when a judge needs to be fine-tuned and measured in its own right.
Do evaluations have to run on-premise?
+
Not always, but if the evaluation set contains client material, privileged documents or personal data, then the evaluation is a data-processing activity and inherits every constraint the production system has. In that case the harness runs inside the same boundary as the model. See private AI inference.
How often should a model be re-evaluated?
+
At every change to the model, the prompt, the retrieval layer or the output parser, and on a fixed calendar regardless of changes, because the input distribution moves even when the system does not. Continuous measurement of a deployed system is covered in AI model monitoring.
Bring a model
to the bench.
Privilege AI builds evaluation sets, runs held-out comparisons and reports the checks that do not work alongside the ones that do.