AI Model Evaluation
What to measure, on which data, and how to tell a real gain from a measurement artefact.
The set is only
held out once.
A held-out set is a consumable resource. Every look at it spends a little, and most evaluation programmes spend it long before they notice.
READ THE DEFINITION ↓SPLITS · LEAKAGE · EXHAUSTION
01 / DEFINITION
The short answer first.What is held-out evaluation?
Held-out evaluation measures a model on data excluded from every stage of its development: training, prompt iteration, threshold selection and error analysis. The exclusion is what makes the result an estimate of performance on new work rather than a description of what the model has already absorbed.
The definition is simple and the discipline is not. Exclusion has to hold against every channel by which information can reach the model, and in a language model pipeline there are more channels than in classical machine learning: the weights, the prompt, the few-shot examples, the retrieval index, the threshold, and the engineer who has read the failures and adjusted accordingly.
02 / LEAKAGE PATHS
How a held-out set stops being held out.| Path | What happens | Practical defence |
|---|---|---|
| Prompt iteration | The prompt is tuned until the held-out score improves | A separate development set; the held-out set is read once per decision |
| Near duplicates | The same clause or template appears in train and test | Near-duplicate detection before splitting, not after |
| Group leakage | Different documents from the same matter land on both sides | Split by matter, client or source, never by row |
| Temporal leakage | Training data postdates test data, so the future is visible | Split by date, with the test set strictly later |
| Pretraining contamination | The material was public and is already in the base model | Prefer material written after the base model's cut-off |
| Label reuse | The same annotator labelled both sides with drifting criteria | Blind labelling, with agreement measured separately |
Group leakage is the one that produces the largest and most convincing overestimates. Splitting a corpus row by row when rows are clustered by matter, author or template means the model has effectively seen the test items in another form. The score can be dramatically higher than real performance and looks entirely legitimate, because nothing was technically duplicated.
03 / SET EXHAUSTION
The problem nobody budgets for.Every look
spends a little.
A held-out set used for forty decisions has been optimised against forty times. It has quietly become a development set.
The mechanism is selection, not cheating. Each time a team makes a choice based on the held-out score — a prompt change, a threshold move, a different retrieval depth — it keeps the variant that happened to do better on those particular items. Repeat this often enough and a meaningful part of the reported gain is fitted to the sample.
Three practices keep the problem manageable:
- A three-way split. Train, development and held-out, with the held-out set read only at acceptance decisions and its use logged.
- Scheduled rotation. Reserve unlabelled material in advance so a fresh set can be created when the current one has been consumed, rather than scrambling for data after the fact.
- A recorded read count. If nobody knows how many times the set has been scored, it should be assumed exhausted.
04 / IN PRACTICE
How Privilege AI reports held-out results.MKC2's published figures come from 4,403 items that were not seen during training, with 45 separate checks scored on each. Both comparison models — the system being replaced and the stock Qwen3-8B base — were re-scored in the same run, on the same items, with the same parser, so the differences reported are differences between models rather than between reports.
The record also states what it does not show. No attorney dispositions have been recorded, so the figures do not demonstrate that MKC2 agrees with a lawyer on live work; they show how it behaves on held-out evaluation data. Keeping that boundary explicit is part of the method, not a disclaimer attached to it. The full record is on the MKC2 page.
HELD-OUT DATA
Train — Develop — Accept
ONE READ PER DECISION05 / QUESTIONS
Asked when an evaluation result looks too good.What proportion of data should be held out?
+
Enough for the smallest reported slice to be measurable rather than a fixed percentage. The common 80/10/10 split is a convention, not a requirement; with a large corpus, a smaller fraction is often ample, and with a rare but important class, a stratified sample matters far more than the overall ratio.
How do you detect near duplicates at scale?
+
Cheap lexical methods first — normalised hashing and shingling catch templates and boilerplate — then embedding similarity for paraphrase. Run detection before splitting and keep duplicate clusters together on one side of the split rather than deleting them, since duplication is often a real property of the domain.
Can a held-out set be reused across model versions?
+
For a small number of acceptance decisions, yes, and that is exactly what it is for. The risk accumulates with the number of decisions made against it, not with calendar time. Log every read and plan a replacement.
Does held-out evaluation apply to retrieval systems?
+
Yes, with an extra trap: the retrieval index must not contain the answers to the held-out questions in a form that only exists because those questions were written. Build the index from the production corpus, then draw evaluation questions from material that is genuinely part of it. See RAG evaluation.
What if there is not enough data to hold any out?
+
Then report cross-validation with confidence intervals and say plainly that no clean acceptance estimate exists. That is a weaker claim, and stating it weakly is better than presenting a development score as an acceptance result.
Build a set
that stays clean.
Privilege AI constructs held-out evaluation sets, detects leakage before it flatters a result, and re-scores every baseline in the same run.