AI Model Evaluation
What to measure, on which data, and how to tell a real gain from a measurement artefact.
MKC2 results ↗A held-out set is a consumable resource. Every look at it spends a little, and most evaluation programmes spend it long before they notice.
01 / DEFINITION
The short answer first.Held-out evaluation measures a model on data excluded from every stage of its development: training, prompt iteration, threshold selection and error analysis. The exclusion is what makes the result an estimate of performance on new work rather than a description of what the model has already absorbed.
The definition is simple and the discipline is not. Exclusion has to hold against every channel by which information can reach the model, and in a language model pipeline there are more channels than in classical machine learning: the weights, the prompt, the few-shot examples, the retrieval index, the threshold, and the engineer who has read the failures and adjusted accordingly.
WORKED EXAMPLE
An engineering team has thousands of maintenance reports from similar machines. One pump failure generates a technician's initial note, a corrected report, a translated copy and a manufacturer's response. A random split can distribute those four documents between training and evaluation. The model may then recognise the incident while appearing to solve a previously unseen diagnosis.
Begin by identifying what should count as an independent unit for the intended deployment. If the system will review new incidents at familiar sites, group the entire incident before splitting. If it must transfer to unfamiliar plants, reserve some plants as well. A valid split depends on the claim you want to make. Neither a random sample nor a time cutoff automatically establishes the right boundary.
Build an incident map from identifiers, equipment references, dates and revision relationships. Inspect uncertain matches on the development side and record your grouping rules. Exact hashes catch identical files but miss translations, paraphrases and reports with a changed cover sheet. Ask an engineer to identify the ordinary document family for one incident and use that example to test the grouping logic.
Freeze the final set and restrict who can inspect its individual labels. Give model developers a separate validation set for prompt changes, threshold selection and adapter tuning. If a developer examines a final failure and changes the model to fix that particular example, move that example into development and arrange fresh confirmation material. Keeping its old label hidden afterward does not undo what the team learned.
Describe how many machines, sites and incidents contribute to the held-out score. An evaluation containing a hundred documents from two independent outages offers different evidence from one containing a hundred separate incidents. Review successes and failures by those units and record whether a particular machine family dominates the result.
When the system is updated, retain the split manifest and the data version. Repeat the comparison against the same incumbent, then obtain a fresh sample for claims about new operating conditions. The resulting record gives a reviewer a concrete answer to a difficult question: what experience could the model or its developers already have had with the material used to justify deployment?
02 / LEAKAGE PATHS
| Path | What happens | Practical defence |
|---|---|---|
| Prompt iteration | The prompt is tuned until the held-out score improves | A separate development set; the held-out set is read once per decision |
| Near duplicates | The same clause or template appears in train and test | Near-duplicate detection before splitting, not after |
| Group leakage | Different documents from the same matter land on both sides | Split by matter, client or source, never by row |
| Temporal leakage | Training data postdates test data, so the future is visible | Split by date, with the test set strictly later |
| Pretraining contamination | The material was public and is already in the base model | Prefer material written after the base model's cut-off |
| Label reuse | The same annotator labelled both sides with drifting criteria | Blind labelling, with agreement measured separately |
Group leakage is the one that produces the largest and most convincing overestimates. Splitting a corpus row by row when rows are clustered by matter, author or template means the model has effectively seen the test items in another form. The score can be dramatically higher than real performance and looks entirely legitimate, because nothing was technically duplicated.
03 / SET EXHAUSTION
A held-out set used for forty decisions has been optimised against forty times. It has quietly become a development set.
The mechanism is selection, not cheating. Each time a team makes a choice based on the held-out score — a prompt change, a threshold move, a different retrieval depth — it keeps the variant that happened to do better on those particular items. Repeat this often enough and a meaningful part of the reported gain is fitted to the sample.
Three practices keep the problem manageable:
04 / IN PRACTICE
MKC2's published figures come from items that were not seen during training, with 45 separate checks scored on each. Both comparison models — the system being replaced and the stock base model — were re-scored in the same run, on the same items, with the same parser, so the differences reported are differences between models rather than between reports.
The record also states what it does not show. No attorney dispositions have been recorded, so the figures do not demonstrate that MKC2 agrees with a lawyer on live work; they show how it behaves on held-out evaluation data. Keeping that boundary explicit is part of the method, not a disclaimer attached to it. The full record is on the MKC2 page.
05 / QUESTIONS
Enough for the smallest reported slice to be measurable rather than a fixed percentage. The common 80/10/10 split is a convention, not a requirement; with a large corpus, a smaller fraction is often ample, and with a rare but important class, a stratified sample matters far more than the overall ratio.
Cheap lexical methods first — normalised hashing and shingling catch templates and boilerplate — then embedding similarity for paraphrase. Run detection before splitting and keep duplicate clusters together on one side of the split rather than deleting them, since duplication is often a real property of the domain.
For a small number of acceptance decisions, yes, and that is exactly what it is for. The risk accumulates with the number of decisions made against it, not with calendar time. Log every read and plan a replacement.
Yes, with an extra trap: the retrieval index must not contain the answers to the held-out questions in a form that only exists because those questions were written. Build the index from the production corpus, then draw evaluation questions from material that is genuinely part of it. See RAG evaluation.
Then report cross-validation with confidence intervals and say plainly that no clean acceptance estimate exists. That is a weaker claim, and stating it weakly is better than presenting a development score as an acceptance result.
WORK WITH PRIVILEGE AI
Test sets that stay trustworthy.
We build evaluation sets that are kept separate from training, so results are not flattered.
Catch leaks early.
We check for overlap between test and training data before it skews a result.
Fair comparisons.
Every model is scored on the same material in the same run.
Talk with Privilege AI
Describe how your examples were collected and which related records could cross a training or evaluation boundary.
CONTACT PRIVILEGE AI
Describe how your examples were collected and which related records could cross a training or evaluation boundary.
Privilege AI constructs held-out evaluation sets, detects leakage before it flatters a result, and re-scores every baseline in the same run.