The test set
is on the internet.
Any benchmark published long enough ago has been scraped into pretraining corpora. Scores on it partly measure recall of the answer key, and the effect is invisible from the outside.
A leaderboard
is not evidence
about your work.
Public benchmarks are useful for shortlisting and poor at predicting performance on a specific corpus. AI model benchmarking is the practice of knowing which is which.
READ THE DEFINITION ↓CONTAMINATION · SATURATION · TRANSFER
01 / DEFINITION
The short answer first.What is AI model benchmarking?
AI model benchmarking is the comparison of models on a shared task and data set under identical conditions. Public benchmarks compare general capability across the industry; private benchmarks compare candidate models on one organisation's own material, which is the only comparison that predicts deployed performance.
Both kinds are legitimate and they answer different questions. A public benchmark answers whether a model is broadly competent. A private benchmark answers whether it is competent at your task, on your documents, with your definition of a mistake. Model selection needs both, in that order, and it goes wrong when the first is treated as a substitute for the second.
02 / WHAT ERODES A PUBLIC SCORE
Three forces, all pointing the same way.The test set
Everyone scores
ninety-something.
Once the top models cluster within a point or two, the ranking is decided by the remaining items, which are disproportionately ambiguous or mislabelled.
The number is
a marketing asset.
Benchmarks that decide procurement get optimised against. That is rational behaviour and it makes the score a weaker signal every year.
None of this makes public benchmarks worthless. It makes them a filter rather than a decision. A model that is far down a well-constructed public leaderboard is unlikely to surprise you on your own task; a model at the top may still be the wrong choice for a corpus in a domain it has barely seen.
03 / BUILDING A PRIVATE BENCHMARK
What a defensible comparison requires.| Condition | Why it matters | How it is commonly broken |
|---|---|---|
| Same items | Different samples have different difficulty | Baseline scored on last quarter's sample |
| Same prompt | Prompt quality dominates small model differences | The new model gets a tuned prompt, the old one does not |
| Same parser | Formatting changes are scored as wrong answers | A parser written around the new model's output style |
| Same run | Infrastructure, quantisation and versions drift | Baseline quoted from a vendor card or an old report |
| Held out from all of it | Iteration against a set destroys it | The set was used during prompt development |
Privilege AI applies these conditions to its own published work. The MKC2 calibration record re-scores both the model it replaces and the stock base model in the same run, on the same 4,403 held-out items, with the same 45 checks. That is the reason the comparison can be stated as a difference between models rather than a difference between reports. The detail is on the MKC2 page.
A private benchmark also needs a defined lifespan. Once a set has guided a few dozen iterations it has become a development set, and a fresh held-out sample is required for the next acceptance decision. Budget for that replacement rather than discovering the need after the fact.
04 / READING A PUBLISHED COMPARISON
Questions that separate evidence from marketing.Five questions
before you believe a number.
- What is the item count, and what is the confidence interval? A two-point gap on 300 items is not a result.
- Was the baseline re-scored in the same run? If not, the comparison includes two prompts and two parsers.
- What is the date of the record, and which model version? A score without a version is not reproducible.
- Which checks failed? A report with no negative findings has either an unusually good model or an unusually incurious author.
- Is the metric the one that matters? Accuracy on a balanced set says little about behaviour on a production distribution where the interesting class is rare.
05 / QUESTIONS
Asked while shortlisting models.Are public AI benchmarks still useful?
+
Yes, as a coarse filter and as a check on general capability. They are unreliable for distinguishing between closely ranked models and for predicting behaviour on specialised material. Treat them as a way to build a shortlist of three, not to pick one.
How do you detect benchmark contamination?
+
Imperfectly. Practical signals include an unusual gap between performance on a public set and on a freshly written set of equivalent difficulty, suspiciously exact reproduction of canonical answer phrasing, and sharply better scores on older benchmark editions than on newer ones. The reliable defence is a private set written after the model's training cut-off.
How big does a private benchmark need to be?
+
Large enough that each reported slice is meaningful. A few hundred items is enough to rule out gross unsuitability; several thousand is needed to distinguish similar models or to report per-check figures. MKC2's record uses 4,403 items precisely so that individual checks can be reported separately.
Can a smaller open-weight model beat a larger commercial one on a private benchmark?
+
It happens on narrow, well-defined tasks, particularly after fine-tuning on domain material. MKC2 is fine-tuned from an 8B open-weight base and outperforms that base substantially on its own supervision task. The claim is specific to that task and should not be read as a general statement about model size.
Should benchmarking include cost and latency?
+
Yes, and on the hardware you will actually deploy on. A model that scores two points higher and runs four times slower on your GPUs may be the worse choice once the throughput requirement is included. See on-premise LLM deployment.
Benchmark it
on your own material.
Privilege AI builds private benchmarks from real documents and re-scores every baseline in the same run.