The test set
is on the internet.
Any benchmark published long enough ago has been scraped into pretraining corpora. Scores on it partly measure recall of the answer key, and the effect is invisible from the outside.
MKC2 results ↗Public benchmarks are useful for shortlisting and poor at predicting performance on a specific corpus. AI model benchmarking is the practice of knowing which is which.
01 / DEFINITION
The short answer first.AI model benchmarking is the comparison of models on a shared task and data set under identical conditions. Public benchmarks compare general capability across the industry; private benchmarks compare candidate models on one organisation's own material, which is the only comparison that predicts deployed performance.
Both kinds are legitimate and they answer different questions. A public benchmark answers whether a model is broadly competent. A private benchmark answers whether it is competent at your task, on your documents, with your definition of a mistake. Model selection needs both, in that order, and it goes wrong when the first is treated as a substitute for the second.
WORKED EXAMPLE
A software company is selecting a model to classify bug reports before they reach engineering. An attractive leaderboard score says little about a ticket that describes two symptoms, quotes an obsolete version number and mentions a workaround. The purchasing decision depends on whether the model routes current tickets to the right team at a cost and latency the support queue can tolerate.
Start with a sample of closed tickets and preserve the version of the routing taxonomy in force when the incidents occurred. Remove information the model would never see at intake, such as the final engineering disposition. Group duplicate reports about the same incident before splitting development and benchmark sets. A second description of a familiar outage is a weak test of generalisation.
Run the existing classifier, the candidate and a simple rules baseline on the same inputs. Give each system the same relevant product metadata. If one candidate retrieves documentation while another receives only the ticket, label that as a system comparison. It may answer the buying question, but it cannot isolate the contribution of the model weights. Keep prompts, versions, decoding settings and retrieval snapshots with the run.
Measure assignment accuracy by destination team, not just across the whole queue. A model that routes routine billing questions well can dominate the overall result while consistently sending firmware regressions to the wrong engineers. Count how often it declines to assign a ticket and what that costs the triage desk. Show the number of examples in each category beside its score.
Review paired disagreements with engineers who know the products but do not know which model produced each answer. Compare the cost of correcting a wrong route with the cost of reviewing an abstention. Record whether the observed gain occurs across independent incidents or depends on one large cluster of similar tickets. A small numerical advantage on a thin category may justify collecting more evidence before switching suppliers.
Include response time under the intended concurrency, operating cost and the effort needed to maintain the taxonomy. A model can win the routing comparison and still be unsuitable if the service cannot process the Monday intake volume. End the report with the specific operating conditions under which you would adopt the candidate, keep the incumbent or rerun the comparison after another independently labelled release.
02 / WHAT ERODES A PUBLIC SCORE
Any benchmark published long enough ago has been scraped into pretraining corpora. Scores on it partly measure recall of the answer key, and the effect is invisible from the outside.
Once the top models cluster within a point or two, the ranking is decided by the remaining items, which are disproportionately ambiguous or mislabelled.
Benchmarks that decide procurement get optimised against. That is rational behaviour and it makes the score a weaker signal every year.
None of this makes public benchmarks worthless. It makes them a filter rather than a decision. A model that is far down a well-constructed public leaderboard is unlikely to surprise you on your own task; a model at the top may still be the wrong choice for a corpus in a domain it has barely seen.
03 / BUILDING A PRIVATE BENCHMARK
| Condition | Why it matters | How it is commonly broken |
|---|---|---|
| Same items | Different samples have different difficulty | Baseline scored on last quarter's sample |
| Same prompt | Prompt quality dominates small model differences | The new model gets a tuned prompt, the old one does not |
| Same parser | Formatting changes are scored as wrong answers | A parser written around the new model's output style |
| Same run | Infrastructure, quantisation and versions drift | Baseline quoted from a vendor card or an old report |
| Held out from all of it | Iteration against a set destroys it | The set was used during prompt development |
Privilege AI applies these conditions to its own published work. The MKC2 calibration record re-scores both the model it replaces and the stock base model in the same run, on the same held-out items, with the same 45 checks. That is the reason the comparison can be stated as a difference between models rather than a difference between reports. The detail is on the MKC2 page.
A private benchmark also needs a defined lifespan. Once a set has guided a few dozen iterations it has become a development set, and a fresh held-out sample is required for the next acceptance decision. Budget for that replacement rather than discovering the need after the fact.
04 / READING A PUBLISHED COMPARISON
05 / QUESTIONS
Yes, as a coarse filter and as a check on general capability. They are unreliable for distinguishing between closely ranked models and for predicting behaviour on specialised material. Treat them as a way to build a shortlist of three, not to pick one.
Imperfectly. Practical signals include an unusual gap between performance on a public set and on a freshly written set of equivalent difficulty, suspiciously exact reproduction of canonical answer phrasing, and sharply better scores on older benchmark editions than on newer ones. The reliable defence is a private set written after the model's training cut-off.
Large enough that each reported slice is meaningful. A few hundred items is enough to rule out gross unsuitability; several thousand is needed to distinguish similar models or to report per-check figures. MKC2's record uses several thousand items precisely so that individual checks can be reported separately.
It happens on narrow, well-defined tasks, particularly after fine-tuning on domain material. MKC2 is fine-tuned from an open-weight base and outperforms that base substantially on its own supervision task. The claim is specific to that task and should not be read as a general statement about model size.
Yes, and on the hardware you will actually deploy on. A model that scores two points higher and runs four times slower on your GPUs may be the worse choice once the throughput requirement is included. See on-premise LLM deployment.
WORK WITH PRIVILEGE AI
Test models on your own material.
We build private benchmarks from your real documents, not public leaderboards.
Fair comparisons.
Every model you are considering is tested on the same material at the same time.
Choose with evidence.
You pick the model that works best for your work, backed by results you can trust.
Talk with Privilege AI
Tell us what you are comparing, which incumbent matters and what evidence would justify replacing it.
CONTACT PRIVILEGE AI
Tell us what you are comparing, which incumbent matters and what evidence would justify replacing it.
Privilege AI builds private benchmarks from real documents and re-scores every baseline in the same run.