MKC2 results ↗Large language models produce text that has no single correct form. LLM evaluation is the practice of scoring that output reliably enough to make a release decision.
01 / DEFINITION
The short answer first.LLM evaluation is the measurement of a language model's output quality where no single reference answer exists. Because the same prompt can produce many acceptable responses, evaluation shifts from string comparison to graded judgement against explicit criteria, applied by rubric, by a judge model, or by a human panel.
Classical machine learning evaluation assumes a label. A classifier is right or wrong, and the metric follows. Language models break that assumption in three ways at once: the output space is open, the model is non-deterministic, and quality is multidimensional. A summary can be factually correct, well written and still useless because it omitted the one clause that mattered.
The consequence is that an LLM evaluation is only as good as its criteria. Writing the criteria down, in advance, in language a second person can apply consistently, is the majority of the work. Teams that skip it end up with a score whose meaning changes every time the rubric is re-read.
02 / THE FOUR HARD PROBLEMS
What makes generative evaluation different.Sampling makes a single run an unreliable estimate. Either fix the seed and temperature and accept that you are measuring one mode, or run each item several times and report the spread rather than one number.
Pairwise judges favour whichever answer appears first, often by several points. Swapping the order and averaging removes most of that bias and costs only a second pass.
Both human and model judges reward length and fluency independently of correctness. Controlling for response length is the difference between measuring quality and measuring verbosity.
The fourth problem is the one that survives longest: criteria drift. A rubric written in January is applied differently in June, because the team has seen six months of outputs and silently recalibrated. Periodically re-scoring a fixed set of historical items is the cheapest way to detect it. If yesterday's scores no longer reproduce, the rubric moved, not the model.
03 / SCORING APPROACHES
Choosing a method you can defend.| Approach | Best for | Cost | Main weakness |
|---|---|---|---|
| Exact or fuzzy match | Extraction, classification, structured fields | Negligible | Fails on any acceptable paraphrase |
| Rubric with a judge model | Drafting, reasoning, review tasks | Moderate | Inherits the judge's own preferences and blind spots |
| Pairwise preference | Comparing two candidate systems | Moderate | Gives a ranking, not an absolute quality level |
| Human expert panel | Setting the ground truth the rest is checked against | High | Slow, and inter-rater agreement must itself be measured |
A decision rule that holds up in practice: use the cheapest method that can distinguish the failure you actually care about. If the failure mode is a fabricated citation, an exact match against a source index beats any rubric and costs a fraction as much. If the failure mode is a plausible but unsound argument, no string comparison will ever find it, and a judged rubric is the only option. AI hallucination detection covers the first case in detail.
04 / REPORTING
Averages conceal the interesting part.A single headline score is the least useful artefact an evaluation can produce. The decisions that follow depend on where the model is weak, not on how it does on average.
Useful LLM evaluation reports break results down by document type, by question type, by input length and by the subpopulation each stakeholder cares about. In MKC2's record, 45 separate checks are scored on every item and reported individually; one check failed to reach the required ranking quality and was switched off rather than shipped inside an average. That level of granularity is what makes it possible to say which parts of a system are trustworthy instead of making a single claim about the whole.
The reporting format matters as much as the numbers. Publish the date, the item count, the model version, the prompt version and the parser version alongside every figure. A score without that context cannot be reproduced, and an irreproducible score is an opinion.
05 / QUESTIONS
Asked when a generative system is close to release.Enough that the difference you want to detect is larger than the noise. Detecting a small improvement between two similar systems needs thousands of items and repeated runs; confirming that a system is not catastrophically broken needs a few hundred. Size the set against the decision, and report confidence intervals rather than bare percentages.
No. Public benchmarks measure general capability on material that may have entered training data, and they say nothing about your documents, your vocabulary or your failure costs. Use them to shortlist, then run your own held-out set to decide. AI model benchmarking covers the contamination problem directly.
Greedy decoding makes runs reproducible and is the right default for regression testing. It is the wrong default for estimating production quality if production runs with sampling, because it measures a mode the users will never see. Many teams run both and treat the gap between them as a property of the system.
Evaluate end to end and then ablate. Measure the full pipeline first, then remove or degrade one component at a time to see where the quality comes from. A retrieval layer that contributes nothing is common, and only an ablation reveals it. RAG evaluation applies this to retrieval systems.
Yes. A harness needs the model weights, the evaluation set and a scoring function. Where a judge model is used, it can be a locally hosted open-weight model, which is the arrangement Privilege AI uses for sensitive material. See on-premise LLM deployment.
Privilege AI designs evaluation harnesses for generative systems, including judge-model calibration and per-slice reporting.