Privilege AIMKC2 results ↗

LLM evaluation

Large language models produce text that has no single correct form. LLM evaluation is the practice of scoring that output reliably enough to make a release decision.

START WITH THE DEFINITION ↓GENERATIVE · REFERENCE-FREE · JUDGED

01 / DEFINITION

The short answer first.

What is LLM evaluation?

LLM evaluation is the measurement of a language model's output quality where no single reference answer exists. Because the same prompt can produce many acceptable responses, evaluation shifts from string comparison to graded judgement against explicit criteria, applied by rubric, by a judge model, or by a human panel.

Classical machine learning evaluation assumes a label. A classifier is right or wrong, and the metric follows. Language models break that assumption in three ways at once: the output space is open, the model is non-deterministic, and quality is multidimensional. A summary can be factually correct, well written and still useless because it omitted the one clause that mattered.

The consequence is that an LLM evaluation is only as good as its criteria. Writing the criteria down, in advance, in language a second person can apply consistently, is the majority of the work. Teams that skip it end up with a score whose meaning changes every time the rubric is re-read.

WORKED EXAMPLE

Evaluate whether a summary preserves a decision

A support team wants an LLM to summarise escalated tickets for engineers. The customer has already tried restarting the device, but a related replacement part has not been tested. A fluent summary can accidentally present both steps as completed and send the engineer in the wrong direction. The evaluation should ask what a reader could reasonably conclude about each action.

Design a pair with one meaningful difference

Write two tickets using the same product and symptoms. In the first, the customer tested the replacement and the fault persisted. In the second, the replacement was only proposed. Ask reviewers to label the completed steps, unresolved hypotheses and next appropriate question. Keep proposals distinct from observations when scoring the output. Overlap with an expected sentence cannot establish that the distinction survived.

Run both tickets with the same system instructions and output limits. Have reviewers compare the candidate with the current workflow without seeing the model names. Keep the better summary and the reason for the preference. If reviewers disagree about whether a restart counts as a meaningful diagnostic step, clarify the rubric before interpreting that disagreement as a model error.

Measure what happens at the handoff. Can an engineer identify the untested hypothesis without reopening the whole conversation? Does the summary preserve uncertainty where the customer gave an incomplete answer? Include one representative failure with the score so the team can choose between changing the prompt, improving the input or retaining human review for that ticket type.

02 / THE FOUR HARD PROBLEMS

01 — VARIANCE

Same prompt,
different answer.

Sampling makes a single run an unreliable estimate. Either fix the seed and temperature and accept that you are measuring one mode, or run each item several times and report the spread rather than one number.

Repeat runsReport variance
02 — POSITION

Order changes
the verdict.

Pairwise judges favour whichever answer appears first, often by several points. Swapping the order and averaging removes most of that bias and costs only a second pass.

Order swapBias control
03 — LENGTH

Longer reads
as better.

Both human and model judges reward length and fluency independently of correctness. Controlling for response length is the difference between measuring quality and measuring verbosity.

Length controlStyle separation

The fourth problem is the one that survives longest: criteria drift. A rubric written in January is applied differently in June, because the team has seen six months of outputs and silently recalibrated. Periodically re-scoring a fixed set of historical items is the cheapest way to detect it. If yesterday's scores no longer reproduce, the rubric moved, not the model.

03 / SCORING APPROACHES

Scoring approaches for open-ended output. Most working harnesses combine two: a cheap automatic layer over every item, and an expensive layer over a sampled subset used to keep the cheap layer honest.
ApproachBest forCostMain weakness
Exact or fuzzy matchExtraction, classification, structured fieldsNegligibleFails on any acceptable paraphrase
Rubric with a judge modelDrafting, reasoning, review tasksModerateInherits the judge's own preferences and blind spots
Pairwise preferenceComparing two candidate systemsModerateGives a ranking, not an absolute quality level
Human expert panelSetting the ground truth the rest is checked againstHighSlow, and inter-rater agreement must itself be measured

A decision rule that holds up in practice: use the cheapest method that can distinguish the failure you actually care about. If the failure mode is a fabricated citation, an exact match against a source index beats any rubric and costs a fraction as much. If the failure mode is a plausible but unsound argument, no string comparison will ever find it, and a judged rubric is the only option. AI hallucination detection covers the first case in detail.

04 / REPORTING

Report the slice,
not the mean.

Averages conceal the interesting part.

A single headline score is the least useful artefact an evaluation can produce. The decisions that follow depend on where the model is weak, not on how it does on average.

Useful LLM evaluation reports break results down by document type, by question type, by input length and by the subpopulation each stakeholder cares about. In MKC2's record, 45 separate checks are scored on every item and reported individually; one check failed to reach the required ranking quality and was switched off rather than shipped inside an average. That level of granularity is what makes it possible to say which parts of a system are trustworthy instead of making a single claim about the whole.

The reporting format matters as much as the numbers. Publish the date, the item count, the model version, the prompt version and the parser version alongside every figure. A score without that context cannot be reproduced, and an irreproducible score is an opinion.

05 / QUESTIONS

How many items does an LLM evaluation need?

Enough that the difference you want to detect is larger than the noise. Detecting a small improvement between two similar systems needs thousands of items and repeated runs; confirming that a system is not catastrophically broken needs a few hundred. Size the set against the decision, and report confidence intervals rather than bare percentages.

Is a public benchmark score enough to choose a model?

No. Public benchmarks measure general capability on material that may have entered training data, and they say nothing about your documents, your vocabulary or your failure costs. Use them to shortlist, then run your own held-out set to decide. AI model benchmarking covers the contamination problem directly.

Should evaluation run at temperature zero?

Greedy decoding makes runs reproducible and is the right default for regression testing. It is the wrong default for estimating production quality if production runs with sampling, because it measures a mode the users will never see. Many teams run both and treat the gap between them as a property of the system.

How do you evaluate a system rather than a model?

Evaluate end to end and then ablate. Measure the full pipeline first, then remove or degrade one component at a time to see where the quality comes from. A retrieval layer that contributes nothing is common, and only an ablation reveals it. RAG evaluation applies this to retrieval systems.

Can LLM evaluation run entirely offline?

Yes. A harness needs the model weights, the evaluation set and a scoring function. Where a judge model is used, it can be a locally hosted open-weight model, which is the arrangement Privilege AI uses for sensitive material. See on-premise LLM deployment.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Describe the language task, the audience reading the output and the error your reviewers most need to catch.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Describe the language task, the audience reading the output and the error your reviewers most need to catch.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Measure the system
you are shipping.

Privilege AI designs evaluation harnesses for generative systems, including judge-model calibration and per-slice reporting.