Privilege AIMKC2 results ↗

LLM evaluation metrics

Accuracy, ranking quality and calibration answer different questions. Reading a model report means knowing which question each number was built to answer.

START WITH THE DEFINITION ↓ACCURACY · AUC · BRIER · ECE

01 / DEFINITION

The short answer first.

What are LLM evaluation metrics?

LLM evaluation metrics are the numerical summaries of a model's behaviour on a test set. Each one supports a different decision: accuracy says whether the model is competent, ranking metrics say whether a threshold can exist, and calibration metrics say whether the model's confidence can be shown to a person.

WORKED EXAMPLE

Choose a metric for an extraction workflow

A shipping desk extracts container identifiers, destinations and collection dates from booking instructions. An exact-match score across the entire record is informative, but it treats a punctuation difference and a wrong destination as equally broken records. Before choosing a metric, ask which fields affect routing and which can safely be normalised. Document those rules before looking at the candidate outputs.

Keep the denominator attached to the result

Imagine twenty instructions, five of which deliberately omit the collection date. A system that invents dates for all five may achieve high field coverage while introducing unsafe information. Calculate date accuracy on instructions where a date is present, and measure correct abstention separately on the five instructions with no date. Report how many opportunities each rate represents. A reviewer should be able to reproduce both numbers without deciding after the fact which cases to exclude.

Use per-field results to diagnose behaviour and record-level results to assess whether the shipping desk can act. A correct destination and date do not rescue a booking assigned to the wrong container. Conversely, a case-sensitive identifier comparison may exaggerate errors if the downstream system intentionally normalises case. Verify such equivalences with the application owner rather than choosing whichever scoring rule flatters the model.

Keep an error sample next to the dashboard. Include a false extraction, a missed value and a justified refusal. Ask the desk how much work each creates. Those examples let the team set an operating threshold that reflects real handling costs and revisit it when the booking format changes.

02 / THE CORE FOUR

The four metrics that appear in most defensible model reports, with the decision each supports. The last column is from the MKC2 calibration record of 21 September 2026, measured on held-out items.
MetricRange and directionDecision it supportsMKC2 vs stock base
Accuracy0 to 1, higher betterIs the model competent at the task at all?MKC2 ahead
ROC AUC / ranking quality0.5 is chance, higher betterCan any threshold separate the two classes?0.9623 vs 0.7281
Brier score0 to 1, lower betterHow wrong are the probability estimates overall?MKC2 ahead (lower)
Expected calibration error0 to 1, lower betterDoes a stated confidence mean what it says?MKC2 ahead (lower)

Accuracy is the easiest to read and the easiest to mislead with. On an imbalanced set, a model that always predicts the majority class can score well while being completely uninformative. Always report the class balance next to it.

ROC AUC measures ordering: the probability that a randomly chosen positive scores above a randomly chosen negative. It is threshold-free, which makes it the right metric when the threshold has not been chosen yet, and it is the first thing to check before anyone designs a routing rule. An AUC near 0.5 means no threshold will work, whatever the accuracy says.

Brier score is the mean squared error of probabilistic predictions. It combines being right with being appropriately confident, which makes it a good single summary and a poor diagnostic: a bad Brier score does not tell you which of the two went wrong.

Expected calibration error buckets predictions by stated confidence and measures the gap between claimed and observed accuracy in each bucket. It is the number to quote when a human will see the model's confidence, and it is sensitive to bucket count, so the bucketing scheme should be published alongside it.

03 / BEYOND THE CORE

01 — AGREEMENT

Cohen's kappa,
Krippendorff's alpha.

When ground truth comes from human raters, measure the raters first. Agreement below the level you expect from the model means the labels, not the model, are the limiting factor.

Inter-raterLabel quality
02 — RETRIEVAL

Recall@k, nDCG,
context precision.

Generation quality is capped by whether the right passage was retrieved at all. Retrieval metrics isolate that ceiling from the model's reasoning.

Recall@knDCG
03 — GROUNDEDNESS

Support rate and
citation validity.

The share of generated claims traceable to a retrieved source. A direct measure of fabrication risk, and cheaper to compute than a full rubric.

FaithfulnessAttribution

Two older metrics deserve a caution. BLEU and ROUGE measure n-gram overlap with a reference text. They remain useful for tightly constrained tasks such as translation or templated extraction, and they are close to meaningless for open-ended drafting, where a correct answer that shares no vocabulary with the reference scores near zero. Reporting ROUGE on an open-ended task is usually a sign that no one chose a metric deliberately.

Retrieval metrics are treated in full on retrieval evaluation, and groundedness on AI hallucination detection.

04 / HOW METRICS MISLEAD

↳ 01

The aggregate hides the slice.

A strong mean across 45 checks can contain one check at chance level. Per-slice reporting is the only way to find it; averaging is how it survives to production.

↳ 02

The metric outlives its threshold.

A threshold chosen on last year's distribution encodes last year's base rate. When the input mix shifts, the same threshold produces a different false-positive rate without any model change.

↳ 03

The metric became the target.

Any metric optimised against for long enough stops measuring what it was chosen for. Rotating a held-out set is the standard defence.

05 / QUESTIONS

Which single metric should we report?

None. The minimum honest report is three: accuracy for competence, a ranking metric for separability and a calibration metric for whether the confidence is usable. Any one of the three can look acceptable while another is failing.

What is a good ROC AUC?

It depends entirely on the task and the base rate, so context matters more than a universal threshold. What can be said generally is that values near 0.5 mean the score carries no ordering information, and that the gap between a fine-tuned and a stock model on the same items is more informative than either absolute value.

What is the difference between Brier score and calibration error?

Brier score combines discrimination and calibration into one number, so it drops if either gets worse. Expected calibration error isolates the calibration part: it can be near zero for a model that is uninformative but honestly uncertain. Report both, because neither is interpretable alone.

Do these metrics apply to generative text?

They apply wherever the system produces a score or a decision, which includes most review, classification and flagging tasks built on generative models. For free-form drafting, the scoring layer has to produce a judgement first, and the metrics then measure that judgement. See LLM-as-a-judge.

How should confidence intervals be reported?

Alongside every figure, derived from the item count. Bootstrapping over the test set is adequate and easy to implement. Without intervals, any comparison of two similar models is unreadable, because the reader cannot tell a real difference from sampling noise.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Share the output your team scores today and the kind of mistake an aggregate metric conceals.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Share the output your team scores today and the kind of mistake an aggregate metric conceals.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Report the numbers
that carry a decision.

Privilege AI builds evaluation reports that separate competence, separability and calibration instead of collapsing them into one score.