Whether it agrees with your practitioners.
Held-out evaluation measures behaviour on evaluation data. Agreement on live matters requires a recorded disposition history, which is a separate exercise.
MKC2 results ↗Legal AI evaluation is mostly the work of writing down what counts as defective. The measurement is straightforward once that exists.
01 / DEFINITION
The short answer first.Legal AI evaluation is the measurement of an AI system's performance on legal work against an explicit standard for what counts as defective. It requires a held-out set drawn from real material, checks defined by practitioners, and reporting per check rather than as a single accuracy figure.
The reason the standard comes first is that legal quality is multidimensional in a way general evaluation is not. A passage can be well drafted and wrong on the law, correct and unsupported by its cited authority, accurate and procedurally inappropriate for the matter. A single quality score collapses those into a number nobody can act on.
WORKED EXAMPLE
A legal drafting assistant reviews a draft notice against its source agreement. One version correctly states the notice period; another silently changes calendar days to business days. A useful evaluation asks whether the system identifies the change, quotes the applicable passage and leaves interpretation and acceptance with the attorney. A generic writing-quality score cannot establish those outcomes.
Create synthetic agreements that express the condition in different ways. Keep an agreement, its amendments, related drafts and derivative questions in the same evaluation group. If an amendment appears in training and a paraphrase appears in the final test, a strong result may reflect familiarity with the matter rather than transferable review ability. Split by the underlying matter before scoring individual documents.
Have reviewers label the defect and its supporting evidence without seeing model identities. Include a case where the wording differs but the meaning is preserved, and one where the available materials are insufficient to decide. Review disagreement explicitly. A forced binary label can make a model look wrong when the reviewers have not defined a consistent legal review criterion.
Run the incumbent and candidate on the same untouched cases. Count missed material defects, unsupported warnings and cases correctly referred for human review. Inspect whether a warning identifies the right clause and revision. A correct overall flag attached to the wrong source passage can still waste attorney time or support the wrong conclusion.
Keep the test results separate from operational authority. A measured improvement on the held-out date cases is evidence about that defined task. It does not by itself establish agreement with attorneys across live matters or justify automatically blocking a document. Give the decision-maker the scope, examples, unresolved questions and review-only boundary alongside the score.
02 / BUILDING THE STANDARD
A workable standard is a bank of specific questions, each with a defined answer space and each narrow enough that two practitioners agree on the answer. MKC2's framework applies 45 such checks to every item. The decomposition is the expensive part and it is also the durable part: it survives model changes, which a prompt does not.
Three properties make a check usable:
03 / THE WORKED EXAMPLE
MKC2, as reported.Privilege AI's calibration record for MKC2, dated 21 September 2026, measures 45 checks on held-out items, with both the model being replaced and the stock base model re-scored in the same run on the same material.
| Measure | MKC2 | Stock base | Model replaced |
|---|---|---|---|
| Ranking quality | 0.9623 | 0.7281 | 0.6457 |
| Answers correct | Best | Second | Third |
| Brier score | Best | Second | Third |
| Calibration error | Best | Second | Third |
Calibration record, 21 September 2026. Benchmark results on held-out evaluation data, not live-matter outcomes. No attorney dispositions have been recorded, so none of these figures show that MKC2 agrees with a lawyer on live work. One check of 45 is refused outright for insufficient ranking quality; 25 carry report-only authority. MKC2 reports; it does not act. Full record on the MKC2 page.
The ranking figure is the one that decides whether the system is usable. A score that orders defective work above sound work only loosely cannot support a review queue, whatever its accuracy: no threshold separates the classes. The model being replaced scored 0.6457 there, below the stock base model, which is the clearest statement of why the work was done.
04 / WHAT A BENCHMARK CANNOT TELL YOU
Held-out evaluation measures behaviour on evaluation data. Agreement on live matters requires a recorded disposition history, which is a separate exercise.
Performance is corpus-dependent. Vocabulary, document types and the definition of a defect vary between practices.
A system that finds defects nobody has time to read has not improved anything. The review workflow is part of the measurement.
Two questions are worth asking any supplier. Were the baselines re-scored in the same run, or quoted from elsewhere? And which checks failed to reach the bar? A report with no negative findings has either an unusually good model or an unusually incurious author. See AI model benchmarking and held-out evaluation.
05 / QUESTIONS
Accuracy alone is the wrong question. Ask for ranking quality — whether the system orders defective work above sound work — and for calibration, because those decide whether the output can be used to direct review. A high accuracy figure with weak ranking quality cannot support a queue.
Enough for every reported check to be measurable, which for a decomposed standard means thousands rather than hundreds. MKC2's record uses several thousand items precisely so that individual checks can be reported separately rather than averaged.
The practitioners who carry the consequence, not the engineering team. A standard written for measurement convenience drifts towards what is easy to score and loses the confidence of the people expected to rely on it.
As a filter, not as a decision. Performance is corpus-dependent, and a figure produced on someone else's documents with someone else's defect definition says little about your material. Run your own held-out set.
That a check records a finding for a human reader and changes nothing on its own. It is the tier assigned where a check's measured evidence supports reporting but not gating. Twenty-five of MKC2's 45 checks sit there by design.
WORK WITH PRIVILEGE AI
Measure it on your own work.
We test legal AI against your standard of what counts as a defect.
Clear, honest reporting.
We publish which checks reach the bar and which do not.
Confidence before adoption.
You know how a tool performs before your lawyers rely on it.
Talk with Privilege AI
Tell us which legal review task you need to evaluate and which defects an attorney would consider material.
CONTACT PRIVILEGE AI
Tell us which legal review task you need to evaluate and which defects an attorney would consider material.
Privilege AI builds held-out evaluation sets and per-check reporting, and publishes the checks that do not reach the bar.