Every check has a named owner.
The owner defines what a correct answer is and reviews disagreements. This is a domain role, not an engineering one; the engineer owns the harness, not the definition.
Checks with owners,
and limits.
An evaluation framework is the structure that decides what gets measured, who answers for each measure, and how much authority a passing score actually buys.
READ THE DEFINITION ↓CHECKS · TIERS · REFUSALS
01 / DEFINITION
The short answer first.What is an AI evaluation framework?
An AI evaluation framework is the standing structure an organisation uses to measure models: a bank of defined checks, the data each one runs on, the metric each one reports, the threshold each must clear, and the authority a passing check grants. It converts one-off evaluations into a comparable, auditable record.
The difference between a framework and a script is that a framework survives a change of model. When a new candidate arrives, nothing needs to be invented: the same checks run, the same thresholds apply, and the comparison is immediate. Organisations without one re-litigate the measurement design every time a model changes, which is both slow and a quiet invitation to pick whatever metric flatters the new system.
02 / THE QUESTION BANK
Decomposing a task into checks.One task,
forty-five questions.
A single quality judgement is almost never measurable. Breaking it into specific, independently scorable checks is what makes partial failure visible.
A check is a question with a defined answer space, a defined data source and a metric. Written well, it is narrow enough that two reviewers agree on the answer and broad enough to occur often enough to measure. MKC2's framework applies 45 such checks to each item, scored separately, on 4,403 held-out items.
Decomposition also changes what a failure means. When a model is evaluated as a single unit, a weakness anywhere is an argument against deployment. When it is evaluated as 45 checks, a weakness is localised: that check loses authority, and the rest continue. The evaluation framework becomes the mechanism for partial trust rather than a gate that is either open or shut.
03 / AUTHORITY TIERS
What a passing check is allowed to do.| Tier | What the check may do | Evidence required |
|---|---|---|
| Refused | Nothing. The check is switched off. | Ranking quality too weak to separate the classes |
| Report-only | Record a finding for a human reader. | Useful ordering, but calibration or volume does not support a threshold |
| Flag | Raise an item for mandatory human review. | Strong ranking quality and calibration at the operating threshold |
| Act | Change an outcome without a human in the loop. | All of the above, plus a measured failure cost the business accepts |
In MKC2's published record, 25 of the 45 checks carry report-only authority and one is refused outright for insufficient ranking quality. The model itself operates in review-only mode: it reports and flags, and it does not act. That is a deliberate ceiling rather than a limitation discovered late, and it is the pattern set out in AI model supervision.
The value of an explicit tier system is that it gives a team somewhere to put a check that is interesting but not yet trustworthy. Without tiers, such a check is either shipped as if it were reliable or discarded, and both outcomes lose information.
04 / OWNERSHIP AND CADENCE
Who answers for each number.Every figure has a date and a version.
Model version, prompt version, parser version, index version and set version. A figure missing any of them cannot be reproduced later.
Every refusal is published.
Checks that fail to reach their threshold are reported alongside those that pass. A framework that only reports successes is a marketing instrument.
Cadence follows the same logic as software: fast, deterministic checks run on every change, while the full framework runs on a schedule and at every acceptance decision. The full run is what produces the dated record; the fast subset is what stops a regression reaching it. That division is described in AI model testing.
05 / QUESTIONS
Asked when an evaluation capability is being built.Should we build an evaluation framework or buy one?
+
The harness is largely generic and can be bought or assembled from open components. The question bank, the thresholds and the authority tiers are specific to your domain and your tolerance for error, and cannot be bought. Most of the value, and most of the work, is in the part that is not generic.
How many checks is the right number?
+
As many as there are distinct ways the work can be wrong, subject to each one occurring often enough to measure. Forty-five is what MKC2's legal review task decomposes into; a narrower task might need six. Checks that never fire should be retired rather than kept for completeness.
Who should own the framework?
+
Domain experts own the definitions and thresholds; engineering owns the harness and the reproducibility; a single accountable owner signs the dated record. Splitting the first two is what keeps the framework tied to real consequences instead of to whatever is easy to compute.
How does a framework handle a model that fails one check badly?
+
By removing authority from that check, not from the model. The check drops to report-only or is switched off, the deployment continues with a documented gap, and the gap is a work item with an owner. Whole-model gates make this impossible and so tend to be waived.
Does an evaluation framework help with AI governance obligations?
+
It produces most of the evidence such obligations ask for: what was measured, on what data, by whom, with what result and what limitation. It does not substitute for governance decisions about acceptable use. See AI governance.
Give every check
an owner and a limit.
Privilege AI builds evaluation frameworks with per-check thresholds, explicit authority tiers and published refusals.