Every check has a named owner.
The owner defines what a correct answer is and reviews disagreements. This is a domain role, not an engineering one; the engineer owns the harness, not the definition.
MKC2 results ↗An evaluation framework is the structure that decides what gets measured, who answers for each measure, and how much authority a passing score actually buys.
01 / DEFINITION
The short answer first.An AI evaluation framework is the standing structure an organisation uses to measure models: a bank of defined checks, the data each one runs on, the metric each one reports, the threshold each must clear, and the authority a passing check grants. It converts one-off evaluations into a comparable, auditable record.
The difference between a framework and a script is that a framework survives a change of model. When a new candidate arrives, nothing needs to be invented: the same checks run, the same thresholds apply, and the comparison is immediate. Organisations without one re-litigate the measurement design every time a model changes, which is both slow and a quiet invitation to pick whatever metric flatters the new system.
WORKED EXAMPLE
A service desk wants a model to recommend the owner of an incident. The framework must connect the recommendation to a defined responsibility. Who labels the correct team? What counts as an acceptable handoff when two teams share the work? Which cases must stay with a human dispatcher? Record those decisions before selecting an evaluation tool.
Create one incident with a known owner, one requiring a shared handoff and one outside the supported taxonomy. Ask two service managers to label them independently. If they disagree, refine the rule or preserve an explicit ambiguous category. An unresolved operating policy is not a model defect. Training a judge to force a single answer can conceal that disagreement without solving it.
Run the incumbent and candidate on the same versioned examples. Save the inputs, recommendations, reviewer decisions and reasons. Separate evidence about routine routing from evidence about escalation. A gain on the frequent category cannot automatically authorise the model to decide rare incidents with higher operational consequences.
At release review, identify who can approve deployment, which changes trigger a new test and which conditions require a rollback. Give the service owner a short record they can use without the original evaluation team. If an incident category is renamed or a team takes on new responsibilities, the owner should be able to find the relevant tests and refresh the expected behaviour before relying on the old score.
02 / THE QUESTION BANK
A single quality judgement is almost never measurable. Breaking it into specific, independently scorable checks is what makes partial failure visible.
A check is a question with a defined answer space, a defined data source and a metric. Written well, it is narrow enough that two reviewers agree on the answer and broad enough to occur often enough to measure. MKC2's framework applies 45 such checks to each item, scored separately, on held-out items.
Decomposition also changes what a failure means. When a model is evaluated as a single unit, a weakness anywhere is an argument against deployment. When it is evaluated as 45 checks, a weakness is localised: that check loses authority, and the rest continue. The evaluation framework becomes the mechanism for partial trust rather than a gate that is either open or shut.
03 / AUTHORITY TIERS
| Tier | What the check may do | Evidence required |
|---|---|---|
| Refused | Nothing. The check is switched off. | Ranking quality too weak to separate the classes |
| Report-only | Record a finding for a human reader. | Useful ordering, but calibration or volume does not support a threshold |
| Flag | Raise an item for mandatory human review. | Strong ranking quality and calibration at the operating threshold |
| Act | Change an outcome without a human in the loop. | All of the above, plus a measured failure cost the business accepts |
In MKC2's published record, 25 of the 45 checks carry report-only authority and one is refused outright for insufficient ranking quality. The model itself operates in review-only mode: it reports and flags, and it does not act. That is a deliberate ceiling rather than a limitation discovered late, and it is the pattern set out in AI model supervision.
The value of an explicit tier system is that it gives a team somewhere to put a check that is interesting but not yet trustworthy. Without tiers, such a check is either shipped as if it were reliable or discarded, and both outcomes lose information.
04 / OWNERSHIP AND CADENCE
The owner defines what a correct answer is and reviews disagreements. This is a domain role, not an engineering one; the engineer owns the harness, not the definition.
Model version, prompt version, parser version, index version and set version. A figure missing any of them cannot be reproduced later.
Checks that fail to reach their threshold are reported alongside those that pass. A framework that only reports successes is a marketing instrument.
Cadence follows the same logic as software: fast, deterministic checks run on every change, while the full framework runs on a schedule and at every acceptance decision. The full run is what produces the dated record; the fast subset is what stops a regression reaching it. That division is described in AI model testing.
05 / QUESTIONS
The harness is largely generic and can be bought or assembled from open components. The question bank, the thresholds and the authority tiers are specific to your domain and your tolerance for error, and cannot be bought. Most of the value, and most of the work, is in the part that is not generic.
As many as there are distinct ways the work can be wrong, subject to each one occurring often enough to measure. Forty-five is what MKC2's legal review task decomposes into; a narrower task might need six. Checks that never fire should be retired rather than kept for completeness.
Domain experts own the definitions and thresholds; engineering owns the harness and the reproducibility; a single accountable owner signs the dated record. Splitting the first two is what keeps the framework tied to real consequences instead of to whatever is easy to compute.
By removing authority from that check, not from the model. The check drops to report-only or is switched off, the deployment continues with a documented gap, and the gap is a work item with an owner. Whole-model gates make this impossible and so tend to be waived.
It produces most of the evidence such obligations ask for: what was measured, on what data, by whom, with what result and what limitation. It does not substitute for governance decisions about acceptable use. See AI governance.
WORK WITH PRIVILEGE AI
A framework you can rely on.
We build evaluation frameworks where every check has a clear owner, a clear threshold and clear limits.
Honest about what does not work.
We report the checks that are not good enough alongside the ones that are.
Confidence before you deploy.
You know what your AI system has been tested on before it reaches users.
Talk with Privilege AI
Tell us which release decision needs evidence and who must sign off on the result.
CONTACT PRIVILEGE AI
Tell us which release decision needs evidence and who must sign off on the result.
Privilege AI builds evaluation frameworks with per-check thresholds, explicit authority tiers and published refusals.