Record the disposition.
For every flag: accepted, rejected, or unclear, with the reviewer and the time taken. This is the only data that measures the loop rather than the model.
MKC2 results ↗Human oversight is the most cited AI safeguard and the least designed. Review only catches errors if the reviewer has the time, the information and a reason to disagree.
01 / DEFINITION
The short answer first.Human-in-the-loop AI is a design in which a person reviews or approves an AI system's output or proposed action before it takes effect. It is a genuine control only when the reviewer has enough time, enough context and enough incentive to reach a different conclusion from the system.
The phrase appears in nearly every AI governance document and is rarely specified. A human in the loop who receives 400 proposals a day, each marked high confidence, with no indication of what to look at and no record of their decisions, is a formality. Designing the loop means addressing capacity, ordering, information and measurement explicitly.
WORKED EXAMPLE
A publishing assistant proposes replacing a product specification across a catalog. A reviewer sees a prominent approve button and a summary saying the change is minor. If the interface hides the original value, supporting evidence and affected products, the human is present but poorly equipped to exercise judgement. Start by defining what the reviewer must be able to inspect and change.
Show the previous specification, proposed replacement, source document and exact list of affected items. Let the reviewer approve, reject or request clarification. Approval should apply to this version of the proposal. If the assistant later adds another product or changes a unit, the earlier decision should not automatically release the new work.
Test an obvious error and a subtle one, such as a value copied correctly with the wrong unit. Observe whether reviewers notice the difference and how long the review takes. A system that requires confirmation on every harmless edit can create enough friction that people begin approving without inspection. Use the results to decide which changes need a stronger review and which can remain informational.
Finally test a missing reviewer, a rejected proposal and an expired approval. The application should preserve the last accepted catalog state and give the next responsible person the evidence behind the decision. A useful human-in-the-loop design makes those ordinary interruptions recoverable instead of treating them as reasons for the agent to proceed on its own.
02 / WHY REVIEW DEGRADES
| Failure | Mechanism | Design response |
|---|---|---|
| Automation bias | A confident machine answer anchors the reviewer's judgement | Show the evidence before the recommendation; show uncertainty honestly |
| Rubber-stamping | Volume exceeds real review capacity, so approval becomes the default | Cap the queue at reviewable volume; route the rest differently |
| Deskilling | Reviewers lose the practice needed to spot subtle errors | Rotate reviewers through unassisted work; seeded known errors |
| Responsibility diffusion | Everyone assumes someone else in the chain checked | One named approver per item, recorded |
Rubber-stamping is arithmetic rather than attitude. If genuine review of an item takes six minutes and a reviewer receives two hundred items in a day, the system has specified a control that cannot be performed. The honest responses are to reduce volume through better filtering, to increase capacity, or to accept a lower level of assurance and say so — not to record an approval that did not happen.
03 / ORDERING THE QUEUE
A queue ordered by calibrated confidence turns a fixed amount of expert attention into the maximum number of caught errors.
This is where supervision and human review meet. A model that ranks defective work above sound work reliably makes the first hour of review the most valuable hour; one that ranks close to random makes review order irrelevant, and the only way to catch a given share of errors is to read a corresponding share of the work.
Ordering depends on calibration as well as ranking. If the confidence attached to a flag does not mean what it says, reviewers learn to discount it, and the ordering information is lost even though the model computed it. That is why Privilege AI reports calibration separately from accuracy — MKC2's calibration is measured on held-out items against the stock base model it was fine-tuned from. Detail on the MKC2 page and in AI model calibration.
04 / MAKING OVERSIGHT MEASURABLE
For every flag: accepted, rejected, or unclear, with the reviewer and the time taken. This is the only data that measures the loop rather than the model.
A small, tracked share of items with deliberately introduced defects. Detection rate on those is a direct measure of review attention.
A reviewer agreeing with the system on nearly everything is either confirming an excellent model or not reviewing. The two are distinguishable only with seeded errors.
Disposition records are also the route to evidence about live performance. Privilege AI states plainly that MKC2's published figures are benchmark results on held-out data and that no attorney dispositions have been recorded, so the record does not show agreement with a lawyer on live work. Building the disposition loop is what would eventually make that claim available — which is a reason to design the loop as a measurement instrument, not only as a safeguard.
05 / QUESTIONS
Only if the review is real. A recorded approval that a reviewer had no capacity to perform documents a control rather than providing one, and it is the first thing an informed reviewer of the process will test — usually by asking how long each review takes and how many arrive.
Measure it rather than assume it: time a sample of genuine reviews and multiply. The answer is almost always lower than the volume the system produces, which is the fact that determines whether the design is honest.
Yes, if it is calibrated, because it helps them allocate attention. If it is not calibrated it is worse than nothing, since an overconfident score anchors judgement in the wrong direction. Calibrate first, display second.
Before any irreversible action, and as far upstream as the information allows. Confirming a proposal is a real control; being notified after an action has happened is monitoring, which is useful but different.
It can change what humans review, not remove them. A supervision model orders and filters so that expert attention reaches the right items; it does not carry accountability for a decision. MKC2 is deliberately built this way: it reports and flags, and it does not act.
WORK WITH PRIVILEGE AI
Review that catches errors.
We design review so your people spend their time on the work most likely to be wrong.
A record of every decision.
What the reviewer decided is recorded alongside the work.
People stay in control.
The AI assists; your team makes the call.
Talk with Privilege AI
Tell us which decision must stay with a person and what evidence that reviewer needs before acting.
CONTACT PRIVILEGE AI
Tell us which decision must stay with a person and what evidence that reviewer needs before acting.
Privilege AI builds supervision that orders review by calibrated confidence and records what the reviewer decided.