Fix the unit.
Per document, per decision, per matter. Risk expressed per system is not comparable to anything and cannot be aggregated.
The risk is not
that it is wrong.
AI risk management is mostly about detectability. A wrong answer you catch is an inconvenience; a wrong answer that looks right is the actual exposure.
READ THE DEFINITION ↓FAILURE MODES · DETECTION · EXPOSURE
01 / DEFINITION
The short answer first.What is AI risk management?
AI risk management is the identification of the ways an AI system can fail, the estimation of what each failure would cost, and the selection of controls that change the outcome. For AI systems the decisive variable is usually detectability: whether a failure is visible before it has consequences.
Generic risk frameworks score likelihood against impact. For AI systems that pair is insufficient, because the dominant failure mode is confident, plausible and wrong — a category that ordinary software mostly does not have. A crash is discovered immediately; a fabricated citation in a well-written document may never be discovered at all.
02 / A FAILURE-MODE CATALOGUE
What actually goes wrong, grouped by detectability.| Failure mode | Detectability | Control that changes the outcome |
|---|---|---|
| Service unavailable or timing out | Immediate | Ordinary resilience engineering |
| Malformed or unparseable output | Immediate | Schema validation and parse-failure alerting |
| Refusal to answer a legitimate request | Fast, via users | Refusal-rate monitoring; an escalation path |
| Subtly wrong content in valid form | Slow or never | Supervision, sampled expert review, groundedness checks |
| Confidence that does not mean anything | Invisible without measurement | Calibration measurement and recalibration |
| Quiet drift after deployment | Only with baselines | Distribution monitoring and periodic re-evaluation |
| Unauthorised action or disclosure | Often only after the fact | Least privilege, egress control, trajectory logging |
The top three rows are ordinary engineering problems and are usually handled. The bottom four are where AI-specific risk lives, and they share a property: none of them produces an error. They are detected by measurement or not at all, which is why an AI risk register with no measurement programme attached is a list of concerns rather than a management system.
03 / ESTIMATING EXPOSURE
Turning a failure mode into a number.Measure the rate.
From held-out evaluation for pre-deployment estimates, and from sampled review for live rates. An assumed rate carries the assumption forward invisibly.
Multiply by what escapes review.
The exposure is the error rate times the share that review does not catch. Review capacity is therefore a risk parameter.
State the residual plainly.
What remains after controls, in the same unit. A residual risk that cannot be stated has not been quantified.
Step three is the one that changes decisions. Teams tend to treat human review as a control that eliminates a failure mode, when in practice it reduces it by whatever share reviewers actually catch — a number that depends on volume, ordering and attention. A review step with no capacity analysis behind it is an assumption, not a mitigation, and the arithmetic is set out in human-in-the-loop AI.
04 / MITIGATIONS THAT MOVE THE NUMBER
Ranked by effect, not by effort.Four mitigations do most of the work in practice, and they are not the ones most often listed first.
- Reduce the authority. A system that proposes rather than acts converts many failure modes from incidents into rejected suggestions. The single largest available reduction, and the ceiling Privilege AI applies to its own supervision model.
- Order the review queue. Ranking output by calibrated confidence increases the share of errors caught without increasing review capacity.
- Bound the blast radius. Volume caps, short-lived credentials and egress allowlists limit what a single failure can reach.
- Measure continuously. Converts silent failures into detected ones, which is the difference between the bottom four rows of the catalogue and the top three.
Prompt-level instructions and staff guidance appear high on many mitigation lists and move the number least, because neither survives an input the model finds more persuasive. They are worth having and should not be counted as controls in a risk assessment.
05 / QUESTIONS
Asked when building an AI risk register.How do you assess risk for a system nobody can fully explain?
+
Behaviourally. Model interpretability is not a prerequisite for risk assessment: measured error rates on representative held-out data, plus a bounded account of what a failure can reach, are sufficient to make a decision. Waiting for explainability defers the assessment indefinitely.
What is residual risk for an AI system?
+
The exposure remaining after controls: the error rate that survives review, times the consequence, expressed per unit of work. Stating it in that form makes it comparable across systems and forces the review-capacity question into the open.
Should AI risk be managed separately from other technology risk?
+
It should use the same register and the same governance, with an AI-specific failure taxonomy. A separate parallel process tends to lose the connection to the business consequence, which is where the risk actually sits.
How often should an AI risk assessment be refreshed?
+
On every material change to the system, and on a fixed cycle regardless, because the input distribution moves without any change on your side. An assessment based on a year-old measurement describes a system that no longer exists.
What is the highest-value control for most AI deployments?
+
Reducing authority. Most AI risk becomes manageable when the system's worst case is a proposal a person declines, and most of it becomes hard to bound when the system can act on the outside world.
Measure the risk
you cannot see.
Privilege AI builds the measurement that separates a detected failure from a silent one: held-out evaluation, calibration and supervision.