Privilege AIMKC2 results ↗

AI risk management

AI risk management is mostly about detectability. A wrong answer you catch is an inconvenience; a wrong answer that looks right is the actual exposure.

READ THE DEFINITION ↓FAILURE MODES · DETECTION · EXPOSURE

01 / DEFINITION

The short answer first.

What is AI risk management?

AI risk management is the identification of the ways an AI system can fail, the estimation of what each failure would cost, and the selection of controls that change the outcome. For AI systems the decisive variable is usually detectability: whether a failure is visible before it has consequences.

Generic risk frameworks score likelihood against impact. For AI systems that pair is insufficient, because the dominant failure mode is confident, plausible and wrong — a category that ordinary software mostly does not have. A crash is discovered immediately; a fabricated citation in a well-written document may never be discovered at all.

WORKED EXAMPLE

Turn a risk register entry into an operating decision

A parts distributor uses an assistant to recommend substitutes when a component is unavailable. The risk register says that the system might propose an incompatible replacement. That description is a starting point, but it does not tell the warehouse when to stop a shipment, which substitutions require an engineer or how the team will know that its control is working.

Choose one product family and identify a mismatch that matters. Two parts might share a connector while differing in voltage or operating temperature. Ask the product owner which facts establish interchangeability and which conditions require confirmation. Write the failure as an observable event: an unverified substitute is presented as an approved replacement and reaches the picking workflow.

Test controls against both correct and incorrect substitutions

Give the assistant a verified substitute, a similar but incompatible part and a case with incomplete documentation. The expected outcomes differ. It should be able to progress the verified item, flag the incompatible one and request more evidence for the uncertain case. A system that rejects every substitution will avoid one risk while disrupting legitimate orders. Include that operating cost when deciding on the control.

Identify where the decision is enforced. A warning beside a recommendation is weaker than a warehouse rule that prevents release until a named engineer approves an unverified substitute. Check the authority of that engineer, the exact item approved and the expiry of the decision. A previous approval for a related product should not automatically authorise a later substitution.

Revisit the risk after a supplier change

When a supplier changes a component revision, repeat the test using the new specification. Sample approved shipments as well as blocked ones so the team can find confident mistakes. Record the review date, product owner, control owner, residual uncertainty and the conditions that would suspend automatic recommendations.

Use those observations to decide whether the system may continue assisting, needs a narrower product scope or should return certain decisions to manual review. The register then supports an operational choice rather than merely listing a technical concern. Keep the evidence and the decision together so a later team can reproduce the reasoning when products, volumes or supplier arrangements change.

Further reading · NIST AI Risk Management Framework

02 / A FAILURE-MODE CATALOGUE

AI failure modes ordered by how hard they are to notice. The controls differ by row: loud failures need handling, quiet ones need measurement.
Failure modeDetectabilityControl that changes the outcome
Service unavailable or timing outImmediateOrdinary resilience engineering
Malformed or unparseable outputImmediateSchema validation and parse-failure alerting
Refusal to answer a legitimate requestFast, via usersRefusal-rate monitoring; an escalation path
Subtly wrong content in valid formSlow or neverSupervision, sampled expert review, groundedness checks
Confidence that does not mean anythingInvisible without measurementCalibration measurement and recalibration
Quiet drift after deploymentOnly with baselinesDistribution monitoring and periodic re-evaluation
Unauthorised action or disclosureOften only after the factLeast privilege, egress control, trajectory logging

The top three rows are ordinary engineering problems and are usually handled. The bottom four are where AI-specific risk lives, and they share a property: none of them produces an error. They are detected by measurement or not at all, which is why an AI risk register with no measurement programme attached is a list of concerns rather than a management system.

03 / ESTIMATING EXPOSURE

01

Fix the unit.

Per document, per decision, per matter. Risk expressed per system is not comparable to anything and cannot be aggregated.

02

Measure the rate.

From held-out evaluation for pre-deployment estimates, and from sampled review for live rates. An assumed rate carries the assumption forward invisibly.

03

Multiply by what escapes review.

The exposure is the error rate times the share that review does not catch. Review capacity is therefore a risk parameter.

04

State the residual plainly.

What remains after controls, in the same unit. A residual risk that cannot be stated has not been quantified.

Step three is the one that changes decisions. Teams tend to treat human review as a control that eliminates a failure mode, when in practice it reduces it by whatever share reviewers actually catch — a number that depends on volume, ordering and attention. A review step with no capacity analysis behind it is an assumption, not a mitigation, and the arithmetic is set out in human-in-the-loop AI.

04 / MITIGATIONS THAT MOVE THE NUMBER

Four mitigations do most of the work in practice, and they are not the ones most often listed first.

  • Reduce the authority. A system that proposes rather than acts converts many failure modes from incidents into rejected suggestions. The single largest available reduction, and the ceiling Privilege AI applies to its own supervision model.
  • Order the review queue. Ranking output by calibrated confidence increases the share of errors caught without increasing review capacity.
  • Bound the blast radius. Volume caps, short-lived credentials and egress allowlists limit what a single failure can reach.
  • Measure continuously. Converts silent failures into detected ones, which is the difference between the bottom four rows of the catalogue and the top three.

Prompt-level instructions and staff guidance appear high on many mitigation lists and move the number least, because neither survives an input the model finds more persuasive. They are worth having and should not be counted as controls in a risk assessment.

05 / QUESTIONS

How do you assess risk for a system nobody can fully explain?

Behaviourally. Model interpretability is not a prerequisite for risk assessment: measured error rates on representative held-out data, plus a bounded account of what a failure can reach, are sufficient to make a decision. Waiting for explainability defers the assessment indefinitely.

What is residual risk for an AI system?

The exposure remaining after controls: the error rate that survives review, times the consequence, expressed per unit of work. Stating it in that form makes it comparable across systems and forces the review-capacity question into the open.

Should AI risk be managed separately from other technology risk?

It should use the same register and the same governance, with an AI-specific failure taxonomy. A separate parallel process tends to lose the connection to the business consequence, which is where the risk actually sits.

How often should an AI risk assessment be refreshed?

On every material change to the system, and on a fixed cycle regardless, because the input distribution moves without any change on your side. An assessment based on a year-old measurement describes a system that no longer exists.

What is the highest-value control for most AI deployments?

Reducing authority. Most AI risk becomes manageable when the system's worst case is a proposal a person declines, and most of it becomes hard to bound when the system can act on the outside world.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Tell us which decision creates risk, who bears the consequence and how your team currently detects a failure.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Tell us which decision creates risk, who bears the consequence and how your team currently detects a failure.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Measure the risk
you cannot see.

Privilege AI builds the measurement that separates a detected failure from a silent one: held-out evaluation, calibration and supervision.