Golden cases.
A small, hand-curated set of inputs with known-good outputs. Fast, deterministic, and the first thing to run. If a golden case breaks, stop.
A result you
cannot reproduce
is an anecdote.
AI model testing turns a one-off evaluation into a suite that runs on every change, pins every version and refuses a release that moves the wrong way.
WHAT TESTING MEANS HERE ↓GOLDEN SETS · GATES · ADVERSARIAL
01 / DEFINITION
The short answer first.What is AI model testing?
AI model testing is the repeatable execution of a fixed set of cases against a model or pipeline, with pinned versions and recorded results, so that any change in behaviour can be attributed to a specific change in the system. Testing answers whether a build regressed; evaluation answers how good it is.
The distinction matters operationally. An evaluation is an event with a report. A test suite is infrastructure that runs unattended and produces a pass or fail. Teams that only do the former discover regressions from users; teams that only do the latter often cannot say what their absolute quality level is, only that it has not dropped.
Both are needed, and they usually share a data set but not a workflow. The evaluation set is large, expensive and read carefully by a person. The test set is small, fast, deterministic where possible, and read by a pipeline.
02 / SUITE DESIGN
Four layers, each catching something different.Regression corpus.
Every input that has ever produced a defect, kept forever. This is the layer that stops a fixed bug from returning two releases later.
Adversarial cases.
Inputs designed to break the system: injected instructions, contradictory sources, truncated documents, wrong language, empty retrieval results.
Statistical slice.
A sampled run over the large evaluation set, reported with confidence intervals rather than treated as a binary gate.
Layers one to three are binary gates. Layer four is not, and trying to make it one is a common source of flaky pipelines: a two-point move on a sampled metric is usually noise, and a gate that fires on noise gets disabled within a month. Gate on the deterministic layers, alert on the statistical one.
The adversarial layer is the one most teams add last and need first. For systems that read untrusted documents, it should include the prompt-injection cases described in AI agent prompt injection, because a retrieval pipeline that ingests a hostile PDF is an injection target whether or not it has agentic tools.
03 / PINNING
What has to be recorded for a result to mean anything.Six versions,
not one.
A test result is only attributable if everything that can change is recorded. In a language model pipeline that is at least six things: the model weights, the decoding parameters, the system prompt, the retrieval index, the output parser and the scoring function. A quality drop caused by a re-indexed corpus looks identical to one caused by a model swap unless both are pinned.
Two of these are routinely forgotten. The output parser silently determines the score whenever a model changes its formatting habits, and a parser that fails to extract an answer usually records a wrong answer rather than an error. The scoring function, particularly a judge model, has its own version and its own drift; pinning the judge is as important as pinning the model being judged.
04 / GATES
What a failing test should actually block.Block on regression, not on absolutes.
An absolute threshold ages badly and gets waived. A gate on movement relative to the current production build survives, because it always compares like with like.
Separate capability from authority.
A model that fails a check should lose the authority attached to that check, not the whole deployment. Per-check authority makes partial failure survivable.
Record the waiver.
Overrides happen. An override with a name, a date and an expiry is governance; an override applied silently in a pipeline configuration is a future incident.
Per-check authority is how MKC2 handles partial failure: of 45 checks, 25 carry report-only authority and one is refused outright for insufficient ranking quality. The model still ships; the check does not. That structure means a weak check degrades one flag rather than blocking the whole system, and it is the practical form of the argument made in AI model supervision.
05 / QUESTIONS
Asked by teams wiring evaluation into a pipeline.How do you test a non-deterministic model in CI?
+
Pin the decoding parameters to greedy for the deterministic layers so they behave like ordinary unit tests, and accept variance only in the sampled statistical layer. Where the production system samples, run the sampled layer several times per item and compare distributions rather than single values.
How large should a golden set be?
+
Small enough to run in minutes and be reviewed by a person in an afternoon, which in practice means tens of cases, not thousands. Its job is to fail loudly on obvious breakage. The large set does the measuring.
Who owns the test suite: the ML team or the product team?
+
The cases come from whoever understands the failure cost, which is usually the domain expert rather than the ML engineer. The infrastructure belongs to engineering. Splitting it that way keeps the suite anchored to real consequences instead of drifting towards whatever is convenient to measure.
What should happen when a test fails after deployment?
+
The same thing that happens when any production check fails: the affected capability is reduced to report-only or switched off while the cause is found. That is only possible if authority is attached to individual checks rather than to the deployment as a whole.
Does model testing replace human review?
+
No. Testing establishes that the system behaves as it did when a human last judged it acceptable. It does not extend that judgement to new categories of input. See human-in-the-loop AI.
Make the result
repeatable.
Privilege AI builds evaluation harnesses that pin every version, keep a regression corpus and report per-check rather than per-model.