Define the standard.
Written criteria from the experts whose judgement is being encoded. Without them, labellers disagree and the model learns the disagreement.
Specialise the model,
then prove it.
Fine-tuning changes what a model is good at. Whether that change is real is a question for a held-out comparison, not for the training loss curve.
READ THE DEFINITION ↓DATA · ADAPTATION · EVIDENCE
01 / DEFINITION
The short answer first.What is AI model fine-tuning?
Fine-tuning continues training a pretrained model on task-specific examples so that it adopts a particular behaviour, format or judgement standard. It changes the model's weights, which makes the behaviour consistent without prompting for it — and makes any regression a matter of retraining rather than editing text.
02 / WHEN IT IS THE RIGHT TOOL
Three cheaper options come first.| Option | Fixes | Does not fix |
|---|---|---|
| Better prompting | Format, tone, obvious instruction-following gaps | Missing knowledge; inconsistent judgement under load |
| Retrieval | Missing or changing facts the model cannot know | Poor reasoning about retrieved material |
| Few-shot examples | Format and simple pattern matching | Complex criteria that exceed a context window |
| Fine-tuning | Consistent expert judgement, domain vocabulary, calibrated scoring | Facts that change after training; anything retrieval should own |
The clearest signal that fine-tuning is warranted is a task where the criteria are long, specific and stable — a rubric that does not fit comfortably in a prompt and does not change weekly. Supervision and review tasks fit this shape well: the standard for what counts as defective work is elaborate, expert and durable.
The clearest signal against it is a task whose difficulty is informational. If the model fails because it does not know a current fact, fine-tuning will teach it a snapshot that ages. Retrieval is the correct tool, and RAG evaluation covers how to measure it.
03 / THE DATASET IS THE PROJECT
Where fine-tuning actually succeeds or fails.Construct, do not scrape.
Coverage of the hard and rare cases matters more than volume. A thousand well-chosen examples routinely beat fifty thousand convenient ones.
Split before you start.
Hold out evaluation data by matter or source before any training run, so the acceptance measurement is available and uncontaminated.
Measure against the base.
Re-score the stock base model and any incumbent in the same run. The difference is the only defensible claim about the fine-tune.
Step three has a failure mode specific to expert domains: splitting by document rather than by matter or client leaves closely related material on both sides, and the resulting score is inflated in a way that is invisible in the pipeline. The detail is in held-out evaluation.
04 / WHAT FINE-TUNING COSTS YOU
Effects that do not appear in the headline metric.Narrowing.
A model tuned for one task usually gets worse at adjacent ones. If the system needs both behaviours, that is two models or a routing layer, not one fine-tune.
Confidence inflation.
Fine-tuning sharpens the output distribution, so confidence often rises faster than accuracy. Calibration must be re-measured after training, not assumed to carry over.
A version to maintain.
A fine-tuned model is an artefact with a lifecycle: base model upgrades, data refreshes, re-evaluation. Budget for the second year, not just the first.
MKC2 shows both sides of this. Fine-tuned from Qwen3-8B on a legal supervision task, it reaches 0.9623 ranking quality against the stock base model's 0.7281 and 93.17% accuracy against 83.09%, on 4,403 held-out items scored in the same run. Its calibration was measured separately rather than assumed: expected calibration error of 0.0610 against 0.1363 for the base. And it is a specialist — the claim is about legal supervision, not about general capability. Full record on the MKC2 page.
05 / QUESTIONS
Asked when a general model is not good enough.How much data does fine-tuning need?
+
Fewer examples than most teams expect, and better ones than most teams have. For a well-defined judgement task, low thousands of carefully constructed examples with good coverage of the hard cases is a workable starting point. Quality of labelling dominates quantity throughout.
Is fine-tuning or retrieval the right choice?
+
They solve different problems and are frequently combined. Retrieval supplies facts the model cannot hold; fine-tuning supplies judgement the model does not have. A system that needs current information and expert standards needs both.
Can a fine-tuned small model beat a large general one?
+
On a narrow task, often. MKC2 is an 8B-class fine-tune that substantially outperforms its own stock base on legal supervision. That result is specific to the task it was trained for and should not be generalised to other benchmarks.
Does fine-tuning risk memorising confidential training data?
+
It is a real consideration. Training material can be reflected in outputs, so sensitive corpora call for deliberate handling: deployment inside a controlled boundary, restricted access to the resulting weights, and evaluation for memorisation. See AI data privacy.
How long does a fine-tuning project take?
+
The training run is the short part. Defining the standard, constructing and labelling the dataset, and building the held-out evaluation are where the time goes — typically the large majority of the schedule. Any estimate that does not account for that is estimating the wrong activity.
Build the model
for the task.
Privilege AI constructs datasets, fine-tunes models and reports held-out comparisons against the stock base and the incumbent.