Define the standard.
Written criteria from the experts whose judgement is being encoded. Without them, labellers disagree and the model learns the disagreement.
MKC2 results ↗Fine-tuning changes what a model is good at. Whether that change is real is a question for a held-out comparison, not for the training loss curve.
01 / DEFINITION
The short answer first.Fine-tuning continues training a pretrained model on task-specific examples so that it adopts a particular behaviour, format or judgement standard. It changes the model's weights, which makes the behaviour consistent without prompting for it — and makes any regression a matter of retraining rather than editing text.
WORKED EXAMPLE
Imagine a manufacturer extracting maintenance requirements from equipment manuals. The output needs a component identifier, an inspection interval and the sentence that supports it. One manual says to inspect a belt every six months. Another says to inspect it after six months of continuous operation. A model that outputs the same interval for both has missed a condition even if its JSON is perfectly formatted. Write the acceptance rule before choosing a training method.
Give the base model the relevant passage and a short output specification. Run the same material with a few carefully chosen examples in the prompt. If supplying the missing paragraph solves the problem, improve retrieval first. If the passage is available and the model repeatedly drops the operating condition, you have an identifiable behaviour to investigate. Save both outputs and have a domain reviewer identify the exact field that needs to change.
Construct training examples that distinguish calendar time, operating time and instructions conditional on inspection findings. Include cases where the manual gives no interval at all. Hold out entire manual families and revisions together. A rewritten instruction from the same machine is too close to the training example to establish that the model generalises to a new equipment family.
Run the candidate, the base model and the best prompt baseline on the same untouched manuals. Score field extraction, quotation fidelity and unsupported intervals separately. Have the reviewer inspect an example where the candidate improves and one where it regresses. A gain on formatting does not compensate for inventing an inspection interval. Give the owner of the maintenance workflow the actual failing passages, not just a percentage.
Record the base revision, adaptation method, training data version, evaluation split and inference settings. Then test the artefact on the hardware where it will actually run. A promising adapter that misses the response-time requirement or no longer follows the rule for missing evidence needs another iteration before release. The handoff should include the reproducible comparison, the cases still requiring review and a way to restore the previous model.
Further reading · LoRA: Low-Rank Adaptation of Large Language Models
02 / WHEN IT IS THE RIGHT TOOL
| Option | Fixes | Does not fix |
|---|---|---|
| Better prompting | Format, tone, obvious instruction-following gaps | Missing knowledge; inconsistent judgement under load |
| Retrieval | Missing or changing facts the model cannot know | Poor reasoning about retrieved material |
| Few-shot examples | Format and simple pattern matching | Complex criteria that exceed a context window |
| Fine-tuning | Consistent expert judgement, domain vocabulary, calibrated scoring | Facts that change after training; anything retrieval should own |
The clearest signal that fine-tuning is warranted is a task where the criteria are long, specific and stable — a rubric that does not fit comfortably in a prompt and does not change weekly. Supervision and review tasks fit this shape well: the standard for what counts as defective work is elaborate, expert and durable.
The clearest signal against it is a task whose difficulty is informational. If the model fails because it does not know a current fact, fine-tuning will teach it a snapshot that ages. Retrieval is the correct tool, and RAG evaluation covers how to measure it.
03 / THE DATASET IS THE PROJECT
Written criteria from the experts whose judgement is being encoded. Without them, labellers disagree and the model learns the disagreement.
Coverage of the hard and rare cases matters more than volume. A thousand well-chosen examples routinely beat fifty thousand convenient ones.
Hold out evaluation data by matter or source before any training run, so the acceptance measurement is available and uncontaminated.
Re-score the stock base model and any incumbent in the same run. The difference is the only defensible claim about the fine-tune.
Step three has a failure mode specific to expert domains: splitting by document rather than by matter or client leaves closely related material on both sides, and the resulting score is inflated in a way that is invisible in the pipeline. The detail is in held-out evaluation.
04 / WHAT FINE-TUNING COSTS YOU
A model tuned for one task usually gets worse at adjacent ones. If the system needs both behaviours, that is two models or a routing layer, not one fine-tune.
Fine-tuning sharpens the output distribution, so confidence often rises faster than accuracy. Calibration must be re-measured after training, not assumed to carry over.
A fine-tuned model is an artefact with a lifecycle: base model upgrades, data refreshes, re-evaluation. Budget for the second year, not just the first.
MKC2 shows both sides of this. Fine-tuned on a legal supervision task, it reaches 0.9623 ranking quality against the stock base model's 0.7281, on held-out items scored in the same run. Its calibration was measured separately rather than assumed, and it improves on the base there too. And it is a specialist — the claim is about legal supervision, not about general capability. Full record on the MKC2 page.
05 / QUESTIONS
Fewer examples than most teams expect, and better ones than most teams have. For a well-defined judgement task, low thousands of carefully constructed examples with good coverage of the hard cases is a workable starting point. Quality of labelling dominates quantity throughout.
They solve different problems and are frequently combined. Retrieval supplies facts the model cannot hold; fine-tuning supplies judgement the model does not have. A system that needs current information and expert standards needs both.
On a narrow task, often. MKC2 is a compact fine-tune that substantially outperforms its own stock base on legal supervision. That result is specific to the task it was trained for and should not be generalised to other benchmarks.
It is a real consideration. Training material can be reflected in outputs, so sensitive corpora call for deliberate handling: deployment inside a controlled boundary, restricted access to the resulting weights, and evaluation for memorisation. See AI data privacy.
The training run is the short part. Defining the standard, constructing and labelling the dataset, and building the held-out evaluation are where the time goes — typically the large majority of the schedule. Any estimate that does not account for that is estimating the wrong activity.
WORK WITH PRIVILEGE AI
A model built for your task.
We prepare the training data and fine-tune a model to do one job well.
Proof that it is better.
We compare the tuned model against the original and your current system on fresh data.
Something you can run yourself.
A specialised model can be small enough to run inside your own environment.
Talk with Privilege AI
Tell us which behaviour your current model gets wrong, what a correct answer looks like and where the finished model must run.
CONTACT PRIVILEGE AI
Tell us which behaviour your current model gets wrong, what a correct answer looks like and where the finished model must run.
Privilege AI constructs datasets, fine-tunes models and reports held-out comparisons against the stock base and the incumbent.