Privilege AIMKC2 results ↗

AI model fine-tuning

Fine-tuning changes what a model is good at. Whether that change is real is a question for a held-out comparison, not for the training loss curve.

READ THE DEFINITION ↓DATA · ADAPTATION · EVIDENCE

01 / DEFINITION

The short answer first.

What is AI model fine-tuning?

Fine-tuning continues training a pretrained model on task-specific examples so that it adopts a particular behaviour, format or judgement standard. It changes the model's weights, which makes the behaviour consistent without prompting for it — and makes any regression a matter of retraining rather than editing text.

WORKED EXAMPLE

Run a fine-tuning pilot with a decision at the end

Imagine a manufacturer extracting maintenance requirements from equipment manuals. The output needs a component identifier, an inspection interval and the sentence that supports it. One manual says to inspect a belt every six months. Another says to inspect it after six months of continuous operation. A model that outputs the same interval for both has missed a condition even if its JSON is perfectly formatted. Write the acceptance rule before choosing a training method.

Establish a useful baseline

Give the base model the relevant passage and a short output specification. Run the same material with a few carefully chosen examples in the prompt. If supplying the missing paragraph solves the problem, improve retrieval first. If the passage is available and the model repeatedly drops the operating condition, you have an identifiable behaviour to investigate. Save both outputs and have a domain reviewer identify the exact field that needs to change.

Construct training examples that distinguish calendar time, operating time and instructions conditional on inspection findings. Include cases where the manual gives no interval at all. Hold out entire manual families and revisions together. A rewritten instruction from the same machine is too close to the training example to establish that the model generalises to a new equipment family.

Inspect errors that a single score would hide

Run the candidate, the base model and the best prompt baseline on the same untouched manuals. Score field extraction, quotation fidelity and unsupported intervals separately. Have the reviewer inspect an example where the candidate improves and one where it regresses. A gain on formatting does not compensate for inventing an inspection interval. Give the owner of the maintenance workflow the actual failing passages, not just a percentage.

Record the base revision, adaptation method, training data version, evaluation split and inference settings. Then test the artefact on the hardware where it will actually run. A promising adapter that misses the response-time requirement or no longer follows the rule for missing evidence needs another iteration before release. The handoff should include the reproducible comparison, the cases still requiring review and a way to restore the previous model.

Further reading · LoRA: Low-Rank Adaptation of Large Language Models

02 / WHEN IT IS THE RIGHT TOOL

Adaptation options in increasing order of cost and commitment. Fine-tuning is the right answer when the cheaper options have been tried and the failure is about judgement rather than information.
OptionFixesDoes not fix
Better promptingFormat, tone, obvious instruction-following gapsMissing knowledge; inconsistent judgement under load
RetrievalMissing or changing facts the model cannot knowPoor reasoning about retrieved material
Few-shot examplesFormat and simple pattern matchingComplex criteria that exceed a context window
Fine-tuningConsistent expert judgement, domain vocabulary, calibrated scoringFacts that change after training; anything retrieval should own

The clearest signal that fine-tuning is warranted is a task where the criteria are long, specific and stable — a rubric that does not fit comfortably in a prompt and does not change weekly. Supervision and review tasks fit this shape well: the standard for what counts as defective work is elaborate, expert and durable.

The clearest signal against it is a task whose difficulty is informational. If the model fails because it does not know a current fact, fine-tuning will teach it a snapshot that ages. Retrieval is the correct tool, and RAG evaluation covers how to measure it.

03 / THE DATASET IS THE PROJECT

01

Define the standard.

Written criteria from the experts whose judgement is being encoded. Without them, labellers disagree and the model learns the disagreement.

02

Construct, do not scrape.

Coverage of the hard and rare cases matters more than volume. A thousand well-chosen examples routinely beat fifty thousand convenient ones.

03

Split before you start.

Hold out evaluation data by matter or source before any training run, so the acceptance measurement is available and uncontaminated.

04

Measure against the base.

Re-score the stock base model and any incumbent in the same run. The difference is the only defensible claim about the fine-tune.

Step three has a failure mode specific to expert domains: splitting by document rather than by matter or client leaves closely related material on both sides, and the resulting score is inflated in a way that is invisible in the pipeline. The detail is in held-out evaluation.

04 / WHAT FINE-TUNING COSTS YOU

↳ 01

Narrowing.

A model tuned for one task usually gets worse at adjacent ones. If the system needs both behaviours, that is two models or a routing layer, not one fine-tune.

↳ 02

Confidence inflation.

Fine-tuning sharpens the output distribution, so confidence often rises faster than accuracy. Calibration must be re-measured after training, not assumed to carry over.

↳ 03

A version to maintain.

A fine-tuned model is an artefact with a lifecycle: base model upgrades, data refreshes, re-evaluation. Budget for the second year, not just the first.

MKC2 shows both sides of this. Fine-tuned on a legal supervision task, it reaches 0.9623 ranking quality against the stock base model's 0.7281, on held-out items scored in the same run. Its calibration was measured separately rather than assumed, and it improves on the base there too. And it is a specialist — the claim is about legal supervision, not about general capability. Full record on the MKC2 page.

05 / QUESTIONS

How much data does fine-tuning need?

Fewer examples than most teams expect, and better ones than most teams have. For a well-defined judgement task, low thousands of carefully constructed examples with good coverage of the hard cases is a workable starting point. Quality of labelling dominates quantity throughout.

Is fine-tuning or retrieval the right choice?

They solve different problems and are frequently combined. Retrieval supplies facts the model cannot hold; fine-tuning supplies judgement the model does not have. A system that needs current information and expert standards needs both.

Can a fine-tuned small model beat a large general one?

On a narrow task, often. MKC2 is a compact fine-tune that substantially outperforms its own stock base on legal supervision. That result is specific to the task it was trained for and should not be generalised to other benchmarks.

Does fine-tuning risk memorising confidential training data?

It is a real consideration. Training material can be reflected in outputs, so sensitive corpora call for deliberate handling: deployment inside a controlled boundary, restricted access to the resulting weights, and evaluation for memorisation. See AI data privacy.

How long does a fine-tuning project take?

The training run is the short part. Defining the standard, constructing and labelling the dataset, and building the held-out evaluation are where the time goes — typically the large majority of the schedule. Any estimate that does not account for that is estimating the wrong activity.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Tell us which behaviour your current model gets wrong, what a correct answer looks like and where the finished model must run.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Tell us which behaviour your current model gets wrong, what a correct answer looks like and where the finished model must run.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Build the model
for the task.

Privilege AI constructs datasets, fine-tunes models and reports held-out comparisons against the stock base and the incumbent.