Retrieval Evaluation
Measuring the retriever on its own: recall, ranking quality and the judgements behind them.
Bad answer.
Which stage?
A RAG system fails in at least four places. Evaluating the whole pipeline tells you that it failed; stage-wise measurement tells you where.
READ THE DEFINITION ↓RECALL · PRECISION · GROUNDEDNESS
01 / DEFINITION
The short answer first.What is RAG evaluation?
RAG evaluation measures a retrieval-augmented generation system at each stage — chunking, retrieval, ranking, generation — as well as end to end. Stage-wise measurement is what makes a failure attributable: an answer can be wrong because the passage was never retrieved, or because it was retrieved and ignored.
A single end-to-end score is the right thing to report and the wrong thing to debug with. Two systems can produce identical answer quality for opposite reasons: one retrieves well and reasons poorly, the other retrieves poorly and compensates with a strong model. Their improvement paths are completely different, and only stage-wise numbers distinguish them.
02 / THE FOUR STAGES
What to measure where.| Stage | Measure | Symptom when it fails |
|---|---|---|
| Chunking | Whether answers survive intact in one chunk | Answers that span a boundary are never found |
| Retrieval | Recall@k: is the answer passage in the candidate set? | Confident answers with no relevant source |
| Ranking | Context precision: is it near the top? | The right passage is retrieved but crowded out |
| Generation | Groundedness and answer quality | Correct sources present, conclusion unsupported |
Chunking is the least measured and frequently the most consequential. A chunk boundary through the middle of a definition, a table or a clause makes the answer unretrievable at any k, and no amount of model improvement recovers it. The diagnostic is simple: for a sample of evaluation questions, check by hand whether the answer exists intact inside a single chunk. If it does not, the pipeline has a ceiling that no retrieval tuning will lift.
Retrieval recall is the next ceiling. If the relevant passage is not in the top k, the generator cannot use it, and the honest options are refusal or fabrication. Measuring recall separately turns a vague quality complaint into a specific, fixable number.
03 / GROUNDEDNESS
Whether the answer follows from the sources.Sourced
is not the same as
supported.
Groundedness is the share of claims in an answer that are actually supported by the retrieved passages. It is distinct from correctness — an answer can be correct and ungrounded, or grounded in a source that is itself wrong — and it is the more useful production metric, because it can be computed without knowing the truth.
The practical procedure is claim-level: split the answer into individual assertions, check each against the retrieved context, and report the share supported. Both claim extraction and support checking can be automated with a model, and both introduce error, so the pipeline needs validating against human labels before its numbers are trusted. See AI hallucination detection and LLM-as-a-judge.
The complementary measure is the refusal rate on unanswerable questions. An evaluation set should deliberately include questions the corpus cannot answer. A system that answers them anyway has a fabrication problem that no groundedness score on answerable questions will reveal.
04 / ABLATIONS
Checking that retrieval is doing anything.Four cheap experiments establish what each component contributes. Run them once at the start of a project and again whenever the pipeline changes materially.
- No retrieval. Answer from the model alone. If quality barely drops, the retrieval layer is not contributing and the corpus may be adding nothing the model does not already know.
- Perfect retrieval. Supply the correct passage by hand. The gap to the live system is the retrieval headroom; the gap from perfect to correct is the generation headroom.
- Shuffled retrieval. Supply irrelevant passages. A system whose answers barely change is ignoring its context, which is a finding in itself.
- Varying k. More context is not monotonically better. Quality frequently peaks and then declines as relevant passages are diluted.
The second ablation is the one that directs effort correctly. Teams routinely spend months tuning prompts when the perfect-retrieval experiment shows that the ceiling is in the retriever, or tuning the retriever when the model is ignoring good context. Ten evaluation questions run by hand will usually tell you which. Held-out design for these sets is covered in held-out evaluation.
05 / QUESTIONS
Asked when a retrieval system underperforms.What is the most common cause of poor RAG quality?
+
Retrieval, not generation. In practice the answer passage is often absent from the candidate set, and the model is then blamed for an answer it had no way to produce. Measure recall@k before changing anything about the prompt.
How do you build a RAG evaluation set?
+
Draw questions from real user queries where possible, with answers located in specific passages so retrieval can be scored independently. Include unanswerable questions deliberately, and keep the set held out from prompt and retriever tuning.
Is more context always better?
+
No. Quality typically peaks at a moderate k and then declines as relevant material is diluted by marginal passages. Longer contexts also cost memory and latency. Find the peak empirically rather than defaulting to the maximum.
Should RAG be evaluated online or offline?
+
Both. Offline evaluation on a held-out set gives attribution and comparability; online signals — empty-retrieval rate, answers without sources, user escalations — catch the changes that offline sets miss. See AI model monitoring.
Does RAG remove the need for fine-tuning?
+
It solves a different problem. Retrieval supplies facts the model does not hold; fine-tuning supplies judgement it does not have. Systems that need current information and expert standards need both. See AI model fine-tuning.
Find the stage
that is failing.
Privilege AI works on retrieval-augmented generation and knowledge systems, including the evaluation that attributes a failure to a stage.