Privilege AIMKC2 results ↗

RAG evaluation

A RAG system fails in at least four places. Evaluating the whole pipeline tells you that it failed; stage-wise measurement tells you where.

READ THE DEFINITION ↓RECALL · PRECISION · GROUNDEDNESS

01 / DEFINITION

The short answer first.

What is RAG evaluation?

RAG evaluation measures a retrieval-augmented generation system at each stage — chunking, retrieval, ranking, generation — as well as end to end. Stage-wise measurement is what makes a failure attributable: an answer can be wrong because the passage was never retrieved, or because it was retrieved and ignored.

A single end-to-end score is the right thing to report and the wrong thing to debug with. Two systems can produce identical answer quality for opposite reasons: one retrieves well and reasons poorly, the other retrieves poorly and compensates with a strong model. Their improvement paths are completely different, and only stage-wise numbers distinguish them.

WORKED EXAMPLE

Separate a retrieval miss from an unsupported answer

A field technician asks what inspection interval applies to the revised coupling on a machine. The knowledge base contains the old manual, a service bulletin changing the interval and a catalog page that identifies the coupling. A RAG system can return a confidently wrong answer because it missed the bulletin, because it ranked the old manual first or because the generator ignored the revision condition in a correctly retrieved passage.

Give the retriever and generator different tests

Label the bulletin and the specific passage that supports the new interval. Run retrieval alone and inspect the first candidate set, the post-reranking order and the passages actually placed in the model context. Record the first stage at which the required evidence disappears. Finding the correct file in a broad candidate list is different from delivering the relevant passage to the generator.

Then supply the correct bulletin directly to the generator in a controlled test. Ask for the interval, the affected revision and the supporting quotation. If the answer still omits the revision, improving retrieval alone will not repair the behaviour. Include a question about an unaffected machine so the model has to preserve the boundary in both directions.

At release review, show one successful answer and one failure with their intermediate records. Keep retrieval success, answer support, citation accuracy and appropriate abstention in separate columns. This gives each team an acceptance test it can own and prevents a change in the final answer score from being automatically attributed to the retriever.

Keep the bulletin revision with the test so a later manual update does not silently change the expected inspection interval.

02 / THE FOUR STAGES

Stages of a RAG pipeline, what to measure at each, and the symptom of failure. Most teams measure only the last row.
StageMeasureSymptom when it fails
ChunkingWhether answers survive intact in one chunkAnswers that span a boundary are never found
RetrievalRecall@k: is the answer passage in the candidate set?Confident answers with no relevant source
RankingContext precision: is it near the top?The right passage is retrieved but crowded out
GenerationGroundedness and answer qualityCorrect sources present, conclusion unsupported

Chunking is the least measured and frequently the most consequential. A chunk boundary through the middle of a definition, a table or a clause makes the answer unretrievable at any k, and no amount of model improvement recovers it. The diagnostic is simple: for a sample of evaluation questions, check by hand whether the answer exists intact inside a single chunk. If it does not, the pipeline has a ceiling that no retrieval tuning will lift.

Retrieval recall is the next ceiling. If the relevant passage is not in the top k, the generator cannot use it, and the honest options are refusal or fabrication. Measuring recall separately turns a vague quality complaint into a specific, fixable number.

03 / GROUNDEDNESS

Sourced
is not the same as
supported.

Whether the answer follows from the sources.

Groundedness is the share of claims in an answer that are actually supported by the retrieved passages. It is distinct from correctness — an answer can be correct and ungrounded, or grounded in a source that is itself wrong — and it is the more useful production metric, because it can be computed without knowing the truth.

The practical procedure is claim-level: split the answer into individual assertions, check each against the retrieved context, and report the share supported. Both claim extraction and support checking can be automated with a model, and both introduce error, so the pipeline needs validating against human labels before its numbers are trusted. See AI hallucination detection and LLM-as-a-judge.

The complementary measure is the refusal rate on unanswerable questions. An evaluation set should deliberately include questions the corpus cannot answer. A system that answers them anyway has a fabrication problem that no groundedness score on answerable questions will reveal.

04 / ABLATIONS

Four cheap experiments establish what each component contributes. Run them once at the start of a project and again whenever the pipeline changes materially.

  • No retrieval. Answer from the model alone. If quality barely drops, the retrieval layer is not contributing and the corpus may be adding nothing the model does not already know.
  • Perfect retrieval. Supply the correct passage by hand. The gap to the live system is the retrieval headroom; the gap from perfect to correct is the generation headroom.
  • Shuffled retrieval. Supply irrelevant passages. A system whose answers barely change is ignoring its context, which is a finding in itself.
  • Varying k. More context is not monotonically better. Quality frequently peaks and then declines as relevant passages are diluted.

The second ablation is the one that directs effort correctly. Teams routinely spend months tuning prompts when the perfect-retrieval experiment shows that the ceiling is in the retriever, or tuning the retriever when the model is ignoring good context. Ten evaluation questions run by hand will usually tell you which. Held-out design for these sets is covered in held-out evaluation.

05 / QUESTIONS

What is the most common cause of poor RAG quality?

Retrieval, not generation. In practice the answer passage is often absent from the candidate set, and the model is then blamed for an answer it had no way to produce. Measure recall@k before changing anything about the prompt.

How do you build a RAG evaluation set?

Draw questions from real user queries where possible, with answers located in specific passages so retrieval can be scored independently. Include unanswerable questions deliberately, and keep the set held out from prompt and retriever tuning.

Is more context always better?

No. Quality typically peaks at a moderate k and then declines as relevant material is diluted by marginal passages. Longer contexts also cost memory and latency. Find the peak empirically rather than defaulting to the maximum.

Should RAG be evaluated online or offline?

Both. Offline evaluation on a held-out set gives attribution and comparability; online signals — empty-retrieval rate, answers without sources, user escalations — catch the changes that offline sets miss. See AI model monitoring.

Does RAG remove the need for fine-tuning?

It solves a different problem. Retrieval supplies facts the model does not hold; fine-tuning supplies judgement it does not have. Systems that need current information and expert standards need both. See AI model fine-tuning.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Tell us which buyer or operator question fails, what evidence should answer it and where your current pipeline loses that evidence.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Tell us which buyer or operator question fails, what evidence should answer it and where your current pipeline loses that evidence.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Find the stage
that is failing.

Privilege AI works on retrieval-augmented generation and knowledge systems, including the evaluation that attributes a failure to a stage.