Privilege AIMKC2 results ↗

Retrieval evaluation

Retrieval quality sets the ceiling for everything downstream. It is also the cheapest part of a RAG system to measure properly.

READ THE DEFINITION ↓RECALL · RANKING · JUDGEMENTS

01 / DEFINITION

The short answer first.

What is retrieval evaluation?

Retrieval evaluation measures whether a search component returns the right passages for a query, independently of what a model later does with them. It reports set-based metrics such as recall@k and rank-sensitive metrics such as nDCG, against human or model judgements of relevance.

Evaluating the retriever separately is worth the effort because it is fast, deterministic and cheap. A retrieval experiment runs in seconds against a fixed index, with no generation cost and no sampling variance, which means a day of retrieval tuning can cover far more ground than a day of end-to-end experiments.

WORKED EXAMPLE

Evaluate passage ranking for a parts catalog

A buyer searches for a replacement filter using the old commercial name. The correct compatibility table uses a newer model code and never mentions the exact phrase the buyer typed. A lexical search may return an obsolete listing first. A semantic retriever may find a current but incompatible product in the same family. The evaluation needs relevance judgements tied to the purchasing task, not just a list of pages containing similar words.

Label degrees of usefulness

Ask a catalog specialist to distinguish the authoritative compatibility passage, a helpful model-identification guide and an unrelated product description. Keep those graded labels with the query and the catalog version. Measure the rank the shopper actually reaches. A relevant passage at position twenty may not help an application that passes only five results to its next stage.

Compare lexical, semantic and combined retrieval on the same queries and index snapshot. Include old names, exact part codes and questions that should have no match. Review disagreements rather than assuming one retrieval method wins every query class. A wrong result for an exact part number can matter more than an improvement on several broad category searches.

Save the candidate lists and the relevance rule so a later index refresh can be evaluated against the same decision. When a product is superseded, revise the expected destination deliberately and label the change; otherwise a correct current result can appear to be a regression against an obsolete benchmark.

Further reading · BEIR retrieval benchmark

02 / THE METRICS

Retrieval metrics and what each is for. Recall@k is the one that bounds a RAG system; the rank-sensitive metrics matter when context budget is tight.
MetricQuestionUse when
Recall@kIs the answer anywhere in the top k?Feeding a generator: this is the ceiling on the whole pipeline
Precision@kHow much of the top k is relevant?Context budget is limited and dilution costs quality
nDCGAre the best passages ranked highest?Position matters, with graded rather than binary relevance
MRRHow far down is the first relevant result?A single correct answer, as in lookup-style queries
Context precisionWhat share of supplied context was used?Tuning k and reranking for a generation pipeline

For RAG, recall@k is the primary metric and the reason is structural: a passage that is not retrieved cannot be used, so recall places a hard ceiling on answer quality that no prompt or model change can lift. Precision matters second, because dilution degrades generation, and rank-sensitive metrics matter most where the context budget is small enough that position determines what survives truncation.

03 / THE JUDGEMENT PROBLEM

Every metric
depends on a label.

Where a retrieval test set gets its labels.

Retrieval metrics are only as good as the relevance judgements underneath them, and those judgements are harder to produce than the metrics suggest.

Two problems recur. The first is pooling bias: relevance is usually judged only for documents that some existing system retrieved, so a new retriever that surfaces a genuinely relevant passage nobody judged is scored as wrong. The standard mitigation is to pool candidates from several diverse retrievers before judging, and to re-judge newly surfaced results rather than assuming a miss.

The second is judgement quality. Relevance is graded, not binary, and two reasonable people disagree more often than expected. Measure inter-rater agreement before treating the labels as ground truth: if assessors agree on 70% of cases, a retriever scoring 70% against one assessor is performing at human level, not failing. Model-generated judgements are now practical and need validating against human labels in exactly the same way — see LLM-as-a-judge.

A cheaper alternative avoids judgements altogether: draw evaluation questions from passages. Pick a passage, write a question it answers, and the correct retrieval target is known by construction. This scales well and biases towards questions that are answerable by a single passage, so it should be supplemented with real user queries.

04 / WHAT USUALLY IMPROVES RECALL

↳ 01

Hybrid search.

Combining lexical and vector retrieval reliably outperforms either alone. Lexical finds exact identifiers and rare terms; vectors find paraphrase.

↳ 02

Better chunking.

Boundaries that respect document structure, with overlap. An answer split across two chunks is unretrievable at any k.

↳ 03

Reranking.

Retrieve broadly, then rerank a larger candidate set with a stronger model. Improves precision at a fixed context budget.

Hybrid search matters most in domains with specialised vocabulary and exact identifiers — matter numbers, clause references, product codes — where pure vector search is unreliable precisely where accuracy matters most. A near-duplicate identifier is semantically almost identical and legally entirely different, which is a failure mode lexical matching does not have.

Query rewriting and expansion are also frequently worth the cost, particularly for short or ambiguous queries, and they are the easiest place to introduce a confidentiality mistake: a rewrite step that calls an external model transmits the query outside the boundary. See RAG security.

05 / QUESTIONS

Is vector search enough on its own?

Rarely, in specialist domains. Pure vector search handles paraphrase well and exact identifiers badly, and identifiers are often where precision matters most. Hybrid retrieval combining lexical and vector scoring is the usual answer.

What recall@k should we aim for?

High enough that retrieval is not the binding constraint on answer quality — which means measuring the perfect-retrieval ablation and comparing. Chasing a universal target number is less useful than knowing whether the retriever or the generator is your ceiling.

How many queries does a retrieval evaluation set need?

A few hundred gives stable aggregate numbers; per-category reporting needs enough in each category to be meaningful. Passage-derived questions make it cheap to build a large set, and real user queries keep it honest.

Does chunk size matter more than the embedding model?

Frequently, yes. A poor chunking strategy caps recall regardless of embedding quality, and chunking is cheaper to change. Test chunk size and overlap before investing in embedding model comparisons.

Can retrieval be evaluated without human labels?

Partly. Passage-derived questions give labels by construction, and model-generated judgements scale well. Both need validating against a human-labelled sample, because both introduce systematic biases of their own.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Share the kind of query your users issue and an example of a relevant passage your search system misses or buries.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Share the kind of query your users issue and an example of a relevant passage your search system misses or buries.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Fix the ceiling
before the prompt.

Privilege AI builds retrieval and knowledge systems, and measures the retriever separately from the model that consumes it.