Retrieval, Measured: Where RAG Actually Breaks

A team ships a RAG feature, wires up an eval, and gets one number back: accuracy is 61 percent. Then a prompt change or a chunking tweak drops it to 54 percent, and nobody can say why. Was the wrong passage retrieved this time? Or was the right passage retrieved and the model just answered badly? The aggregate number cannot tell you, so the team guesses, usually by reaching for a bigger model or a fancier prompt, which is exactly the wrong lever if the problem was never generation at all.

This is the default state of RAG evaluation at most companies: one accuracy score, no diagnosis. It feels like measurement, but it answers the wrong question. You do not need to know whether the system got the answer right in aggregate. You need to know, for every miss, which half of the pipeline broke.

The thesis of this post, and of the small evaluation kit behind it, is simple: most RAG failures are retrieval failures, not generation failures, and you can prove it with numbers instead of a hunch. This post is the retrieval half of that proof, and it is fully reproducible with no API key at all. The generation half, where an LLM judges answer correctness and faithfulness, is the forthcoming Part 2, and I want to be upfront about that split rather than blur it: everything below is retrieval-only, keyless, and already committed to the repo.

The decomposition

A RAG pipeline has two jobs done by two different components. Retrieval has to find the right evidence. Generation has to turn that evidence into a correct answer. A single accuracy number conflates both, so a drop in accuracy is consistent with at least two very different root causes: the retriever stopped finding the right passages, or the generator started ignoring or misreading passages it was handed correctly.

The fix is not a better single metric. It is refusing to collapse the two questions into one. Retrieval quality gets its own metrics, measured against the gold passages a question actually needs. Generation quality gets its own metrics, measured against the gold answer, conditioned on what was actually retrieved. Once you have both, a regression tells you where to look before you touch a single line of code.

The other piece of the decomposition is a ladder: a sequence of configurations where each rung changes exactly one variable from the rung before it (the retriever, the embedder, the context width, the reranker, the fusion strategy). Change one thing at a time and any metric delta between two rungs is attributable to that one thing. Change three things at once, the way most teams "try a new RAG stack," and you learn nothing about which change did the work.

The benchmark: HotpotQA distractor

The numbers below come from HotpotQA's distractor setting: each question ships with 10 candidate paragraphs, exactly 2 of which are the gold supporting passages and 8 of which are distractors, plus a gold answer. That is a deliberately small, clean search space. There is no Wikipedia-scale index to build, no ANN tuning, no infrastructure between you and the numbers, and it runs in minutes on a laptop.

What makes HotpotQA the right benchmark for this argument is that most of its questions are multi-hop: answering them requires combining evidence from two separate passages, and the second passage is frequently only findable after you have read the first. A retriever that ranks everything in a single pass has no mechanism to "notice" it needs a second, related fact, so multi-hop questions are exactly where single-shot retrieval is structurally weakest.

That is why this evaluation defines a metric beyond plain recall: sufficiency@k, the fraction of questions where all gold passages land in the top-k results. Recall@k asks "on average, how much of the gold evidence did we find." Sufficiency@k asks the harder, more honest question for multi-hop: "for this specific question, did we find everything it needs, together, at once." A system can have decent average recall while still failing most multi-hop questions outright, because it is retrieving one gold passage here and a different one there, never both for the same question.

The ladder, one variable at a time

These are the committed keyless retrieval results (HotpotQA distractor, n=300, seed=7):

Six things fall out of these numbers.

First, plain lexical search (L0, BM25) retrieves both gold paragraphs for only 23.3 percent of questions at k=2. That is the multi-hop miss, quantified rather than asserted: for roughly three out of every four multi-hop questions, BM25 alone never assembles the full evidence set the question needs.

Second, dense retrieval helps, and helps in a predictable order. Moving from lexical (L0) to a small dense embedder (L1) lifts recall from 0.530 to 0.625 and sufficiency from 0.233 to 0.333. Moving to a stronger embedder (L2) lifts it again, to 0.670 recall and 0.397 sufficiency. The gain from small to strong embedder is real but modest next to the gain from lexical to dense in the first place.

Third, the reranker (L4) is the best individual component by ranking quality: it posts the best MRR of any retrieval configuration, 0.955, meaning when it has the right passage in its shortlist, it puts it first almost every time. Its sufficiency at k=2, 0.443, is also the best of any real retrieval rung run at k=2.

Fourth, and this is the counterintuitive one, none of that reranking sophistication is what gets sufficiency the highest of any real retrieval rung. That distinction belongs to L3, which is just L2's strong embedder with the context window widened from k=2 to k=5. No reranker, no fusion, just more slots. Sufficiency jumps to 0.740, more than triple L0's 0.233 and well past L4's reranked 0.443.

Fifth, that gain is not free. L3's precision at k=5 is 0.345, because widening the net pulls in more distractors along with the extra gold passage. You are trading precision for the property that actually matters on multi-hop questions.

Sixth, fusion does not automatically help. Hybrid BM25-plus-dense with reciprocal rank fusion (L6) actually underperforms plain dense retrieval here, at 0.607 recall and 0.320 sufficiency versus L2's 0.670 and 0.397. Averaging in a weaker lexical ranker drags the combined result down rather than complementing the stronger dense one.

Reproduce it yourself (no key)

Every number above comes from a run you can execute yourself, with no API key, in a few minutes:

uv sync
uv run pytest -q
uv run rageval --corpus hotpotqa --stage retrieval --config all -k 2 -n 300

The run is deterministic by construction: a fixed seed (seed=7), pinned embedding models, and cached embeddings, so the same inputs produce the same numbers every time. The scorers themselves are proven in both directions before a single API dollar is ever spent, gold input scores perfectly, adversarial input scores zero, the same "safe to run twice" discipline I wrote about in the idempotency masterclass applied to an eval harness instead of a payments API. The full source, tests, and committed results are in the public repo: rag-eval-ladder.

What this can't measure

This section is a deliverable, not a footnote.

1. This is the retrieval half only. Answer correctness (EM and token-F1), faithfulness judging, and the full 2x2 diagnostic below all need an API key to run the generation stage, and none of them are done yet. They are the forthcoming Part 2, not a claimed result here.
2. HotpotQA's distractor setting is easier than open-corpus retrieval. Ten candidate paragraphs per question is a much smaller search space than millions of documents in a real index. The relative ordering of the ladder (dense beats lexical, wider k beats a reranker on sufficiency, hybrid fusion can underperform) should generalize. The absolute recall and sufficiency numbers will not transfer directly to a production-scale corpus.
3. LLM-judged faithfulness, when Part 2 lands, will be fallible by construction. An LLM judge marking answers as supported, partial, or unsupported is itself a model that can be wrong. It will ship together with its own measured accuracy, precision, and recall against a labeled set, and it will never be presented as ground truth.
4. The 2x2 diagnostic is a framework here, not a result. No counts exist yet, because the generation stage has not run. What it will do, once it does, is cross whether evidence was retrieved (sufficiency@k) against whether the answer was correct:

The cell to watch is the bottom left: a correct answer produced without the evidence that was supposed to justify it. That is not a win, it is a model guessing correctly (often from parametric memory) in a way that looks identical to a working system until the question changes slightly and the guess stops landing.

Close

The two ideas worth taking away are the decomposition and the 2x2. Neither one is specific to HotpotQA, or to this repo. Any RAG pipeline can be evaluated by separating "did retrieval find the right evidence" from "did generation use it correctly," and any generation stage can be diagnosed with the same evidence-versus-correctness cross tab once you have gold labels or a reasonable proxy for them. The kit is small enough to read in a sitting and fork onto your own pipeline, and that is exactly what it is for.

Part 2 picks this up on the generation side: EM and F1 against HotpotQA's gold answers, an LLM faithfulness judge shipped with its own measured reliability, and the 2x2 filled in with real counts instead of cell labels. For now, the retrieval half stands on its own, reproducible from a cold clone with no key and no waiting.

For background on the questions themselves, see the HotpotQA project page.