I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.
Hierarchical retrieval came third of five on a 240,767-word corpus, behind a much simpler baseline. I blamed chapter selection. A stage-level ablation over 80 questions and 400 scored rows says chapter selection cost me 27 questions — and neighbour expansion, the last stage, cost me 32. Turning that stage off raised recall from 0.3365 to 0.4771 and drew level with the baseline I was losing to.
Hierarchical retrieval is supposed to help on long documents. Pick the right chapter first, then search inside it. Narrow the haystack, then find the needle.
On my long-book benchmark it came third of five, beaten by a chapter-summary chain and by flat chunk RAG, and only barely ahead of naively grabbing the end of the book.
| method | context precision | context recall |
|---|---|---|
naive_first_context | 0.1475 | 0.1458 |
naive_last_context | 0.4150 | 0.3302 |
flat_chunk_rag | 0.3375 | 0.4302 |
chapter_summary_chain | 0.4000 | 0.4771 |
hierarchical_book_rag | 0.3475 | 0.3365 |
My hypothesis was error compounding, and it is the obvious one: if the first stage picks the wrong chapter, every later stage is searching the wrong text, and no amount of good ranking inside that chapter can recover.
That hypothesis turned out to be true and incomplete, which is a worse outcome than being wrong — it would have sent me to fix the right thing for the wrong reason, and stop there.
The ablation
Same corpus, same 80 gold questions, nothing about the manuscript or the questions touched. Five variants, 400 scored rows.
hier_current— the original pipeline: chapter selection, chunk retrieval, neighbour expansion.hier_no_neighbors— identical, with neighbour expansion switched off.chapter_summary_chain— the baseline that was beating it.hier_oracle_chapter— a diagnostic that is given the correct chapter and retrieves only inside it.hier_oracle_chapter_neighbors— the same, with neighbour expansion back on.
The oracle variants are not retrieval methods. They cannot be deployed — they read the gold chapter label. Their only job is to measure how much is lost before the chapter-selection stage versus after it.
What came back
| method | recall | precision | hit@1 | hit@3 | hit@5 |
|---|---|---|---|---|---|
hier_current | 0.3365 | 0.3475 | 0.2000 | 0.3875 | 0.4375 |
hier_no_neighbors | 0.4771 | 0.4000 | 0.2000 | 0.3875 | 0.4375 |
chapter_summary_chain | 0.4771 | 0.4000 | 0.2000 | 0.3875 | 0.4375 |
hier_oracle_chapter | 0.7844 | 0.6796 | 1.0000 | 1.0000 | 1.0000 |
hier_oracle_chapter_neighbors | 0.7688 | 0.6925 | 1.0000 | 1.0000 | 1.0000 |
Two things fall out of that table immediately.
Deleting a feature closed the entire gap to the baseline. hier_no_neighbors lands on 0.4771 recall and 0.4000 precision — the same numbers as chapter_summary_chain, to four decimal places. The method I thought was structurally worse was not worse. It was carrying a stage that was hurting it.
Look at the hit@k columns. They are identical across the first three rows: 0.2000, 0.3875, 0.4375. Chapter selection did not change at all between hier_current and hier_no_neighbors — it could not, it is the same code. Every point of recall that moved, moved downstream of retrieval, in the stage that pads results with adjacent chunks.
Counting the failures directly
Each of the 80 questions was assigned a failure type for the original pipeline.
| failure type | count |
|---|---|
neighbor_dilution — expansion made precision or recall worse | 32 |
wrong_chapter — the expected chapter was never selected | 27 |
right_chapter_wrong_chunk | 7 |
ok | 14 |
My hypothesis accounted for 27 questions. The stage I had not suspected accounted for 32. I would have fixed chapter routing, seen a real improvement, and never looked at the expansion step — because the improvement would have confirmed the theory I walked in with.
How much is routing?
Forcing the correct chapter raises recall from 0.3365 to 0.7844, and precision from 0.3475 to 0.6796. Oracle mapping succeeded for all 80 questions, so that is measured on the whole set rather than a subset.
That number is the ceiling this architecture has if routing were solved perfectly. It is a large gap, and it says first-stage selection genuinely is the dominant remaining constraint once the expansion bug is gone. But it is a diagnostic ceiling, not an achievable score.
The conditional recalls make the same point without an oracle. For hier_current, recall was 0.4000 when the expected chapter made the top 5 and 0.2870 when it did not. For hier_no_neighbors and the baseline, 0.6190 when it made the top 5 and 0.3667 when it did not. Getting the chapter right roughly doubles what the later stages can do — but only once expansion has stopped diluting them.
Neighbour expansion is not simply bad
This is the result I would most want someone to take away correctly, because the headline invites the wrong lesson.
With the oracle chapter, adding neighbours moves recall 0.7844 → 0.7688 and precision 0.6796 → 0.6925. Recall down slightly, precision up slightly. When you are already in the right chapter, expansion is roughly a wash and it can help precision.
The damage happens when expansion runs on top of an uncertain chapter choice. It crowds out the good chunks you did find with adjacent text that is only adjacent, not relevant. It is a stage whose value depends on the confidence of the stage above it — which is exactly the kind of interaction a single end-to-end score cannot show you.
So the finding is not "turn off neighbour expansion." It is treat it as a tunable stage rather than a default, and condition it on how sure the router is.
The generalisable part
A multi-stage retrieval pipeline reports one number, and that number is a sum over stages that can fail independently and in opposite directions. My pipeline had a stage that was helping and a stage that was hurting, and the aggregate said "hierarchical retrieval underperforms" — a conclusion about the architecture that was not true of the architecture.
You cannot interpret that failure without taking the stages apart. In practice the sequence that worked was:
- Disable each optional stage in turn. Cheap, and it found the larger of my two problems.
- Insert an oracle at each boundary. Not deployable, but it partitions the loss into "before this point" and "after this point."
- Only then attribute the failure.
If I had skipped to step three, which is where the instinct goes, I would have published "hierarchical RAG underperforms chapter-summary retrieval on long narrative corpora." That sentence would have been well-supported by my headline numbers and wrong about the cause.
What this does not show
Stated plainly, because the result is narrow and the temptation to widen it is real.
- One private narrative corpus, 80 gold questions, written from the corpus rather than by independent annotators.
- Evidence-term overlap scoring — a lightweight audit signal, not full semantic correctness.
- No confidence intervals. The package did not compute them, and I am not going to imply precision I did not measure.
- The oracle variants are diagnostic only. They read gold labels and are not production-realistic.
- This is not a universal rule against hierarchical retrieval. It is a demonstration that multi-stage pipelines need stage-level ablation before their failures can be interpreted at all.
Paper: Diagnosing Hierarchical Retrieval Failure in Long-Document RAG: A LongBook Verifier Ablation Study — Zenodo, June 2026.
DOI: 10.5281/zenodo.20692450 (concept DOI — always resolves to the newest version).
The public package ships the ablation script, the summary tables, the plots and the packaging script. It excludes the manuscript text, which protects the corpus and does limit full public reproducibility until a public-domain parallel corpus is added.
Related: retrieval as an unreported measurement instrument — what a retriever does not tell you about how little of the corpus it read.