Cornell proposes DecisionRAG: traversing the document tree layer by layer with the Jev decision model

When Jev Meets RAG -> DecisionRAG: No Retriever Needed, Using a Decision Model to Traverse the Document Tree Layer by Layer

A paper from Cornell University targets the capability gap in RAG where "the retriever is too weak, and no matter how strong the generator is, it can't make up for it": it proposes DecisionRAG, which uses the decision model Jev (@typesafeai) to judge layer by layer on the corpus's built-in topic→document→paragraph tree whether to "dig deeper," with a single forward pass, no token generation, and candidates placed in the same context to be compared together. It does not depend on a first-stage retriever, nor does it rely on pre-built vector indexes at query time; compared with "dense retrieval + 8B reranking," it achieves 13.8 points higher R@1 on MultiDoc2Dial and 17.4 points higher Hit@1 on Natural Questions (2.68 million paragraphs), at about 0.7–1.8 seconds and 0.2–2.3 cents per query.

Paper: https://t.co/Fn27yyq0BA

Where does the problem come from? Generative models are getting stronger, but retrieval is still mostly BM25 or fixed vector nearest neighbors; evidence that is missed cannot be recovered by the generation side. Letting an LLM act as an Agent and search step by step works, but generating at every step is too slow and too expensive; rerankers can only rerank what the first stage has already retrieved. Decision models sit right in the middle: given a shared context and a set of questions, a single forward pass outputs a probability for each option (yes/no or multiple choice), without sampling tokens.

How does DecisionRAG work? Treat the corpus as a tree: leaves are paragraphs, and upper layers are help centers / products / clustered topics / document titles / sections. At each layer: 1. Parallel binary judgments: list all candidates at this layer in the shared context, ask for each "is it worth expanding for this question?" and take P(Yes)P(\mathrm{Yes})P(Yes), keep top-kkk, and prune the rest of the entire branch (irreversible). 2. After reaching paragraphs, score them binary again and take top-rrr. 3. Optional: multiple choice compares only the shortlist (usually top 20, top 5 for NQ) once more to stabilize the first place.

If a large layer does not fit in one request, split it into parallel batches; with about 50 candidates per batch, the benefit of shared context basically maxes out. When no ready-made hierarchy exists (e.g., Wikipedia), offline use title + first-sentence embeddings to cluster into 1000 clusters, then have a small model write a one-sentence cluster description. You can also add the top paragraphs from dense retrieval at the final stage to recover evidence missed by pruning.

Latency roughly scales with tree depth, not linearly with the total number of paragraphs: with a branching factor of about 100, a billion paragraphs theoretically take about five decisions and on the order of one second.

Key results Four benchmarks: MultiDoc2Dial, ConditionalQA, MTRAG (IBM Cloud), NQ-DAPR. Without task fine-tuning, DecisionRAG (Jev) beats dense + Qwen3-Reranker-8B on most main metrics, and trades wins with "dense top-20 + GPT-5 list reranking," overall not falling behind.

When the same generator (Qwen3.5-27B) consumes top-5 paragraphs downstream, answer quality also rises; on MTRAG, after adding dense candidates, the judged accuracy is about 47.3% (versus about 39% for the reranking baseline). Under the same tree and same budget, using GPT-5 to generate navigation decisions: accuracy is similar, but it is about 34–145 times slower and about 21–25 times more expensive.

After training Qwen3-8B into a retrieval decision model, MultiDoc2Dial R@1 reaches 60.7, about 5.7 points higher than the published task-trained retriever compared in the paper.

What actually helps in the ablations · Judging together with shared context versus judging each separately: MultiDoc2Dial R@1 about +7.7. · Navigating the whole tree often outperforms "using only the decision model to rerank dense top-100," especially Hit@1 on ConditionalQA / NQ. · Top-level pruning is key to saving money and also a risk point for missing evidence: on NQ, keeping about 15% of clusters can approach the R@10 of "flat scanning of all pages," with clearly lower cost and latency; keeping too few means entire pages can never be reached.

Limitations The tree structure and the quality of node descriptions determine the ceiling; pruning is irreversible. Mitigations are to increase the exploration budget, or merge in dense candidates at the end.