Is RAG really dead? When the question names its document, Agentic Grep comes close to Agentic RAG. When the answer is spread across a library, it trails, 78% to 84%.
- Three setups. Classic RAG: one search, then a model writes the answer. Agentic RAG: an agent searches with RAG as many times as it needs. Agentic Grep: the same agent, with grep in place of RAG.
- The short answer: Classic RAG is dead, but RAG is not. Agentic RAG answered 15–16 more questions correctly than Classic RAG, and stayed ahead of Agentic Grep throughout, most of all on the mixed library, though neither gap over grep is significant alone.
- The first test: 150 FinanceBench questions over 368 reports, or 54k pages, with the same open-weight model throughout, qwen3.6-27b. Another model could change the picture.
- The same text for both. In the main comparison, grep and RAG searched the same parsed markdown, tables included. Compare them on anything else and you are testing the parser too.
- Agentic Grep beat Classic RAG: 85% right against 78%.
- Agentic RAG was at par, slightly above, with 88%.
- The step limit decided the result. With a 10-step cap, Agentic Grep scored only 81%. At 25 it caught up.
- Agents cut wrong answers by about half: 25 for Classic RAG, 14 for Agentic Grep, 11 for Agentic RAG.
- The catch: 147 of those 150 questions name their company, so the agent knew which document to open.
- The harder test: on 32 questions over a library of 383 documents, many needing several of them, Agentic Grep scored 78% and Agentic RAG 84%. The sample is small: the direction is clear, the proof is not.
- The full system adds more: VectoGrid, with a reviewer model and a path for questions about sets of things, scored 91% on the same questions.
FinanceBench graded against the documents, as our published results, with no reviewer model so that only the tool differs: our published system adds one and reaches 91%.
The argument so far is about code and chat logs.
Coding agents moved away from embeddings towards plain text search, and a claim followed: RAG is dead, an agent only needs grep. The published evidence is thin for documents. A May 2026 paper, Is Grep All You Need?, found grep often ahead of vectors, on 116 questions over long chat transcripts. Other work is on code. We found no head-to-head on a large collection of real business documents, with cost and time alongside accuracy.
VectoGrid is one of the systems compared, so we had every answer judged with the system names hidden, and we publish the result as it came out.
Given enough steps, grep comes close.
All three use the same model (qwen3.6-27b). Classic RAG runs one search and writes one answer. Agentic RAG and Agentic Grep share the same loop and the same page-reading tool, and differ only in their search: our hybrid search with a reranker at its default settings, or grep.
| Setup | Right of 150 | Wrong | Median time | Median cost | Total cost | Median steps |
|---|---|---|---|---|---|---|
| Agentic RAG | 132.5 (88.3%) | 11 | 8.7 s | 0.8¢ | $2.95 | 1 |
| Agentic Grep, up to 25 steps | 127 (84.7%) | 14 | 12.5 s | 1.3¢ | $3.84 | 7 |
| Agentic Grep, earlier run at 10 steps | 121 | 20 | 10.2 s | 1.2¢ | $1.93 | 7 |
| Classic RAG | 117 (78.0%) | 25 | 9.7 s | 1.2¢ | $2.08 | 1 |
A step is one action the agent takes after its first search, the last being its answer. Agentic RAG and Classic RAG are averages of two runs: 133 and 132, 117 and 117. Agentic RAG against Agentic Grep: p = 0.21 and 0.33 (paired exact McNemar). Agentic Grep against Classic RAG: p = 0.10. Time, cost and steps are from the latest run of each. Costs are model calls only, and a few long questions cost more than the medians suggest.
Why does an agent help RAG when its median is one step? Most questions are answered from the first search, but 53 of the 150 take more, and most of the questions Agentic RAG gets and Classic RAG misses are among those.
Those costs leave out what each needs before the first question. RAG needs every page embedded, a vector store and a reranker. Grep needs a folder of text. That is part of why it is a serious baseline.
On financial questions a wrong figure is the expensive outcome, so wrong answers are worth watching as closely as right ones. Letting the model search again halved them, whichever tool it had: from 25 to 11 or 14 on average.
The two were not getting the same questions right. Agentic Grep alone got 5 or 6, Agentic RAG alone got 11. Each has real wins: two different ways of getting to about the same place.
Both searched the same text
This matters more than it sounds. RAG searches the markdown our document reader produces, with tables intact. A comparison on anything else would partly measure the parser. So we gave grep the same markdown, one file per page. Anyone comparing these approaches should do the same.
What the agent did with grep
It was real grep: grep -i -E over one file per page. The agent wrote every pattern itself and could restrict a search to documents by name. It saw the number of matching lines and the first 30, then could read any page in full.
It behaved like a person with a search box. Most searches were plain words, and only 44 of 572 used any regular-expression syntax. 424 of the 572 were restricted to a document: it listed the file names, picked the report, searched inside it and read the page.
Ten steps made Agentic Grep look worse than it is.
An earlier run allowed 10 steps, and Agentic Grep scored 121. A rerun of the questions it split with Agentic RAG looked stable, and we were ready to report a clear win for RAG. Then we checked the 32 questions where grep had run out of steps, and gave them 25.
It went from 14 right to 23 on those questions: 10 gained, 1 lost. Many of its earlier failures were "could not find" answers on questions worded differently from the document. With more tries, it found other words. So the gap we nearly published was our limit, not grep's.
A step limit is a design choice, and in any comparison of agents it can decide the result. Anyone reporting agent results should say what the limit was and how many questions hit it.
Spread across a library, Agentic Grep falls behind.
147 of the 150 FinanceBench questions name their company, and 139 a period too. That is grep's best case: find the report, search inside it. Real questions are often not like that. People ask for every item of a kind, or to compare things across documents, without knowing which documents hold the answer.
So we ran Agentic Grep at its best (25 steps, file names, searches limited to documents) over a mixed library of 383 synthetic and public documents: 32 questions, some single lookups and many needing several documents. We ran Agentic RAG the same way, on the same markdown, and compared both with VectoGrid's earlier answers to the same questions, all three judged blind together. The synthetic documents and all the questions are our own.
| 32 questions, 383 documents | Agentic Grep | Agentic RAG | VectoGrid |
|---|---|---|---|
| Score | 25 (78%) | 27 (84%) | 29 (91%) |
| First set, 22 questions | 20 | 21.5 | 22 |
| Second set, 10 harder questions | 5 | 5.5 | 7 |
| Facts found on the second set, of 52 | 31 | 35 | 47 |
| Total cost | $1.21 | $1.59 | $1.12 |
A half point is a partly right answer. VectoGrid here adds a reviewer model and a path for questions about sets of things; the other two have neither.
Agentic RAG beat Agentic Grep on 5 questions and lost 1, which proves nothing alone (p = 0.22). VectoGrid then beat Agentic RAG on 3 more and lost none. So part of the 13 points between grep and the full system comes from search, and part from the system around it.
Where Agentic Grep fell short, the failures have a pattern. It left items out of lists, compared two things when the question needed all of them, answered no when the fact sat in a document it had not thought to open, and once said no answer existed when one did. Each is the same problem: a search for words finds what you already know to look for.
What this says, and what it does not.
- The harder test is small, and partly ours. 32 questions, one run, and the gap between Agentic RAG and Agentic Grep is not statistically significant. We used the 10 harder questions while building VectoGrid, so its score there may be flattering. The 22 of the first set were not used: Agentic Grep 20, Agentic RAG 21.5, VectoGrid 22, which points the same way and proves nothing on its own.
- The grep here is plain. It showed 30 lines, at most 5 a page, with no context lines and no file counts. A richer tool or a stronger model might do better, and file counts could help most where it struggled: knowing which documents to open.
- Neither collection is large, so this says little about scale. The reports are about 340 MB of text over 54,000 pages. The library is only 3.6 MB over 731 pages, so grep did not fall behind there because of size. Size still shows: 44% of grep's searches across all the reports matched more lines than the 30 it could show, against 19% of those limited to a document, and the worst matched 23,028 lines. A larger collection would make whole-collection searches noisier and slower, while a search inside one known document would stay the same. We measured neither.
- Few runs. Answers vary between runs, and more runs would narrow these figures. We cannot say the two agents are equal, only that our runs do not separate them.
- One model, one benchmark. Other agents may use grep better or worse, and the paper above found the agent harness matters a lot. FinanceBench is public, so models may have seen its questions.
- Grading. Claude read every answer with the setup names hidden. Six references the documents contradict are graded as in our published results, using the same six and no others. Other references also look doubtful, and those count against every setup equally.
- Permissions and proof. We only measured answers. Which documents a user may see, and showing the source of each figure, are separate questions for any real deployment.
How the test was run
- FinanceBench's 150 open questions over its 368 reports, unedited. qwen3.6-27b on Groq, no reviewer model. Search effort 1,600, the default.
- Agentic Grep read our reader's markdown, one file per page, at most 5 matches a page; a section spanning pages sits under its first page. Agentic RAG used bge-m3 embeddings, keyword search and a Nemotron reranker on Qdrant. Each agent could also read a page and list file names.
- The same agent with BM25 scored 108 at the 10-step limit, with 47 questions hitting it, so it is not comparable with the 25-step figure and we leave it out. We also tried a research keyword engine, in a setup too preliminary to report.
- Paired exact McNemar tests on the 150 questions. Every answer and verdict is in the download (CSV).
For our published system, see our FinanceBench result.
Deciding how your agents should search?
Published 7 November 2026 · Philippe Vergnet, VectoGrid