As of November 2026, this is, to our knowledge, the highest published, verifiable FinanceBench score using only open-weight models and unedited questions. All 150 answers can be downloaded, along with the 14 we did not get.
*Graded against the documents. Six decisions differ from the benchmark's reference answers, all shown in the download. Against the references exactly as written: 130 of 150 (87%), up from 128 in October.
| October 2026 | November 2026 | |
|---|---|---|
| Score, graded against the documents | 135 of 150 (90%) | 136 of 150 (91%) |
| Score against the references as written | 128 of 150 (85%) | 130 of 150 (87%) |
| Model calls, all 150 | about $20 | $2.97 |
| Time per question | about 1 minute | about 12 s (mean 26 s; one in ten over 58 s) |
| All 150 questions | about 1 h 20, twelve at a time | about 15 minutes, four at a time |
| How an answer is produced | two independent methods and a referee | one agent and a reviewer |
| Models | gpt-oss-120b, Qwen 3.6 27B, Mistral Large 3 | Qwen 3.6 27B, Mistral Large 3 |
Both runs: the same 150 questions, unedited, over all 368 reports, with no hint of which report each one concerns. Search also uses bge-m3 and NVIDIA's Nemotron reranker. Partial answers receive no credit.
This started with speed, not accuracy. We set out to make search faster, and got a search of the full library down to about 27 ms once embedding and reranking scores are cached. That headroom let every search look wider in the same time, and it raised a question: a system that can't find the evidence needs a lot of machinery to compensate, with extra searches, second opinions and a referee, but what if it can find it?
So we tried the plainest thing possible: one search and one model call, no agent. It scored 76% on its own, at about 5 seconds and about a cent a question. From there we worked up, adding a layer only where the one below fell short.
The top results used to include several hits from the same part of a report. Now each result comes from a different part, so the same few results carry more evidence. The agent reads each passage whole, because a match can sit anywhere in it.
Reading our own stored pages, we found that 69 of the 168 pages holding benchmark evidence had lost their table: every number sat on its own line, and the column headings no longer lined up with them. A model cannot reliably tell 2022 from 2021 that way. We now rebuild those tables from the position of each word on the page. 59 of the 69 came back as tables and the other 10 turned out to be prose, with no figure altered in a check of 1,364 decimal values.
October's two methods and referee are gone. One agent, Qwen 3.6 27B, has a single tool, the search, and up to six steps: it decides what to look for, when it has enough, and when the documents don't say. Most questions take one or two steps. Mistral Large 3 then reviews each answer against the passages it rests on. A doubt goes back to the agent, which can change its answer or keep it, because reviewers are wrong too. That happened on 54 of the 150 questions.
Graded against the documents, like the 91%, one run each. The first row was graded mainly from the opening of each answer, the others from the whole answer, and its cost is an estimate.
Simpler has a cost, and it should be plain. This version has no calculator: the model does its own arithmetic and the reviewer checks it. It has no financial reference library.
A second reranking pass, a financial reference library, a calculator tool, a faster host for the agent, having a second model write the answer, and a checker for wrong-year columns, which only raised false alarms once the tables were fixed.
Every miss is in the download with a note. In plain words:
| Group | Question | What happened |
|---|---|---|
| Our mistakes | JPMorgan, lowest segment revenue | Compared the four main segments and left out Corporate, which was the lowest. |
| Pfizer, growth in PPNE | Did not recognise the abbreviation for property, plant and equipment. Said yes, the right call, about the wrong measure. | |
| Best Buy, store count | Counted only the large-format stores, not all of them. | |
| Ulta Beauty, wages | Took FY2023 to mean the year ending January 2024 and gave up. The reference means January 2023. | |
| Boeing, gross margin trend | Argued the measure was the wrong lens and said no. The reference says it improved. | |
| 3M, liquidity | Used a narrower measure than the reference (0.85 against 0.96) and called the position healthy. | |
| Netflix, EBITDA margin | Added content amortization to depreciation: 56.8% against 5.4%. | |
| Arguable | J&J, earnings guidance | Followed guidance as it was raised during the year. The reference uses January's. |
| PepsiCo, restructuring costs | Answered 0, as the question instructs when there is no separate line. The reference takes $411 million from elsewhere in the report. | |
| Best Buy, best category | Gave the biggest category. The reference means the fastest growing. | |
| AES, return on assets | Used total earnings where the reference uses earnings attributable to the company. | |
| Unanswerable as asked | Pfizer, "this 10K" | The question names no year. It picked 2018; the reference is the 2021 report. |
| MGM, Adjusted EBIT coverage | MGM does not report the figure. It derived -3.25x; the reference says zero. | |
| JPMorgan, payout in liquidation | We could not derive the reference's $66.56 from the report. It answered $82.20. |
These groups are our own reading. None of the 14 is counted as correct. Unlike the six decisions above, where the documents contradict the reference, nothing here is credited to us.
The first full run on the rebuilt tables scored lower than this one. Reading the wrong answers, we traced some to figures that were not in the reports at all. A new version of a PDF library on our test machine moved the punctuation of table cells to the end, so 1,615.9 was stored as 16159 , . with no error raised. About 6,000 stored passages in 80 reports were affected.
We pinned the library to the version that reads them correctly, added a test on a real statement page that fails on the bad version, read those reports again, and threw that run away. A silent error like this is the reason every answer has to be traceable to its page.
| System | Library | Reported score | Models | Answers published |
|---|---|---|---|---|
| ATG | 368 reports | 92%, with 11 questions edited (company and period added) | not named; providers include OpenAI and Google | yes |
| VectoGrid, November | 368 reports | 91% | open-weight (Qwen 3.6 27B, Mistral Large 3) | yes |
| VectoGrid, October | 368 reports | 90% | open-weight (gpt-oss-120b, Qwen 3.6 27B, Mistral Large 3) | yes |
| Mistral Agentic Search | 368 reports | 86% | open-weight (Zhipu's GLM-5.2) | no |
| Mistral Agentic Search | 368 reports | 79% | open-weight (Mistral Medium 3.5) | no |
| Dewey | 85 reports | 83.7% | Claude Opus 4.6 | yes |
Each score as reported by its authors, with their own judging, as listed in our October article. Different libraries, judges and question edits prevent a single like-for-like ranking. Not listed: LinqAlpha reports 97.2% on 144 of the 150 questions, without publishing its answers.
Bring the questions your team spends hours answering.
Measure VectoGrid on the work that matters to you: accuracy, evidence quality, speed and cost.