VectoGrid scores 90% on FinanceBenchusing only open-weight models
- VectoGrid answers questions about company reports and shows the page each answer comes from.
- On FinanceBench, 150 real analyst questions over 368 reports, it scored 90%: the best published result we know of with open models a company can run on its own servers.
- About 13 cents a question, and every answer can be downloaded with its sources.
As of October 2026, this is, to our knowledge, the world's highest published, verifiable FinanceBench score using only open-weight models and unedited questions. Every answer is available to inspect, with its supporting source quotes and calculations.
*Graded against the documents. Seven decisions differ from the benchmark's reference answers, all shown in the download.
Where financial AI proves itself.
In sport, a shared course and a stopwatch let performance speak for itself. For AI on financial reports, FinanceBench is that course: real analyst questions, real company filings, and reference answers written by financial experts. Created by Patronus AI, it is a standard benchmark for financial-document AI. Mistral uses it to show its search technology, Dewey to evaluate systems built on Claude Opus, and researchers to measure progress.
A common examination. Real company filings. Answers you can check.
The questions ask for the work an analyst does every week:
A reported figure
What was 3M's capital expenditure in 2018?
A ratio, and what it means
Does AMD have a healthy liquidity profile based on its quick ratio?
A change
How did operating margin change, and what drove it?
A business call
Is this company capital-intensive?
Examples of benchmark question types, shortened for display.
The answers sit in tables, footnotes and disclosures. A calculation may need figures from several statements. And the trap: the same company appears across many years, with near-identical tables holding different numbers. A system has to find the right evidence, choose the right period, understand what the figure means and calculate it correctly. When the benchmark was introduced, GPT-4 Turbo with retrieval answered just 19% correctly.
Open models. Serious financial analysis.
Indexing the full library, 54,000 pages, took about an hour on a single 8-core server. We then ran all 150 questions over it twice, at about $20 in model calls per run (about 13 cents a question) and about 1 h 20 per run with twelve questions at a time. Partial answers receive no credit.
Metered answering model calls per full run
Published estimated cost per configuration
FinanceBench: published results
| System | Library | Reported score | Models | Time to answer one question | Answers published |
|---|---|---|---|---|---|
| ATG | 368 reports | 92%, with 11 questions edited (company and period added) | not named; providers include OpenAI and Google | 11 s on average (fast mode, their figure) | yes |
| VectoGrid | 368 reports | 90% | open-weight (gpt-oss-120b, Qwen 3.6 27B, Mistral Large 3) | about 1 min (all 150: 1 h 20, twelve at a time) | yes |
| Mistral Agentic Search | 368 reports | 86% | open-weight (Zhipu's GLM-5.2) | not stated | no |
| Mistral Agentic Search | 368 reports | 79% | open-weight (Mistral Medium 3.5) | not stated | no |
| Dewey | 85 reports | 83.7% | Claude Opus 4.6 | not stated (all 150: 5–10 h, their estimate) | yes |
Each score as reported by its authors, with their own judging. Different libraries, judges and question edits prevent a single like-for-like ranking. VectoGrid takes about a minute per question because every answer is produced twice, by two independent methods, and checked before it is shown.
Not listed: LinqAlpha reports 97.2% on 144 of the 150 questions, after leaving out the six that name no company or fiscal year, without stating how documents were provided to the system and without publishing its answers.
Find it. Follow it. Check it.
The page you would have looked for.
Ask in your own words. VectoGrid searches every report at once and brings back the right passage, in the right report, for the right year, even when the table looks the same as last year's.
The figures, the way you would.
It reads what a number depends on, fetches the second figure from another statement or another report, and does the arithmetic in code, not in its head.
Its own work, before you see it.
Two independent methods answer every question and a referee chooses. Every answer comes with its source quotes, so checking it takes seconds, not an afternoon.
1. The grind toward retrieval perfection
An AI cannot reason from evidence it never finds. So before answering a single benchmark question, we spent the work on retrieval, against a test of our own: 1,000 questions, scored by recall@3: a question counts as found only when the passage holding its answer comes back in the first three results. The questions span about 40 documents in English and French, some real reports and some built specifically to trip search up. Every change had to raise the count, or it was thrown out.
Plenty was tried and thrown out along the way: a reranker eight times larger (worse overall, twelve times slower), translating every question (no gain), a vision model re-reading pages (one fix, two breaks).
2. The agentic layer for answering
On top of retrieval we built two ways of answering. A pipeline plans the question, reads each value twice and computes in code. An agent decides what to search and read next, with a calculator and a financial reference built from public material: the SEC's Beginners' Guide to Financial Statements, public ratio definitions from Wikipedia, and the US GAAP taxonomy. A referee chooses between their answers, after two checks: an answer resting on a different year's report than the one asked about loses, and so does a verdict that contradicts its own figures. The agent's quotes are matched word for word to the pages it actually read.
3. Open-weight models throughout
gpt-oss-120b and Qwen 3.6 27B answer, Mistral Large 3 referees, bge-m3 embeds and NVIDIA's Nemotron reranks. All of them can run on your own servers, so your documents never have to leave the building.
Every figure, back to its page.
This is the heart of VectoGrid. An answer is only as good as your ability to check it, so every figure comes with the passage it was read from, and with a picture of the page itself, the line highlighted. You see what the AI saw, in the original document, in one glance.
What is the FY2018 capital expenditure amount (in USD millions) for 3M?
The FY2018 capital expenditure (purchases of property, plant and equipment) for 3M was $1,577 million, as reported in the Consolidated Statement of Cash Flows under Cash Flows from Investing Activities.
Purchases of property, plant and equipment (PP&E) (1,577)
All 150 answers are in the download: the question as published, the reference answer, our answer, the passage it rests on, and our verdict, with a note wherever it differs from the reference.
How we tested
- The 150 questions exactly as published, with no hint of which report each concerns. The system searches all 368 reports as they are.
- Every answer marked by hand against the documents. Seven decisions differ from the benchmark's reference answers; all are shown in the download.
- No special cases: nothing in the system is written for a particular company, ratio or question. FinanceBench has been public since 2023, so we welcome a test on questions we have never seen.
Built to be audited.
Runs on your infrastructure.
Every model is open-weight: answering, refereeing, embedding and reranking. The whole stack can run on your own servers, so your documents never have to leave the building.
Provenance by design.
Every figure is traced to the passage it was read from. Quotes are checked word for word against the pages the system actually read; any that can't be found are dropped, and the published answers show exactly where that happened.
A transparent benchmark run.
This benchmark used public company reports, with metered calls to hosted copies of these same open-weight models. In deployment, they run wherever you decide.
Bring your documents.
Bring the questions your team spends hours answering.
Measure VectoGrid on the work that matters to you: accuracy, evidence quality, speed and cost.