A billion documents How we would build document AI for a library that size
- At a billion documents, storing and searching the text is a solved problem: a keyword index such as Elasticsearch or OpenSearch does it, and should be the backbone.
- The hard parts are elsewhere: searching only what the question is about, getting the right passage into the top 3 rather than the top 100, and reading every document well. Every operation repeated across the collection needs a cost budget: the design embeds each passage once, writes no passage text with a generative model, and keeps the expensive relevance scoring for question time.
- This is a design, not a measurement: we say below which parts we have measured, on what, and which we have not.
Our retrieval tests run on thousands of passages; our scaling work goes to a million documents. Some organisations hold a thousand times more: every statement, form, letter, contract and report they have ever produced. Here is how the retrieval we measured in our comparison with Elasticsearch would carry over to that size, and what would have to change.
Ten billion passages.
Most documents in a large organisation are short: a letter, a statement, a form. Say ten searchable passages each on average, so ten billion passages for a billion documents. Stored as vectors for meaning search, at 1,024 numbers each:
| 10 billion passages × 1,024 numbers | Raw vector payload |
|---|---|
| Full precision (32-bit) | 41 TB (40.96) |
| Half precision (16-bit) | 20 TB (20.48) |
| 1-bit, exact vectors kept on disk for a second look | 1.3 TB (1.28) of bits, 41 TB of exact vectors on disk |
Calculated, not run, in decimal units. The index structures, metadata, any corrective data a quantized index keeps beside the bits, the text and replicas come on top; how much sits in memory depends on the engine and its settings.
The keyword index is the proven part: engines like Elasticsearch and OpenSearch have run keyword search at this scale for years (their storage depends on mappings, settings and the documents themselves). The vectors are where the new bill is: every passage is embedded once, and how its vector is stored decides the memory.
Six decisions.
A question goes through four stages; two more decisions are made once, at ingest and in testing.
1. Scope before search
Many operational questions concern a small part of the library: one customer's file, one product's terms, one set of rules, one period. Three different mechanisms narrow a search, and they should not be confused:
- Access control excludes what the user may not read. The organisation enforces it anyway; VectoGrid writes every document with its owner's access label and filters every search by it.
- Scope filters restrict the search to a known customer, period or document type, from metadata written at ingest, when the question or the application says which.
- Ranking boosts favour likely sources without excluding the others: VectoGrid keeps a short description of every document and lifts the passages of the documents closest to the question. This is the one we measured: from 72 to 57 questions without their evidence in the top 3.
When the scope is known, applying it before retrieval leaves fewer near-identical passages competing for the top places; we have not measured that gain at scale. Collection-wide questions, for discovery, audit or compliance, need a strategy of their own: searching in stages, or by partition and merging.
2. Keywords and meaning, every passage, stored for its use
The keyword index covers every passage: it is cheap, robust, exact on codes, numbers and names, and it is what Elasticsearch does best. Meaning search (vectors) is what finds a passage worded differently from the question, or in another language; in our tests it cut the misses from 103 to 72 when merged with keywords. Every passage is embedded once, at ingest: the one model step every passage needs, budgeted as such. What changes with scale is how the vectors are stored:
- 1-bit vectors with a second look for the bulk: 32 times less memory, and in our measurements on three engines, 99.3% to 99.5% of the exact top 10 kept when the 400 best candidates by the bits are scored again against the exact vectors on disk to keep 10 (details). That measures agreement with full-precision vector search, not whether the passages found answer the question.
- Half-precision vectors as well for the documents asked about most: the recent, the active, the frequently consulted.
- Vectors computed by an open-weights multilingual model that runs on the organisation's own machines, so neither documents nor questions leave them.
3. A reranker per question
In our tests the reranker was the largest single step: from 57 questions without their evidence to 29. It reads the question and each of the first stage's best candidates together, and re-orders them. Its cost depends on the number and length of the candidates it scores per question, not on the size of the library. A 1-billion-parameter open-weights cross-encoder runs on one GPU; questions are batched; the candidates it reads are bounded (each first-stage list's best 40).
4. Enrichment without models
The last step in our tests, from 29 misses to 4, came from a line written for every passage: what it depends on but does not say, carried from what the document says earlier. Every row of that comparison read the documents the same way, with the same corrected reading, so the step is the line's. VectoGrid writes it without a generative model, as part of reading the text.
The alternative, a model writing such a line for every passage, is what many systems do. We measured it at about $0.27 per 1,000 passages with a small hosted model: about $2.7 million for ten billion passages, paid again each time the instruction changes, before the time it takes, which depends on the throughput bought. In our tests the model-free line did as well overall. At this scale the measured cost is the argument: a step repeated ten billion times has to earn its price.
5. Read every document well, family by family
Two fixes made during our tests were in reading PDFs, not in search: section titles set in bold inside a paragraph block now recognised as titles, and a table of key figures read with its row labels and periods instead of as bare numbers. On the full configuration they rescued three questions that no search setting could. A passage read badly is lost to every layer above it, and no reranker recovers a label that was never read.
A billion documents are not a billion different layouts. They come in families: each kind of statement, form, letter or report printed from the same template. So reading quality is measured per family, on a sample: does every page keep its headings, does every table keep its labels, are scanned pages read. A fix for a family fixes millions of documents at once.
6. Measure the ladder, per family
Every step above is measured the same way: real questions over sampled documents, some written blind to any search result, each step added on its own, and the count of questions whose evidence is not in the top 3, rather than an average. This is how we know which of our steps does what (the ladder), and it is the only way to know whether a change helps a billion documents or only the ten someone looked at.
Keeping it current
A library this size changes every day: documents arrive, are replaced, deleted, and their permissions change. Each change has to reach every index. In VectoGrid a document written again keeps the stored vector of every passage whose text did not change, so a correction re-embeds only what changed; a deletion removes all of a document's records; a permission change rewrites labels without touching vectors; and moving a collection to a larger cluster copies the stored vectors instead of embedding again. How long ingestion, updates and rebuilds take at a billion documents is not measured, and has to be planned with the hardware.
Every answer traced to its passage.
At a billion documents nobody can check an answer by browsing. So every value VectoGrid gives is quoted from the passage it came from, located on its page and checked against it; the enrichment line beside a passage helps find it but is never accepted as evidence. A reviewer, an auditor or a regulator sees the source of each value, not just the value.
All of it runs on infrastructure the organisation chooses, with open-weights models: the keyword index, the vectors, the reranker and the reading.
What we have shown, and what we have not.
- Measured: each retrieval step on 1,000 questions over real and generated documents in English and French (the comparison); a keyword index scoring the same on OpenSearch as on VectoGrid's own; 1-bit vectors with a second look on three engines over 100,000 passages (scaling); the cost of a model-written line per passage.
- Calculated, not run: the vector payload for ten billion passages, and the cost of model-written lines at that size.
- Not yet measured: anything at a billion documents; retrieval accuracy as a library grows; the gain of scope filters; reading quality across document families we have not seen; the embedding and reranker throughput under production load; ingestion, update and rebuild times.
Building search over a very large library?
Published 1 November 2026 · Philippe Vergnet, VectoGrid