From one report to millions How VectoGrid prepares document search for growing libraries
- An AI answer is only as good as the passage it finds. As a library grows, the search index has to scale, and its limit is memory.
- Three steps: one small server up to about 10,000 documents; the same server with 1-bit vectors up to about 100,000, at 0.6 percentage points lower recall in our test; a cluster for a million, at full accuracy, without changing a line of the application.
Document AI demos run on a handful of PDFs. Real libraries do not: a fund's data room, a firm's archive or a regulator's filings run to tens of thousands of documents, and keep growing. Here is how VectoGrid is built to grow with them, what each step costs, and which of those costs we have measured.
18-page documents (~150 passages each). Accuracy measured on 100,000 passages and real questions; the larger sizes are calculated from it. Details below.
Every document becomes vectors, and vectors live in memory.
To search by meaning, VectoGrid cuts each document into passages and turns each passage into a vector: 1,024 numbers that place it in a "space of meaning", where passages about the same thing sit close together. A question becomes a vector too, and the search returns the passages nearest to it. That search is fast only when the vectors, and the index over them, sit in memory.
How many passages? On FinanceBench, 368 annual reports and 54,000 pages, VectoGrid's index holds 445,810 searchable passages: about 8 per page, so about 150 for an 18-page document. Each vector at full precision takes 4 KB. The arithmetic follows:
| 18-page documents | Passages | Full precision | Half precision (default) | 1-bit |
|---|---|---|---|---|
| ~700 | 100,000 | 0.4 GB | 0.2 GB | 0.01 GB |
| ~6,700 | 1 million | 4 GB | 2 GB | 0.13 GB |
| ~67,000 | 10 million | 41 GB | 20 GB | 1.3 GB |
| ~1 million | 150 million | 614 GB | 307 GB | 19 GB |
Raw vector memory at 1,024 dimensions; the index, the text and the application need room on top.
Three levers move that memory, and VectoGrid uses them in order of cost.
Half precision: half the memory, nothing measurable lost.
Each of a vector's 1,024 numbers is normally stored with 32 bits. Half precision stores it with 16. Rounding introduces small numerical differences, which can change the order of closely scored passages.
We measured it end to end on a Qdrant 1.17 server running in a container on a laptop: 1,000 questions over about 40 documents, with VectoGrid's full retrieval (meaning, keywords and a reranker), once at full precision and once at half.
| 1,000 questions | Right passage first | In the top 3 | In the top 10 |
|---|---|---|---|
| Full precision (32-bit) | 950 | 997 | 999 |
| Half precision (16-bit) | 951 | 998 | 999 |
One question changed rank, a near-tie that moved from fourth to first. Half precision is therefore VectoGrid's default: the raw vectors of about 6,700 documents take 2 GB instead of 4, with no slowdown expected.
1-bit vectors: 32× smaller, rescued by a second look.
Binary quantization keeps a single bit per number, its sign, so a vector shrinks from 4 KB to 128 bytes. On its own that is a crude ranking. So the search runs in three steps:
The second look cannot recover a passage the bits ranked outside the 400, and that is the whole cost. We measured it on the three engines VectoGrid supports, with real questions on a server deliberately too small for the full vectors: the 150 FinanceBench questions over the 101,703 passages of its 84 reports, each engine alone on a server with less memory than the 400 MB of full vectors, so the second look reads them from disk.
| Exact top 10 found · median search | Qdrant (300 MB) | Postgres + pgvector (300 MB) | OpenSearch |
|---|---|---|---|
| Half precision | 99.9% · 13 ms | 99.6% · 165 ms | 100% · 61 ms (needed 1.6 GB) |
| 1-bit, the bits alone | 52.8% · 11 ms | 52.7% · 13 ms | 52.6% · 61 ms |
| 1-bit, 10× re-checked | 95.2% · 12 ms | 94.6% · 33 ms | 95.9% · 102 ms |
| 1-bit, 40× re-checked (default) | 99.3% · 14 ms | 98.9% · 87 ms | 99.5% · 222 ms (1.1 GB) |
Recall@10 against an exact brute-force search, at the same search effort for both levels. Meaning search alone, before VectoGrid merges it with keyword search and reranks; not a measure of answer accuracy.
Three things follow. The bits alone lose half of the right passages on every engine: the second look is not optional. Forty times the candidates brings 1-bit within a point of half precision, which is why it is the default. And the time it costs depends on the engine: on Qdrant 1-bit searched about as fast as half precision, on Postgres in half the time, on OpenSearch more slowly. OpenSearch also keeps its half-precision index wholly in memory: it needed 1.6 GB where Qdrant and Postgres ran in 300 MB. For a million documents, 1-bit changes the hardware: 19 GB of raw bits in memory, with the exact vectors on an SSD, instead of 307 GB of raw vectors at half precision. That makes one large server possible at 1-bit's lower recall; a cluster keeps half precision's.
Shards and copies: grow the cluster, not the code.
One server goes a long way: tens of thousands of documents at half precision fit on a single machine. Beyond that, the index spreads over several machines, and VectoGrid is built so that this never touches the application.
One, then twelve.
Every collection starts in one piece (shard): the fastest search on one machine. When the library outgrows it, one call copies every collection into a cluster cut in 12 pieces, stored vectors as they are: nothing is re-embedded.
Take whole pieces.
Add a machine and pieces move to it whole: 6 each on two machines, 4 each on three, up to twelve. Nothing is re-written.
Survive a failure.
A second copy of each piece on another machine keeps answering when one machine dies. Copies are added or removed on the live cluster.
The application keeps one address. Any machine of the cluster, or a load balancer in front of them, takes the question and gathers the answers from every piece. The application never sees the machines.
Why start with one piece: pieces cost search time until there are machines to spread them over, and we measured how much. On one server limited to 300 MB, with the 150 FinanceBench questions over 101,703 passages:
| Half precision, one server | Median search | Exact top 10 found |
|---|---|---|
| 1 piece (default) | 13 ms | 99.9% |
| 12 pieces | ~215 ms | 100% |
Twelve small pieces of about 8,500 passages each fall below the size at which Qdrant builds a search index, so every search reads them all: exact, and sixteen times slower. An assistant that searches many times per question feels that. So a collection stays in one piece until it needs machines, and is copied into twelve when it does. We also checked why copies matter: with a piece moved to a second machine and that machine stopped, every search failed until it came back. With a copy elsewhere, the search goes on.
From 10,000 to a million documents.
| Library (18-page documents) | Setup | Accuracy | Search time |
|---|---|---|---|
| Up to ~10,000 (1.5 million passages) | One small server: 3 GB of half-precision vectors, fits in 8 GB of memory. | Full: no drop observed against full precision. | Fastest: 13 ms on 100,000 passages. |
| Up to ~100,000 (15 million passages) | The same kind of server with 1-bit vectors: 1.9 GB of bits in memory, 60 GB of full vectors on an SSD. | 0.6 percentage points lower recall@10 in this test (99.3% against 99.9%). | About the same on Qdrant, half on Postgres, 3.6× on OpenSearch, measured on 100,000 passages; not yet at 15 million. |
| Up to ~1,000,000 (150 million passages) | A cluster at half precision: 307 GB of raw vectors, about six 64 GB machines by raw arithmetic, before indexes, application memory and copies. Or one large server with 1-bit vectors: 19 GB of bits in memory and 614 GB of full vectors on an SSD. | Cluster: the same as one server. 1-bit: 0.6 points lower, as above. | Cluster: longer, every search asks every machine; 1-bit: disk reads per search. Neither measured at this size. |
All of it runs on infrastructure the customer chooses, in its own cloud or data centre. VectoGrid's search has the same interface on Qdrant, on Postgres with pgvector, and on OpenSearch, and all three support half precision and 1-bit vectors. Shards and copies apply to Qdrant and OpenSearch; Postgres grows with a larger machine and its own replication. The documents never have to leave it.
What we have shown, and what we have not.
- Measured: half precision against full on 1,000 questions; half precision and 1-bit on Qdrant, Postgres and OpenSearch with 150 real questions on servers smaller than the full vectors (time and recall); the search time of 1 and 12 pieces on one server, and the copy of every collection into a cluster in pieces; pieces moved and copied between two cluster nodes, and a search failing without a copy (both on test containers, not production hardware).
- Calculated, not run: the memory figures for millions of documents. Our largest library so far is FinanceBench's 54,000 pages, indexed in about an hour on one 8-core server.
- Not yet measured: retrieval accuracy at growing library sizes, 1-bit search on dedicated servers and at millions of passages, and a cluster under production load.
How the tests were run
- Half against full precision: our 1,000-question retrieval test, written over about 40 documents in English and French (described in the FinanceBench write-up). A question counts as found when the passage holding its answer comes back in the first results. VectoGrid's full retrieval: meaning search, keyword search, and a reranker over the best 20 candidates. Qdrant 1.17 server in a container on a laptop.
- Engines: one 4-core cloud server with an SSD (DigitalOcean, shared CPU). Each engine and level in its own container: written without a memory limit, then limited (Qdrant and Postgres 300 MB; OpenSearch 1.1 GB for 1-bit and 1.6 GB for half precision, the least it ran in), restarted and the cache dropped, then 150 questions, top 10, one at a time. One piece per collection, so both levels search one graph; the same search effort (128 and 512) for both. Figures shown at effort 512.
- Pieces: the same 150 questions and 101,703 passages, half precision, one Qdrant 1.17 server limited to 300 MB, with 1 piece and with 12, search effort 512.
Accuracy at scale starts with retrieval. For what VectoGrid does with the passages it finds, see VectoGrid scores 90% on FinanceBench, with every answer and its source published.
Have a library that keeps growing?
Published 28 October 2026 · Philippe Vergnet, VectoGrid