Jev promises very cheap, very fast AI decisions. Could it also make our document-intelligence pipeline cheaper?
At VectoGrid, finding the right passage is only half the job. We still need to extract the right figure, date or name, and check that it answers the question. Jev could potentially help with either step. So we tested it twice: first on short yes/no checks, then on document extraction.
Our answer: use Jev to check, not to extract.
Jev is TypeSafe AI's decision model. It does not write free-form answers: it answers structured questions, such as picking an option or judging whether a statement is true, each with a probability.
TypeSafe reports 193.6× faster and 444.6× cheaper results in its own workflow evaluations. We did not rerun those evaluations: we tested different tasks, so our numbers neither confirm nor refute TypeSafe's.
Our question was narrower: where, if anywhere, would Jev improve our own pipeline? We explored three roles:
| Role | What Jev would do |
|---|---|
| Make a decision | Check whether a proposed value answers a question |
| Extract a value | Select an answer from candidates found by code |
| Check a language model's answer | Flag an extracted value for further review |
We gave each setup 100 examples containing a question, a sentence from a report and a proposed answer. The task: does this value answer the question, according to the sentence? Half the proposed answers were labelled correct, half incorrect.
| Setup | Correct of 100 | Cost per 1,000 decisions | Median response time |
|---|---|---|---|
| Jev, TypeSafe API | 93 | $0.015 | 0.23 s |
| Claude Opus 5.5, medium reasoning | 89 | $2.15 | 1.93 s |
| Claude Opus 5.5, low reasoning | 87 | $1.31 | 1.49 s |
| Claude Haiku 4.5 | 82 | $0.11 | 0.58 s |
| gpt-oss-120b | 86 | $0.067 | 0.47 s |
| gpt-oss-20b | 87 | $0.036 | 0.19 s |
On these tasks, Jev's cost advantage over Opus was large and its speed advantage much smaller than in TypeSafe's evaluations: even at high reasoning (a four-decision probe, see technical details), Opus answered these one-sentence questions in about 3 seconds. Against a small model on a fast host, even the speed advantage disappeared.
Jev had the highest observed accuracy, but 100 questions are not enough to establish that it is better than every alternative: only its lead over Claude Haiku 4.5 is clear.
There was also a useful difference in behaviour. Jev accepted three incorrect values and rejected four correct ones. The Opus setups accepted no incorrect values, but rejected 11–13 labelled-correct answers. For a checker, that trade-off matters: catching every error and avoiding unnecessary reviews are different objectives.
Jev needs code to propose possible answers before it can select one. We tested the open-source jevextract library, which supplies that candidate code, and a modified approach that gives Jev the whole passage and the library's candidates in one request, rather than going sentence by sentence. Each language model received the same passage and returned an answer with a supporting quote.
| Setup | Correct: real reports (140) | Correct: generated documents (160) | Cost per 1,000 extractions | Median time |
|---|---|---|---|---|
| jevextract + Jev | 36% | 4% | $4.48 | 4.5 s |
| Jev, one request | 47% | 17% | $0.27 | 0.33 s |
| gpt-oss-20b | 94% | 96% | $0.11 | 0.22 s |
| gpt-oss-120b | 95% | 100% | $0.24 | 0.53 s |
| Qwen 3.6 27B | 99% | 100% | $1.47 | 0.80 s |
| Claude Haiku 4.5 | 98% | 98% | $1.65 | 0.95 s |
For our extraction workload, gpt-oss-20b was more accurate, cheaper and faster than either Jev setup.
Passage size matters. Our passages had a median of 640 words, what a reader gets in our pipeline. The library makes one Jev call per sentence, 63 calls per question at the median, which is why it costs 17× more than one request. On a single short paragraph it would make a handful of calls and cost far less (details below).
The main bottleneck was getting suitable candidates to Jev. On 203 of the 300 questions, including 72 of the 140 real-report questions, our automatic audit found no candidate holding the expected figures; figures in tables were a frequent problem. The audit is not exact: four answers it found no candidate for were judged correct. Where it did find one, Jev in one request picked it 89 times out of 97.
Giving Jev the whole passage in one request helped substantially. But it did not close the gap with the language models. This is evidence about these implementations on our workload, not a limit on every possible Jev-based extractor.
The tempting workflow is to accept Jev's confident extractions and send the rest to a language model.
With the one-request approach, a cut-off of 0.9 accepted 78 of 300 answers, two of them wrong. Both errors dropped the scale: €2,446 instead of €2,446 million. Even a high-probability selection can therefore return an incomplete value when the candidate boundaries omit essential information. The other 222 would still need a language model. On this sample, the score did help separate correct from incorrect answers.
But it does not make this workflow cheaper than starting with gpt-oss-20b. The Jev pass alone cost $0.27 per 1,000 extractions, against $0.11 for the small language model, and the fallback calls come on top. The economics might differ against a more expensive fallback model or with a better candidate generator; they do not favour this setup against our cheapest language model. And the 0.9 cut-off was picked after seeing these results, so it needs confirming on fresh examples.
We then reversed the workflow: a language model extracts, and Jev is asked whether the proposed value answers the question, given the passage, as xmemory did.
On the four language models' extractions, Jev flagged all 12 incorrect values while also flagging 18 of 1,170 correct ones, about 1.5%, at about $0.06 per 1,000 checks. (The other 18 of the 1,200 answers were "not in the passage", with no value to check.)
These errors were mostly conspicuous: a label returned instead of a number, or a value from the neighbouring section. Our existing deterministic checker caught 11 of the 12, but also flagged 60% of correct answers: it was designed for structured fields with labels and units, and is too restrictive for free-form questions. Simply confirming that a quote appears in the passage was not enough: a genuine quote can still support the wrong answer.
We also ran Jev over the 150 answers of our FinanceBench run, 22 of them judged incorrect or incomplete. Here the errors were subtler: a figure from the wrong year, a calculation slip, a misread question.
| Review rule | Errors flagged (of 22) | Correct answers flagged (of 128) | Answers to review |
|---|---|---|---|
| Our existing flag: two methods disagree | 17 | 37 | 54 |
| Jev alone, below 0.9 | 17 | 53 | 70 |
| Existing flag, or Jev below 0.7 | 19 | 46 | 65 |
On its own, Jev caught the same number of errors as our existing flag while sending more correct answers for review. Combined with it, Jev caught two more errors for 11 more reviews (two incorrect answers and nine correct ones). That result is exploratory: we tried several cut-offs and chose 0.7 after seeing them, so it needs confirming on fresh answers. At best it is a useful extra layer, not a replacement, and its value depends on what a missed error costs against an extra review.
Cheapest, highest score and near the fastest. The margin depends on the baseline: very large against Opus, small against gpt-oss-20b, and far below 190× in speed on these tasks.
Through the library, worse than every language model on accuracy, cost and speed. The bottleneck is the code that proposes candidates, not Jev's choice among them.
It caught every error in the language models' extractions for about 6 cents per 1,000 checks. On harder answers, it added a modest benefit on top of our existing review flag.
Our choice: keep the small open model as extractor and evaluate Jev as an additional checker. The goal is fewer costly mistakes or needless reviews across the whole pipeline.
The extraction test has 300 single-passage questions from our 1,000-question retrieval benchmark: all 140 written from real English and French reports, and 160 from generated documents built to confuse (sections alike word for word, tables that differ only by a column). Questions needing several passages are excluded. Every setup receives the same answer-containing passage; for 280 questions it was search's own first result. This isolates extraction after successful retrieval; it does not measure end-to-end answer quality.
The decision test uses 100 examples from this extraction work: the incorrect proposals were errors made by the Jev-based extraction, the correct ones came from two language models. It is a diagnostic sample, not an estimate of production error rates. Some labelled-correct examples are table rows without their column headers; rejecting those can be a defensible response to incomplete evidence, so read decision accuracy with the false acceptances and rejections.
An answer was accepted automatically when every figure of two digits or more and every name in the expected answer appeared in it, each as a whole token. That rule does not check scale words, units, currency or sign, so we re-read every accepted answer that differed from the expected one on any of those, or held extra figures: 59 of 1,285. Two were wrong (the scale dropped, both from Jev in one request) and are counted wrong; the other 57 only added a unit or a conversion. The 497 answers the rule did not accept were judged by Claude against the passage, without knowing the setup. Claude Haiku 4.5 is itself one of the setups. Real-report extraction accuracy with its 95% interval: jevextract + Jev 36% (29–45), Jev one request 47% (39–55), gpt-oss-20b 94% (88–97), gpt-oss-120b 95% (90–98), Qwen 3.6 27B 99% (96–100), Claude Haiku 4.5 98% (94–99). Paired comparisons use McNemar's test; small differences among the language models need caution, and not finding a difference does not establish equivalence.
For Jev in one call, the cut-off applies to the probability Jev gave the chosen candidate; for the library, to each candidate's probability of being the answer. Neither uses Jev's separate confidence field. The cut-offs quoted (0.9 for extraction and checking, 0.7 combined with our flag on FinanceBench) were chosen after looking at results for several values, so they describe this sample: a production cut-off should be chosen on separate examples.
To explore whether more reasoning widened the gap, we also ran Opus 5.5 at high reasoning on four decisions. This small probe is separate from the main 100-example comparison: about 3 s and $3 per 1,000, thinking for 64 to 306 tokens. Against that, Jev was about 200× cheaper and 13× faster.
Passages had a median of 640 words (about 4,300 characters; 95% under 1,160 words). The library splits a passage into sentences, table rows counting as sentences, and makes one Jev call per sentence:
| Per question (median) | Jev calls | Candidates judged | Jev input tokens |
|---|---|---|---|
| jevextract + Jev | 63 | 652 | about 99,000 |
| Jev, one call | 1 | 114 (mean 281) | about 3,300 |
Both setups use the library's candidate code, sentence by sentence, unchanged. The library judges each sentence's candidates in that sentence's call, so a text proposed by several sentences (a year, a heading word) is judged each time; the one-request setup keeps each distinct text once. Nothing else is filtered or cut: more than 254 candidates go to Jev in several blocks of the same call. Each library call also repeats its framing and a sentence of context on each side, so it submitted about 90 times as many input tokens as the passage contains (some 1,100); a language model is sent the passage once.
Jev takes at most 255 options in one choice, and 122 of the 300 passages produced more. Those candidates went in one request as several choice questions of up to 254 candidates each, plus "none", and the answer was the candidate with the highest probability in any block. Probabilities from different blocks are conditional on different candidate sets, so comparing them is a heuristic we did not validate in general. Among questions where the audit found a matching candidate, accuracy was 57/63 with multiple blocks and 32/34 with one block. These groups contain different questions, so this comparison does not isolate the effect of splitting candidates. Keeping each distinct text once can also merge occurrences that mean different things, such as one figure under two different years. The value returned is the same either way, but which occurrence it came from is lost.
Jev used TypeSafe's own API, Claude used Anthropic's, and the open models ran on groq: these compare hosted setups, not model architectures on equal hardware. Times for decisions and language-model extraction are client-observed per call, from request to reply, network included, after one untimed warm-up call; times for Jev extraction are per question, every Jev call and the library's local candidate code included. Calls were made from France. Timings include network latency, which can noticeably affect comparisons between these short requests. Through OpenRouter, Jev took 0.28 s and gave the same answer on 99 of the 100 decisions. The complete study cost $3.26.
TypeSafe's documentation does not say which languages Jev supports. About a third of our real-report questions are in French: Jev got 20 of 22 French decisions right against 73 of 78 English ones, and 45% against 48% in one-request extraction. We did not observe a large French–English performance gap in this sample. The French subset is small, however, so these results do not establish equivalent performance across languages.
jevextract has no question-answering mode: each question became one field described as "the value that answers this question: …" (a probe rated that far above the bare question), with the library's candidate code unchanged, so the results concern that use of the library. Each setup ran once.
Every question, answer, verdict and check, for every setup. Passages are included for the generated documents; the real ones come from published reports, articles and market studies we do not redistribute, so the file names the document and gives each language model's supporting quote.
See also VectoGrid on FinanceBench.