Jev verifier costs 187x less, but its false-approval rate is higher than GPT-6 Astra
The paper compares Jev and GPT-6 Astra on 495 human-annotated claims: Jev's accuracy is only 0.6 percentage points lower and its cost is 187x lower, but its approval rate for claims unsupported by evidence reaches 23.2%, versus 14.1% for Astra. A cascade in which Jev answers first and the rest are routed to Astra lowers the bill, but randomly assigning an equal proportion of claims to Astra also saves a similar amount of cost. The authors suggest that when RAG products choose a claim checker, they should focus on testing the false-approval rate rather than looking only at overall accuracy.
Body
This paper finds that a cheap verifier can match a frontier LLM's accuracy at checking RAG claims, while letting more unsupported claims slip through.
RAG pipelines keep asking whether evidence supports a claim, a simple yes or no. Most teams get that answer from a text-generating LLM at generation prices.
They tested Jev, which returns only a label and a probability, against GPT-6 Astra on 495 human-labelled claims. They also built a cascade where Jev answers confident claims and passes the rest to Astra.
Jev landed within 0.6 points of Astra's accuracy at 187 times lower cost, but it approved 23.2% of unsupported claims against Astra's 14.1%. The cascade also cut Astra's bill, yet sending the same share of claims to Astra at random saved about as much.
Before picking a claim checker for a RAG product, measure how often it approves unsupported claims, since headline accuracy hides that.