Understand this story
On October 27, 2026, a study compared the RAG claim-checking performance of Jev and GPT-6 Astra on 495 manually annotated claims. The results showed that Jev's accuracy differed from GPT-6 Astra's by only 0.6 percentage points, at 187 times lower cost; however, its pass rate for claims unsupported by evidence reached 23.2%, higher than Astra's 14.1%. The study also noted that although a cascade in which Jev answers first and the rest are routed to Astra lowers the bill, randomly assigning an equal proportion of claims to Astra also saves a similar amount of cost. The authors recommend that when choosing a claim checker for RAG products, emphasis should be placed on testing the false-approval rate rather than looking only at overall accuracy.
AI-generated from public reports
Latest development · Report
Jev verifier costs 187x less, but its false-approval rate is higher than GPT-6 AstraDevelopments
1 confirmed developments
Study: Jev's accuracy in verifying RAG claims approaches GPT-6 Astra
Source:Rohan Paul
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 27
Rohan Paul
Jev verifier costs 187x less, but its false-approval rate is higher than GPT-6 Astra
The paper compares Jev and GPT-6 Astra on 495 human-annotated claims: Jev's accuracy is only 0.6 percentage points lower and its cost is 187x lower, but its approval rate for claims unsupported by evidence reaches 23.2%, versus 14.1% for Astra. A cascade in which Jev answers first and the rest are routed to Astra lowers the bill, but randomly assigning an equal proportion of claims to Astra also saves a similar amount of cost. The authors suggest that when RAG products choose a claim checker, they should focus on testing the false-approval rate rather than looking only at overall accuracy.