Artificial Analysis releases the AA-Omniscience v1.1 benchmark
Artificial Analysis has upgraded its knowledge and hallucination benchmark AA-Omniscience to v1.1, reducing the question bank from 6,000 to 3,000 questions and adding a new 300-question public set, whose Pearson correlation coefficient with the private set's Net Score is 0.99. Gemini 4 Carbon (high) ranks first with a Net Score of 54, while Claude Opus 5.5 and Claude Fable 5.1 score 48 and 46 respectively; among open-weight models, Muse Spark Open (xhigh) leads, trailing the top by 28 points. On the new question set, models' Net Score drops by an average of 1.9 points, with MiniMax M3.1 Flash falling 22 points (from 16 to -6); the median hallucination rate drops by 1.4 percentage points, with Gemini 4 Argon (high) lowest at 4%.
Body
We’ve released AA-Omniscience v1.1, updating our knowledge and reasoning benchmark with a refreshed public set, removal of saturated questions, and ambiguity fixes
AA-Omniscience is our proprietary benchmark measuring model knowledge and hallucinations, including whether models admit when they don’t know an answer. Our primary metric, Net Score, rewards correct answers, penalizes incorrect answers, and treats abstentions neutrally.
AA-Omniscience was updated to v1.1 as part of the Artificial Analysis Intelligence Index v5 release.
Key changes in AA-Omniscience v1.1:
➤ Updated 3,000-question dataset: Down from 6,000 questions, with previously public and saturated questions removed, errors and ambiguities corrected, and new questions added
➤ Newly calibrated 300-question public set: AA-Omniscience v1.1 has a new public set on @huggingface calibrated to match the difficulty of the private set. At release, public and private scores have a 0.99 Pearson correlation and 0.99 Spearman rank correlation for AA-Omniscience Net Score, with a median difference of 0.0 points. Accuracy and hallucination rate also have 0.99 correlations, with median differences of -0.07 and -0.29 percentage points, respectively
➤ Slightly updated domain distribution: Questions span the same 6 domains and 42 topics as version 1, with a revised distribution: Science, Engineering and Mathematics (600), Software Engineering (600), Finance (450), Health (450), Humanities and Social Sciences (450), and Law (450)
Key results for AA-Omniscience v1.1:
➤ Rank order largely preserved: AA-Omniscience v1.1 has a 0.98 Spearman rank correlation with v1. The recently released Gemini 4 Carbon (high) leads with a Net Score of 54, followed by Claude Opus 5.5 (48, max with fallback) and Claude Fable 5.1 (46, max with fallback). Muse Spark Open (xhigh) is the leading open weights model, 28 points behind Gemini 4 Carbon
➤ Scores average ~2 points lower on the refreshed set: Models score 1.9 points lower on Net Score on average compared to v1, with a median decrease of 0.6 points. Some models see much larger declines, particularly MiniMax M3.1 Flash, down 22 points (16 to -6), and MiMo V2.6 Pro, down 13 points (8 to -5)
➤ Slightly lower accuracy: AA-Omniscience Accuracy measures the percentage of questions the model answers correctly. Models answer a median of 1.5 percentage points fewer questions correctly on average compared to v1. Claude Fable 5.1 (max with fallback) has the highest accuracy at 68%, followed by Claude Opus 5.5 (max with fallback) at 66%
➤ Hallucination rates decrease: AA-Hallucination measures how often models choose to answer when they do not know the answer. The median Hallucination Rate falls 1.4 percentage points, with Gemini 4 Argon (high) recording the lowest hallucination rate at 4%