Benchmarks & Deep Dives · Free

AA-Omniscience

AA-Omniscience is a knowledge and hallucination benchmark built and run by Artificial Analysis, the independent AI model benchmarking firm. It measures two things at once: whether a language model can recall facts correctly across economically relevant domains, and whether it admits when it does not know an answer instead of guessing. Models answer short, closed-ended questions with no context and no tools, so the test reflects what a model can recall on its own. The current version, AA-Omniscience v1.1, was released on October 31, 2026 as part of the Artificial Analysis Intelligence Index v5. It contains 3,000 questions across six domains and 42 topics: Science, Engineering and Mathematics (600 questions), Software Engineering (600), Finance (450), Health (450), Humanities and Social Sciences (450) and Law (450). Version 1.0, described in the AA-Omniscience paper on arXiv, had 6,000 questions. For v1.1, Artificial Analysis removed saturated questions and all previously public questions, corrected ambiguous or erroneous questions and added new ones. Artificial Analysis says v1.1 scores are not directly comparable with v1.0 scores. The benchmark reports three metrics. The Net Score (also called the AA-Omniscience Index) runs from -100 to 100 and equals 100 times correct answers minus incorrect answers, divided by the total number of questions; partial answers and abstentions score zero, so a wrong guess costs more than saying "I don't know". Accuracy is the share of all questions answered correctly. The Hallucination Rate is the share of incorrect answers among all responses that were not correct, which shows how often a model guesses when it lacks the right answer. Answers are graded by a separate model into four classes: correct, incorrect, partial answer and not attempted. Artificial Analysis publishes both the answer prompt and the full grader prompt on the public dataset card. A 300-question public subset, AA-Omniscience-Public v1.1, is available on Hugging Face under the Apache 2.0 license. It is a 10% sample that keeps the full set's domain split, while the other 2,700 questions stay private to protect the evaluation. At launch, Artificial Analysis scored 86 models on both sets and reported correlations of about 0.99 between public and private results for Net Score, Accuracy and Hallucination Rate. The public set is suited to overall comparisons but, according to Artificial Analysis, is too small for reliable results on a single domain or subtopic. Results, per-domain breakdowns, token usage and cost per task for each evaluated model are published on the Artificial Analysis leaderboard page. The benchmark is free to view, and the public dataset is free to download and run.