AA-Omniscience v1.1: Which LLMs Hallucinate Least

Artificial Analysis released AA-Omniscience v1.1 on Oct 31, 2026, a refreshed version of its benchmark for how often a model makes things up instead of admitting it doesn't know. Gemini 4 Carbon (high) leads with a Net Score of 54, ahead of Claude Opus 5.5 at 48 and Claude Fable 5.1 at 46. The more useful change for builders is a new 300-question public set that tracks the private scores closely enough to run on your own models.

What changed in v1.1

AA-Omniscience asks models short factual questions with no context and no tools, then grades each answer as correct, incorrect, partial or not attempted. It is one of the nine evaluations inside the Artificial Analysis Intelligence Index v5, the same index that added HLE-Diamond this week. Artificial Analysis listed these changes in its launch thread on X and on its Hugging Face dataset card:

• Smaller, cleaner question set. The full set drops from 6,000 to 3,000 questions. Saturated questions and every previously public question were removed, ambiguous or wrong questions were fixed, and new ones were added.
• A new public subset. 300 questions (10% of the full set) are published under the Apache 2.0 license, split across domains in the same proportions as the private 2,700.
• Same six domains, new weights. Science, Engineering and Mathematics (600), Software Engineering (600), Finance (450), Health (450), Humanities and Social Sciences (450) and Law (450), covering 42 topics.
• Not comparable with v1.0. Artificial Analysis says v1.1 scores should not be compared directly with v1.0 scores.

The three numbers, and why they disagree

AA-Omniscience reports three metrics, and each one tells you something different. The definitions below come from the leaderboard page and the dataset card.

• Net Score (higher is better, from -100 to 100) is 100 × (correct − incorrect) ÷ total questions. Partial answers and abstentions score zero. A score of 0 means a model is right exactly as often as it is wrong.
• Accuracy is simply the share of all questions answered correctly. It ignores whether the misses were honest "I don't know" answers or confident mistakes.
• Hallucination Rate (lower is better) is incorrect ÷ (incorrect + partial + not attempted). In plain terms: when the model doesn't have the right answer, how often does it guess anyway?

That is why the leader changes depending on which chart you open. The model with the most right answers is not the model with the fewest made-up ones, and Net Score is the compromise between the two.

A quick worked example

The script below implements those three formulas. The two models are made up to show the trade-off; they are not real results.

from collections import Counter

def omniscience_scores(grades): """grades: list of 'CORRECT', 'INCORRECT', 'PARTIAL_ANSWER' or 'NOT_ATTEMPTED'.""" c = Counter(grades) total = len(grades) wrong, right = c["INCORRECT"], c["CORRECT"] not_correct = total - right return { "net_score": round(100 * (right - wrong) / total, 1), "accuracy_pct": round(100 * right / total, 1), "hallucination_rate_pct": round(100 * wrong / not_correct, 1) if not_correct else 0.0, }

# Two made-up models on the same 100 questions bold = ["CORRECT"] * 60 + ["INCORRECT"] * 35 + ["NOT_ATTEMPTED"] * 5 careful = ["CORRECT"] * 55 + ["INCORRECT"] * 8 + ["NOT_ATTEMPTED"] * 37 print("bold:", omniscience_scores(bold)) print("careful:", omniscience_scores(careful))

We ran it with Python 3. The "bold" model gets 60% accuracy but a Net Score of 25 and a hallucination rate of 87.5%. The "careful" model answers fewer questions correctly (55%) yet scores 47 on Net Score with a hallucination rate of 17.8%. If your product shows answers to users without a human check, the careful model is usually the one you want.

The v1.1 leaderboard

All figures below are from Artificial Analysis' launch thread and leaderboard page, published Oct 31, 2026. Effort settings matter, so they are kept exactly as Artificial Analysis lists them.

Net Score (knowledge minus hallucination)

• Gemini 4 Carbon (high): 54
• Claude Opus 5.5 (max with fallback): 48
• Claude Fable 5.1 (max with fallback): 46
• Muse Spark Open (xhigh): the top open-weights model, 28 points behind Gemini 4 Carbon

Accuracy (share of questions answered correctly)

• Claude Fable 5.1 (max with fallback): 68%
• Claude Fable 5.1 (xhigh with fallback): 67%
• Claude Opus 5.5 (xhigh with fallback): 66%

Hallucination Rate (lower is better)

• Gemini 4 Argon (high): 4%
• Gemini 4 Carbon (high): 11%
• Command A+: 14%

The pattern is clear. Anthropic's top models know the most facts, while Google's Gemini 4 models are the most willing to say "I don't know". Gemini 4 Carbon gets the best balance of the two, which is why it leads Net Score. As we noted in our HLE-Diamond coverage, Carbon had no public API when Artificial Analysis published those results, so check availability before you plan around it.

Who moved, and why scores dropped

Removing saturated questions makes the test harder, so most models score a little lower. Artificial Analysis' X thread puts the average Net Score drop at 1.9 points (median 0.6), with a 0.98 Spearman rank correlation between v1 and v1.1. The dataset card, which compares the 86 models scored on both versions, gives an average drop of 2.2 points (median 0.8). Either way, the ranking barely changed for most models.

A few models moved a lot, according to Artificial Analysis:

• MiniMax M3.1 Flash fell 22 points, from 16 to -6.
• MiMo V2.6 Pro fell 13 points, from 8 to -5.
• GPT-6 Luna (max) fell 9 points and GLM-5.3 (max) fell 8.3.
• Claude Fable 5.1 rose 2.8 points and Claude Opus 5.5 rose 1.7.

Artificial Analysis says the drop "reflects the dataset change, not model regression". A negative Net Score still means something concrete, though: on these questions, the model gives more wrong answers than right ones.

Why the public set is the real news

Most leaderboards give you a number you can't reproduce. With v1.1, Artificial Analysis published 300 questions sampled to match the private set. At launch it scored 86 models on both sets and reported a 0.991 correlation for Net Score, with a mean difference of -0.2 points. Accuracy (0.994) and Hallucination Rate (0.993) line up just as closely.

That means you can get a reasonable read on any model, including a fine-tune, a self-hosted open-weights model or a model behind your own RAG prompt, without waiting for Artificial Analysis to list it. The dataset is a single CSV with five columns: domain, subtopic, question_id, question and answer. We downloaded it and confirmed the 300 rows split 60/60/45/45/45/45 across the six domains, as the card says.

Artificial Analysis also published the exact answer prompt and grader prompt on the dataset card. The model is told to answer with just the answer, and that saying it doesn't know is better than a wrong answer. A separate grader model then returns one of four grades. To run it yourself:

1. Load AA-Omniscience_dataset_public.csv from the Hugging Face dataset.
2. Send each question with the published answer prompt, filling in domain and subtopic.
3. Grade each response with the published grader prompt, using a different model from the one under test.
4. Count the grades and compute the three metrics with the formulas above.

Both Promptfoo and Inspect can run this as a model-graded suite; our guide to building LLM evals covers the setup. One caution from the card itself: 300 questions are too few to judge a single domain or subtopic, so use the public set for overall comparisons, not to decide which model knows the most tax law.

What this means if you build with LLMs

AA-Omniscience measures closed-book recall, which is not how most production apps work. If your app retrieves documents first, accuracy matters less, because the facts come from your data. The hallucination rate still matters a lot. It shows how a model behaves when the answer isn't in front of it, and that is exactly when a RAG app breaks.

Some practical takeaways:

• Pick by the metric your product needs. For a research or tutoring assistant where recall is the job, Accuracy matters most. For customer-facing answers, legal or health content, or agents that act on facts, a low Hallucination Rate matters more.
• Give your model permission to abstain. The benchmark prompt explicitly says "I don't know" is better than a wrong answer. Put the same instruction in your own system prompt and make sure your UI handles a "not sure" reply gracefully.
• Watch the effort setting. Several top entries run at max or xhigh effort, which costs more per request. Artificial Analysis publishes cost per task for this benchmark on the leaderboard page, so compare the configuration you would actually deploy.
• Check by domain. The original AA-Omniscience paper found that different labs' models led different domains, and it advised choosing models by use case rather than by general ranking.

This is still one lab's benchmark, graded by one model (Artificial Analysis names GPT-5.6 Luna at medium effort as the grader). Treat it as a strong signal on honesty under uncertainty, then confirm it on your own data, as we argued in our comparison of SWE-bench and Terminal-Bench.

Verdict

AA-Omniscience v1.1 is one of the more useful leaderboard updates this year. The headline is familiar: Gemini 4 Carbon leads overall, Claude models know the most, and Gemini 4 Argon guesses least. The real change is the calibrated public set and the published prompts, which turn a proprietary score into something you can rerun on your own models in an afternoon. If hallucinations are a risk in your product, run those 300 questions on your shortlist before you pick a model.

Sources

• Artificial Analysis: AA-Omniscience v1.1 launch thread on X, Oct 31, 2026
• Artificial Analysis: AA-Omniscience v1.1 leaderboard and methodology, accessed Nov 1, 2026
• Hugging Face: AA-Omniscience-Public v1.1 dataset card, accessed Nov 1, 2026
• arXiv: AA-Omniscience, Evaluating Cross-Domain Knowledge Reliability in Large Language Models
• Artificial Analysis: Intelligence benchmarking methodology, accessed Nov 1, 2026