Skip to content

Benchmarks · Deep dive

Long-Context Benchmarks Explained: RULER, NIAH, LongBench

Needle-in-a-haystack scores say little about how well a model uses a long context. Here is what RULER, NoLiMa, MRCR, LongBench v2 and HELMET each measure, and how to test your own documents.

Metarix Editorial · Nov 15, 2026 · 11 min read

Share

Documentation-based research

Chroma technical report card titled Context Rot: How Increasing Input Tokens Impacts LLM Performance
On this page

A model's context window tells you how many tokens it will accept, not how many it can actually use. Long-context benchmarks exist to measure that gap, and they test very different skills: finding one planted sentence is easy for almost every current model, while combining facts scattered across a 100K-token input is not.

This explainer covers what the main long-context benchmarks measure as of Nov 15, 2026: Needle in a Haystack, RULER, NoLiMa, OpenAI MRCR, LongBench v2, HELMET, BABILong and InfiniteBench. It explains which published results matter, how to read a vendor's long-context claim, and how to run a quick test on your own documents.

Quick answer

Treat a perfect needle-in-a-haystack score as a minimum bar, not evidence of long-context skill. To see how far a model's usable context really goes, look at RULER's effective length or NoLiMa, which removes keyword shortcuts. To predict how it will do on real work, look at LongBench v2 and HELMET, which use real documents and tasks. Then test your own documents at the lengths you plan to use, because no public benchmark matches your workload.

Scroll the table horizontally on smaller screens.

Long-context benchmarks at a glance
BenchmarkFromWhat it testsInput lengthsBest for
Needle in a Haystack (NIAH)Greg Kamradt, open source (2023)Retrieve one or more planted facts at different depthsAny length you configureQuick smoke test of a model or deployment
RULERNVIDIA researchers (COLM 2024)13 synthetic tasks: NIAH variants, multi-hop tracing, aggregation, QAConfigurable; usually 4K to 128KFinding a model's effective context length
NoLiMaAdobe Research and partners (ICML 2025)Needle retrieval when the question and needle share almost no words1K to 32K for all models, 64K and 128K for a few; models claim 128K+Exposing keyword-matching shortcuts
OpenAI MRCROpenAI (open dataset, based on Google's MRCR)Find the i-th of 2, 4 or 8 near-identical requests in a long chatBins from 4K to about 1M tokensLong multi-turn chat and agent histories
LongBench v2Tsinghua THUDM (Dec 2024)503 expert-written multiple-choice questions over real documents8K to 2M words, mostly under 128KRealistic understanding and reasoning
HELMETPrinceton NLP (ICLR 2025)7 application categories: recall, RAG, re-ranking, citations, long QA, summarization, many-shot ICLControllable, up to 128KChoosing a model for real applications
BABILongKuratov et al. (NeurIPS 2024)20 reasoning tasks with facts scattered in long textSplits up to 10M tokensReasoning across very long inputs
InfiniteBench (∞Bench)Zhang et al. (2024)Synthetic and realistic tasks averaging over 100K tokensAverage above 100K tokensStress testing 100K+ inputs
Sources: each benchmark's paper, README or dataset card, accessed Nov 15, 2026. Years are publication or first-release dates.

Why a context window is not usable context

Three findings shaped how the field tests long context. The first is "lost in the middle": a 2023 Stanford-led study found models answered best when the relevant information sat at the start or end of the input, and noticeably worse when it was buried in the middle. Position matters, not just length.

The second is the gap between claimed and effective length. RULER defines a model's effective length as the longest length at which it still beats a fixed bar, Llama 2 7B's score at 4K tokens (85.6 in the RULER README). In the paper, all 17 models evaluated claimed 32K tokens or more, but only about half stayed above that bar at 32K.

The third is that degradation shows up even on simple tasks. Chroma's Context Rot report (July 14, 2025) held task difficulty constant while growing the input across 18 models and still saw performance become less reliable as input length grew, often in non-uniform ways.

Bar chart of claimed context window versus effective context length on RULER for nine models, from Gemini 1.5 Pro to InternLM2.5 7B
Claimed vs effective context on RULER for selected models. These are older model versions from the RULER README, accessed Nov 15, 2026, and not a current ranking. Graphic: Metarix.

The practical point: a 1M-token window means the API will accept 1M tokens. Whether the model reasons well over them is a separate question, and the answer depends on the task.

The long-context difficulty ladder

It helps to sort benchmarks by the skill they add. Literal retrieval sits at the bottom; real tasks over real documents sit at the top. Each step up is harder to game and closer to how you'll use the model.

Five-rung ladder of long-context benchmark difficulty, from literal needle retrieval to real tasks over real documents, with example benchmarks for each rung
Metarix's grouping of long-context benchmarks by the skill each one adds. Editorial grouping, not an official taxonomy. Graphic: Metarix, Nov 15, 2026.

The main benchmarks, one by one

Needle in a Haystack: the smoke test

The original Needle in a Haystack test hides one sentence (the needle) at a chosen depth inside filler text (the haystack, Paul Graham essays by default), asks the model to retrieve it, and sweeps context length against needle depth to produce a heatmap. Version 2 of the open-source tool, installed with pip install needlehaystack, adds multi-fact recall, UUID retrieval and UUID-chain tasks that need several hops.

Use it for: checking that a model or a serving setup handles long inputs at all, for example after changing a quantization, KV cache setting or context limit. Don't use it for: model selection. The question usually shares words with the needle, so a model can win by matching keywords.

RULER: effective context length

RULER, from NVIDIA researchers, generates synthetic examples at any length with configurable difficulty. Its 13 tasks span retrieval (needles that are words, numbers or UUIDs, with multiple keys, values and queries), multi-hop tracing (following chains of variable assignments), aggregation (finding the most common or frequent words across the input) and question answering with relevant passages buried among distractors. Its main output is the effective-length figure described above, and the README keeps a results table for many open and closed models.

Use it for: finding where a model's quality starts to fall off, and comparing models you can run yourself. Limits: it is synthetic, so it says little about summarization, citations or code.

NoLiMa: no literal matches

NoLiMa rewrites the needle test so the question and the needle share minimal vocabulary. The model has to make a latent association from what it knows, instead of spotting a matching word. In the paper, 13 models claiming at least 128K tokens all did well on short inputs, but at 32K, 11 of them fell below 50% of their short-context score; GPT-4o dropped from 99.3% to 69.7% (paper first posted Feb 7, 2025).

Use it for: a reality check on retrieval-heavy features like long-document Q&A, where users rarely phrase questions with the exact words in the source.

OpenAI MRCR: telling near-duplicates apart

OpenAI published MRCR (multi-round co-reference resolution), building on a task introduced in Google's Michelangelo paper. A long synthetic chat contains 2, 4 or 8 identical requests, such as several "poems about tapirs", and the model must return a specific one, like the second. The needles come from the same distribution as the filler, so nothing stands out. The dataset has 100 samples per length bin, from 4K up to about 1M tokens, and is scored with a string-similarity ratio. OpenAI's companion Graphwalks dataset asks models to run breadth-first search or find parent nodes over a graph given as a long edge list.

Use it for: anything with long multi-turn histories, such as coding agents that resend a growing conversation every step.

LongBench and LongBench v2: real documents

The original LongBench (ACL 2024) has 21 English and Chinese datasets across six task types, with an average English length of 6,711 words, short by today's standards. LongBench v2 is the one to watch: 503 hard multiple-choice questions written by nearly 100 people with professional backgrounds, over contexts of 8K to 2M words, covering single- and multi-document QA, long in-context learning, dialogue history, code repositories and structured data.

In the December 2024 paper, human experts with a 15-minute limit scored 53.7%, the best model answering directly scored 50.1%, and o1-preview with longer reasoning reached 57.7%. Newer models are on the project leaderboard. LongBench is useful because each question is labelled by domain, difficulty and length, so you can look at the slice that matches your use case.

HELMET: application-shaped evaluation

HELMET, from Princeton NLP, was built because developers kept judging long-context models on needle tests or arbitrary task subsets. It covers seven categories at controllable lengths up to 128K: synthetic recall, retrieval-augmented generation, passage re-ranking, generation with citations, long-document QA, summarization and many-shot in-context learning. Long QA and summarization use a model-based grader; see our guide to LLM-as-a-judge for the trade-offs of that approach.

Its paper (59 models evaluated) is the strongest evidence that needle scores mislead: synthetic recall did not reliably predict downstream performance, the seven categories correlated weakly with each other, and open models trailed closed ones on tasks needing full-context reasoning, with the gap widening as length grew.

BABILong and InfiniteBench: going very long

BABILong scatters the facts for 20 reasoning tasks (fact chaining, induction, deduction, counting) through long natural text, with splits up to 10M tokens. Its authors report that popular LLMs effectively use only 10 to 20% of their context. InfiniteBench was the first benchmark with an average input length above 100K tokens, mixing synthetic and realistic tasks in English and Chinese, designed so that retrieving a few passages is not enough.

Headline results, with their caveats

These numbers come from each benchmark's own paper and measure different things on different model versions. They are not comparable with each other, and most of the models involved have since been replaced.

Scroll the table horizontally on smaller screens.

What long-context benchmark papers found
Benchmark (date)FindingCaveat
RULER (Apr 2024)All 17 models claimed 32K+ tokens; about half stayed above the 85.6 threshold at 32KSynthetic tasks; models from 2023 and 2024
NoLiMa (Feb 2025)At 32K, 11 of 13 models fell below 50% of their short-context score; GPT-4o went from 99.3% to 69.7%Retrieval only; most results stop at 32K, far short of 1M
LongBench v2 (Dec 2024)Humans (15 min) 53.7%; best direct-answer model 50.1%; o1-preview 57.7%Multiple choice, so 25% is the random-guess floor
HELMET (Oct 2024)Across 59 models, synthetic recall did not reliably predict downstream resultsLengths up to 128K; some tasks graded by GPT-4o
BABILong (Jun 2024)Popular LLMs effectively used only 10 to 20% of their contextSynthetic reasoning tasks built on bAbI
Sources: the arXiv abstracts and READMEs linked in Sources, accessed Nov 15, 2026. Results are as reported by each paper's authors; Metarix did not rerun them.

How to read a vendor's long-context claim

When a model launch says it handles 1M tokens, check these before you trust it. (Our guide to reading LLM leaderboards covers the same habits for benchmarks in general.)

  • Which benchmark? A near-perfect needle heatmap is the weakest evidence. Look for MRCR, RULER, LongBench v2, HELMET or a similar multi-skill test.
  • At what length? Scores averaged across lengths hide the drop-off. Ask for results at the length you'll actually use, such as 128K or 512K.
  • How many needles, how hard? MRCR results with 8 needles are much harder than with 2. Check which variant was reported.
  • Who ran it? Vendor-run results use the vendor's own prompts and settings. Independent reruns sometimes differ, so treat them as claims until reproduced.
  • Reasoning on or off? LongBench v2 showed longer reasoning helps. Compare models in the same mode, and remember reasoning tokens add cost and latency.

Test long context on your own documents

Public benchmarks narrow the shortlist; your data decides. A useful in-house test takes an afternoon:

  • Pick 20 to 50 real questions whose answers you know, from the documents, repos or chat logs you plan to feed the model.
  • Vary length and position. Run each question at your typical input size and at your maximum, with the key passage near the start, middle and end.
  • Rephrase without keywords. Write some questions without the source's exact wording, NoLiMa-style, and some that need two facts combined.
  • Compare against retrieval. Run the same questions through a RAG pipeline that sends only the top passages. If it matches the full-context run, it is usually cheaper and faster.
  • Track cost and latency too. Long prompts are billed per token and raise time to first token; prompt caching can cut the cost of resending the same long prefix.

To get started quickly, the open-source tools have short entry points. The commands below come from each project's README (Nov 15, 2026); they are illustrative, so check the README for current options:

# Needle in a Haystack v2: no-API-key demo, then a real model
pip install needlehaystack
niah demo --fake
niah run my-run.yaml

# LongBench v2: load the 503 questions with Hugging Face datasets
from datasets import load_dataset
ds = load_dataset('THUDM/LongBench-v2', split='train')

For RULER and HELMET, both repositories include scripts that generate or download data and evaluate models served through Hugging Face, vLLM or major APIs. For grading free-form answers in your own suite, see our guide to building LLM evals.

Common mistakes

  • Choosing a model on window size alone. A bigger window is not better use of it; check effective-length style results.
  • Trusting a green needle heatmap. Most current models ace single-needle retrieval, so it rarely separates them.
  • Testing only at short lengths. Problems often show up only beyond 32K or 64K tokens.
  • Mixing benchmark versions. LongBench and LongBench v2, or MRCR with different needle counts, are different tests. Compare like with like.
  • Ignoring your serving stack. Truncation limits, quantization and KV cache settings in your gateway or inference engine can lower long-context quality. Rerun a needle check after changing them.

Verdict

Use needle-in-a-haystack as a smoke test and nothing more. For model choice, weight harder evidence: RULER's effective length and NoLiMa for how far retrieval really reaches, MRCR for long chat and agent histories, and LongBench v2 and HELMET for realistic tasks. The research consistently shows quality falling well before the advertised limit, so plan for the context you can use reliably, not the number on the pricing page.

Before committing to a long-context design, run a small test on your own documents at your real lengths and compare it with a retrieval baseline. That one afternoon will tell you more than any leaderboard.

Sources

Tools covered in this article

Frequently asked questions

What is the needle in a haystack test for LLMs?
It hides a specific fact (the needle) at a chosen depth inside long filler text (the haystack) and asks the model to retrieve it, sweeping context length and needle position. Most current models score near-perfectly, so it works as a smoke test rather than a way to compare models.
What is effective context length?
It is the longest input length at which a model still performs acceptably. RULER defines it as the longest tested length where a model beats Llama 2 7B's score at 4K tokens (85.6). It is often far shorter than the advertised context window.
Which long-context benchmark is best?
No single one. Use RULER or NoLiMa to see how far reliable retrieval reaches, OpenAI MRCR for long multi-turn histories, and LongBench v2 or HELMET for realistic tasks. Then test your own documents at the lengths you plan to use.
Is a 1M-token context window better than RAG?
Not automatically. Long-context benchmarks show quality dropping well before the maximum length, and long prompts cost more and respond slower. Compare a full-context run against a retrieval pipeline on your own questions; if retrieval matches it, it is usually cheaper.
What is the difference between LongBench and LongBench v2?
LongBench (2023) has 21 English and Chinese datasets with an average English length of about 6,700 words. LongBench v2 (December 2024) has 503 harder multiple-choice questions over contexts of 8K to 2M words, written by people with professional backgrounds.

Share

Keep Reading