Benchmarks & Deep Dives · Free
HELMET
HELMET (How to Evaluate Long-context Models Effectively and Thoroughly) is an open-source benchmark for long-context language models from Princeton NLP, described in arXiv 2410.02694 and published at ICLR 2025. The code is on GitHub at princeton-nlp/HELMET under an MIT license, with a project page at princeton-nlp.github.io/HELMET.
Why it exists: the authors argue that developers too often judge long-context models with synthetic tests like needle-in-a-haystack or an arbitrary subset of tasks, and that existing benchmarks give noisy signals because of limited application coverage, insufficient lengths, unreliable metrics such as n-gram overlap, and incompatibility with base (non-instruction-tuned) models.
What it covers: seven application-centric categories, with controllable input lengths up to 128K tokens: synthetic recall (JSON key-value retrieval and RULER-style needle tasks), retrieval-augmented generation (Natural Questions, TriviaQA, PopQA, HotpotQA), passage re-ranking (MS MARCO), generation with citations (ALCE), long-document question answering (NarrativeQA and InfiniteBench tasks), summarization (InfiniteBench and Multi-LexSum) and many-shot in-context learning (intent and question-type classification). Long QA and summarization use model-based evaluation with GPT-4o as the grader, and few-shot prompting lets base models be evaluated fairly.
Key findings from the paper, based on 59 models: synthetic tasks like NIAH do not reliably predict downstream performance; the seven categories show distinct trends with low correlation between them; and while most models achieve perfect NIAH scores, open models lag behind closed ones when tasks need full-context reasoning or complex instruction following, with the gap widening as length grows. The authors recommend the RAG tasks for fast iteration because they are easy to run and predict other categories better.
How to run it: download the preprocessed data (about 34 GB per the README), then run eval.py with a config per task category against a local model, a vLLM server or API models from OpenAI, Anthropic, Google and Together. The repository also hosts LongProc, a companion benchmark for long procedural generation.
Who it is for: model builders and teams choosing a long-context model who want a broad, application-shaped signal rather than a single retrieval score.
Pricing type: free and open source (API-graded tasks incur grader costs). Alternatives include RULER, LongBench v2, NoLiMa, BABILong and OpenAI MRCR. Official repository: https://github.com/princeton-nlp/HELMET (paper: https://arxiv.org/abs/2410.02694).
