Skip to content

Benchmarks & Deep Dives · Free

LongBench

LongBench is a family of open benchmarks for long-context understanding in large language models, built by researchers from Tsinghua University's THUDM group. The code lives at github.com/THUDM/LongBench under an MIT license, and the datasets are on Hugging Face as THUDM/LongBench and THUDM/LongBench-v2.

LongBench (v1), described in arXiv 2308.14508 and published at ACL 2024, was presented as the first bilingual, multi-task long-context benchmark. It has 21 datasets across six task categories in English and Chinese: single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks and code completion. The average length is 6,711 words for English tasks and 13,386 characters for Chinese tasks, so by today's standards it tests moderately long rather than very long inputs.

LongBench v2 (arXiv 2412.15204, released December 2024) is much harder and longer. It contains 503 multiple-choice questions with contexts from 8K to 2M words, most under 128K, across six categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding and long structured data understanding. Questions were written by nearly 100 people with professional backgrounds and filtered with automated and manual review. In the paper, human experts working under a 15-minute limit scored 53.7%, the best model answering directly scored 50.1%, and o1-preview with longer reasoning reached 57.7%. Those are December 2024 results; the project page keeps an updating leaderboard.

Why developers use it: unlike synthetic needle tests, LongBench v2 uses realistic documents and tasks that need reasoning over the whole context, and its multiple-choice format makes scoring automatic and reliable. Each item carries domain, difficulty (easy or hard) and length (short, medium, long) labels, so you can break results down by the slice you care about.

How to run it: load the data with the Hugging Face datasets library, serve your model (the README's example uses vLLM with an OpenAI-compatible endpoint), then run the provided prediction and scoring scripts. Results can be compared with and without chain-of-thought.

Limitations: multiple choice can reward elimination strategies, contexts are counted in words rather than tokens, and public test sets risk contamination over time.

Pricing type: free and open source. Alternatives include RULER, HELMET, InfiniteBench, NoLiMa and OpenAI MRCR. Official repository: https://github.com/THUDM/LongBench (project page: https://longbench2.github.io).

Official site