Skip to content

Benchmarks & Deep Dives · Free

RULER

RULER is an open-source synthetic benchmark for long-context language models, released by researchers at NVIDIA with the paper "RULER: What's the Real Context Size of Your Long-Context Language Models?" (arXiv 2404.06654, first posted April 2024 and published at COLM 2024). The code is on GitHub under NVIDIA/RULER with an Apache 2.0 license.

What it measures: RULER starts from the classic needle-in-a-haystack (NIAH) test, where a model must find one fact hidden in long filler text, and argues that this only shows a shallow form of long-context understanding. It therefore expands NIAH into variants with different needle types (words, numbers, UUIDs), several needles, multiple values per key and multiple queries, and adds new task categories: multi-hop tracing (following chains of variable assignments), aggregation (finding the most common or most frequent words across the whole input), and question answering with SQuAD or HotpotQA passages buried in distractor documents. The paper's main configuration uses 13 tasks across four categories.

Key idea, effective context length: because the examples are generated synthetically, RULER can be run at any sequence length, typically from 4K up to 128K tokens. The authors define a model's effective length as the longest length at which it still beats a fixed threshold, the score of Llama 2 7B at 4K tokens (85.6% in the README). Its headline finding was that although all 17 models evaluated in the paper claimed 32K-token context or more, only about half kept satisfactory performance at 32K, and almost all fell below the threshold before reaching their claimed length. The README keeps a larger results table with claimed and effective lengths for many open and closed models.

Who it is for: model developers, researchers and teams self-hosting open-weight models who want a cheap, configurable stress test of how far a model's usable context really goes, and how quality decays with length. Task complexity is configurable, so teams can make it harder as models improve.

How to run it: the repository provides data-generation scripts and an evaluation pipeline supporting Hugging Face, vLLM, TensorRT-LLM and API models (OpenAI, Gemini). The maintainers now point users to the rulerv1-ns and rulerv2-ns branches for the latest pipelines.

Limitations: RULER is synthetic. It measures retrieval, tracing and aggregation under controlled conditions, not end-to-end application quality, and later work such as HELMET found that synthetic recall scores do not reliably predict downstream task performance.

Pricing type: free and open source. Alternatives and complements include the original Needle in a Haystack test, NoLiMa, LongBench and LongBench v2, HELMET, BABILong and OpenAI MRCR. Official repository: https://github.com/NVIDIA/RULER (paper: https://arxiv.org/abs/2404.06654).

Official site