Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
| Date | Stars |
|---|---|
| 2026-07-31 | 1589 |
| 2026-08-04 | 1594 |
| 2026-08-06 | 1594 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# 📏 RULER: What’s the Real Context Size of Your Long-Context Language Models? This repository contains code for our paper [RULER: What’s the Real Context Size of Your Long-Context Language Models](https://arxiv.org/abs/2404.06654). RULER generates synthetic examples to evaluate long-context language models with configurable sequence length and task complexity. We benchmark 17 open-source models across 4 task categories (in total 13 tasks) in RULER, evaluating long-context capabilities beyond simple in-context recall. Here are our main results. |Models|Claimed Length|Effective Length|4K|8K|16K|32K|64K|128K|Avg.|wAvg. (inc)|wAvg. (dec)| |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |[Llama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) (7B)|4K||85.6| [Jamba-1.5-large*](https://huggingface.co/ai21labs/AI21-Jamba-1.5-Large) (94B/398B)|256k|>128k|<ins>96.7</ins>|<ins>96.6</ins>|<ins>96.4</ins>|<ins>96.0</ins>|<ins>95.4</ins>|<ins>95.1</ins>|96.0|95.7 **(1st)**|96.3 **(1st)**| [Gemini-1.5-pro](https://ai.google.dev/gemini-api/docs/models/gemini#:~:text=Gemini-,Gemini%201.5%20Pro%20(Preview%20only),-Text%20and%20images)|1M|>128K|<ins>96.7</ins>|<ins>95.8</ins>|<ins>96.0</ins>|<ins>95.9</ins>|<ins>95.9</ins>|<ins>94.4</ins>|95.8|95.5 **(2nd)**|96.1 **(2nd)**| [Qwen2.5-14B-Instruct-1M*](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-1m) (14B)|1M|>128K|<ins>97.5</ins>|<ins>97.1</ins>|<ins>94.6</ins>|<ins>94.9</ins>|<ins>94.9</ins>|<ins>92.2</ins>|95.7|TBD|TBD| [Qwen3-235B-A22B*](https://huggingface.co/Qwen/Qwen3-235B-A22B) (235B)|128K|>128K|<ins>97.7</ins>|<ins>97.2</ins>|<ins>96.4</ins>|<ins>95.1</ins>|<ins>93.3</ins>|<ins>90.6</ins>|95.0|TBD|TBD| [Qwen3-14B*](https://huggingface.co/Qwen/Qwen3-14B) (14B)|128K|>128K|<ins>98.0</ins>|<ins>97.8</ins>|<ins>96.4</ins>|<ins>96.1</ins>|<ins>94.0</ins>|<ins>85.1</ins>|94.6|TBD|TBD| [Jamba-1.5-mini](https://huggingface.co/ai21labs/AI21-Jamba-1.5-Mini) (12B/52B)|256K|>128K|<ins>95.6</ins>|<ins>95.6</ins>|<ins>94.8</ins>|<ins>94.6</ins>|<ins>92.8</ins>|<ins>90.0</ins>|93.9|93.1 **(3rd)**|94.8 **(3rd)** [Qwen3-32B*](https://huggingface.co/Qwen/Qwen3-32B) (32B)|128K|>128K|<ins>98.4</ins>|<ins>96.0</ins>|<ins>96.2</ins>|<ins>94.4</ins>|<ins>91.8</ins>|<ins>85.6</ins>|93.7|TBD|TBD| [EXAONE-4.0-32B*](https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B) (32B)|128K|>128K|<ins>96.3</ins>|<ins>94.9</ins>|<ins>93.9</ins>|<ins>93.6</ins>|<ins>91.7</ins>|<ins>88.2</ins>|93.1|TBD|TBD| [Qwen2.5-7B-Instruct-1M*](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-1M) (7B)|1M|>128K|<ins>96.8</ins>|<ins>95.3</ins>|<ins>93.0</ins>|<ins>91.1</ins>|<ins>90.4</ins>|84.4|91.8|TBD|TBD| [Qwen3-30B-A3B*](https://huggingface.co/Qwen/Qwen3-30B-A3B) (30B)|128K|64K|<ins>96.5</ins>|<ins>97.0</ins>|<ins>95.3</ins>|<ins>92.4</ins>|<ins>89.1</ins>|79.2|91.6|TBD|TBD| [GPT-4-1106-preview](https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4#:~:text=gpt%2D4%2D1106%2Dpreview,Up%20to%20Apr%202023)|128K|64K|<ins>96.6</ins>|<ins>96.3</ins>|<ins>95.2</ins>|<ins>93.2</ins>|<ins>87.0</ins>|81.2|91.6|89.0 **(4th)**|94.1 **(4th)**| [Llama3.1](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) (70B)|128K|64K|<ins>96.5</ins>|<ins>95.8</ins>|<ins>95.4</ins>|<ins>94.8</ins>|<ins>88.4</ins>|66.6|89.6|85.5 **(10th)**|93.7 **(5th)**| [Qwen3-8B*](https://huggingface.co/Qwen/Qwen3-8B) (8B)|128K|64K|<ins>96.3</ins>|<ins>96.0</ins>|<ins>91.8</ins>|<ins>91.2</ins>|82.1|77.4|89.1|TBD|TBD| [Mistral-Large-2411](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411) (123B)|128K|64K|<ins>96.4</ins>|<ins>96.3</ins>|<ins>95.3</ins>|<ins>94.0</ins>|<ins>85.9</ins>|48.1|86.0|79.5 **(18th)**|92.5 **(6th)**| [Command-R-plus-0824](https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024) (104B)|128K|32K|<ins>96.0</ins>|<ins>95.1</ins>|<ins>94.0</ins>|<ins>92.4</ins>|85.4|64.6|87.9|83.4 **(13th)**|92.4 **(7th)**| [Qwen2](https://huggingface.co/Qwen/Qwen2-72B-Instruct) (72B)|128K|32K|<ins>96.9</ins
Excerpt of 18,165 characters
Read on GitHub
66
Ying Sheng · RadixArk · United States
9
fayejf
7
Wang Siyuan · Shanghai Jiao Tong University · China
6
2
2
Alphabet boys
2
1
1
1
Ikko Eltociear Ashimine · Japan
1
1
Prateek Vellala
1
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a6499f50b486b7d2, llm:Repository description: 'RULER: What’s the Real Context Size of Your Long-Context Language Models?' (NVIDIA). Language: Python. Focus on evaluating context size of long-context language models.
matched fp:a6499f50b486b7d2, llm:Repository description: 'RULER: What’s the Real Context Size of Your Long-Context Language Models?' (NVIDIA). Language: Python. Focus on evaluating context size of long-context language models.
matched fp:a6499f50b486b7d2, llm:Repository description: 'RULER: What’s the Real Context Size of Your Long-Context Language Models?' (NVIDIA). Language: Python. Focus on evaluating context size of long-context language models.