Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarks
| Date | Stars |
|---|---|
| 2026-07-31 | 274 |
| 2026-08-05 | 274 |
| 2026-08-06 | 274 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# long-llms-learning
<p align="left">
<a href="https://arxiv.org/abs/2311.12351v2">
<img alt="survey" src="https://img.shields.io/badge/survey-arxiv:2311.12351v2-blue">
</a>
</p>
A repository sharing the panorama of the methodology literature on Transformer **architecture** upgrades in Large Language Models for handling **extensive context windows**, with real-time updating the newest published works.
## Overview
### Survey
For a clear taxonomy and more insights about the methodology, you can refer to our **survey**: [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://arxiv.org/abs/2311.12351v2) with a overview shown below
### Flash-ReRoPE
We have augmented the great work [rerope](https://github.com/bojone/rerope) by Su with flash-attn kernel to combine rerope's infinite postional extrapolation capability with flash-attn's efficience, named as **flash-rerope**.
You can find and use the implementation as a flash-attn-like interface function [here](./notebooks/flash_rerope.py), with a simple precision and flops test script [here](./notebooks/test_flash_rerope.py).
Or you can further see how to implement llama attention module with flash-rerope [here](https://github.com/Strivin0311/long-llms-eval/blob/main/evals/aug.py#L809).
## Latest News
### Latest Works
* [2024.07]
* [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://arxiv.org/pdf/2407.08608), located [here](./methodology/efficient_attn_sec/ioaware_attn.md#flash-attention) in this repo.
* [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://arxiv.org/pdf/2407.02490v1), located [here](./methodology/efficient_attn_sec/sparse_attn.md#adaptive-sparsity-patterns) in this repo.
* [2024.04] [Linear Attention Sequence Parallelism](https://arxiv.org/abs/2404.02882), located [here](./methodology/efficient_attn_sec/ioaware_attn.md#lightning-attention) in this repo.
* [2024.02] [Data Engineering for Scaling Language Models to 128K Context](https://arxiv.org/abs/2402.10171), located [here](./methodology/miscellaneous.md#long-contenxt-training) in this repo.
* [2024.01] [Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models](https://arxiv.org/abs/2401.04658), located [here](./methodology/efficient_attn_sec/ioaware_attn.md#lightning-attention) in this repo.
### Latest Baselines
* [2024.3.18] [Longer than long, the Kimi AI assistant launches 200w lossless context window](https://mp.weixin.qq.com/s/UjXKic9IAaf55ARqbnJ3Pg), located [here](./methodology/miscellaneous.md#baseline-table) in this repo.
* [2024.01.30] [chatglm-6b-128k](https://huggingface.co/THUDM/chatglm3-6b-128k) with $L_{max}$ **128k**, located [here](./evaluation/baselines.md#baseline-table) in this repo.
* [2024.01.25] [gpt-4-turbo-preview](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo) with $L_{max}$ **128k**, located [here](./evaluation/baselines.md#baseline-table) in this repo.
### Latest Benchmarks
* [2023.12.19] [InfiniteBench](https://github.com/OpenBMB/InfiniteBench), located [here](./evaluation/datasets.md#dataset-table) in this repo.
* [2023.08.29] [LongBench](https://github.com/THUDM/LongBench), located [here](./evaluation/datasets.md#dataset-table) in this repo.
## More to Learn
### Long-LLMs-Evals
* We've also released a building repo [long-llms-evals](https://github.com/Strivin0311/long-llms-evals) as a pipeline to evaluate various methods designed for general / specific LLMs to enhance their long-context capabilities on well-known long-context benchmarks.
### LLMs-Learning
* This repo is also a sub-track for another repo [llms-learning](https://github.com/Strivin0311/llms-learning), where you can learn more technologies and applicated tasks about the full-stack of Large Language Models.
## Table of Contents
Excerpt of 6,884 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c2d8ef3dd0443628, llm:Repository description: "A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarks" (topics: none).
matched fp:c2d8ef3dd0443628, llm:Repository description: "A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarks" (topics: none).
matched fp:c2d8ef3dd0443628, llm:Repository description: "A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarks" (topics: none).