Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Extend existing LLMs way beyond the original training length with constant memory usage, without retraining
| Date | Stars |
|---|---|
| 2026-07-31 | 735 |
| 2026-08-06 | 735 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Attention Sinks in Transformers for endless fluent generation **TL;DR**: `attention_sinks` adapts pre-trained LLMs to use a modified form of sliding window attention that remains able to produce fluent text indefinitely. ## Benchmark Findings See [Benchmark Setups](#benchmark-setups) for information on how these benchmarks were carried out. ### Perplexity The following figures plot model perplexities under the various different approaches. A higher perplexity is indicative that the model is losing the ability to produce proper language. | Llama-2-7b-hf | Falcon-7B | |:-------------:|:-------------:| |  |  | | **MPT-7B** | **Pythia-6.9B** | |  |  | | **Mistral-7B-v0.1** | **GPT-J-6B** | |  |  | | **Qwen-7B** | **StableLM-3B-4E1T** | |  |  | | **BTLM-3B-8k-base** | **Yi-6B** | | | | The results are clear as day: 1. `transformers`: The VRAM usage is linear as it doesn't do any windowing. The performance heavily falls after the pretraining length. 2. `windowed`: The VRAM is constant usage due to the windowing at 1024 tokens. However, the performance falls as soon as the first tokens leave the window. 3. `attention_sinks`: Constant VRAM usage due to windowing with 4 attention sink tokens + the 1020 most recent tokens. This approach never fails despite the constant VRAM usage. ### Fluency during endless generation See here text generated by the same Llama 2 7B model using the same settings, but loaded using: * [`transformers`](https://github.com/tomaarsen/attention_sinks/blob/main/demo/endless_logs/transformers/meta-llama/Llama-2-7b-hf.txt): Loses fluency after ~1900 tokens and starts endlessly generating broken unicode characters like ` 🤖🧠👨���������������������` ❌. * [`window` attention](https://github.com/tomaarsen/attention_sinks/blob/main/demo/endless_logs/windowed/meta-llama/Llama-2-7b-hf.txt): Loses fluency after ~1000 tokens, generates hundreds of newlines interspersed with text like `OOOMMO̶OANOOAMOO̶OMMO` ❌. * [`attention_sinks`](https://github.com/tomaarsen/attention_sinks/blob/main/demo/endless_logs/attention_sinks/meta-llama/Llama-2-7b-hf.txt): Fluent for the full 10k tokens of the test ✅. ### Fluency during subsequent prompting for chat-style LLMs In this benchmark, I sent subsequent prompts from [MT-Bench](https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts) through the model and automatically detect when fluency gets lost. > [!WARNING] > The automatic detection of fluency losses is very naive: it tries to count the number of real words in the response, but that can result in false positives if e.g. the prompt is to generate some German text. See [demo/streaming_logs](https://github.com/tomaarsen/attention_sinks/tree/main/demo/stream
Excerpt of 19,739 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9a5e698f53b8310d, topic:llm
matched fp:9a5e698f53b8310d, topic:nlp
