Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Efficient LLM Inference over Long Sequences
| Date | Stars |
|---|---|
| 2026-07-31 | 391 |
| 2026-08-03 | 391 |
| 2026-08-13 | 392 |
| 2026-08-19 | 392 |
| 2026-08-27 | 392 |
| 2026-09-20 | 392 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Star Attention: Efficient LLM Inference over Long Sequences
This repository contains code for the paper [Star Attention: Efficient LLM Inference over Long Sequences](https://arxiv.org/abs/2411.17116). Star Attention is a novel block-sparse attention mechanism designed to enable efficient inference on long sequences in transformer-based LLMs. The method operates in two phases:
1. **Phase 1 - Context Encoding**: The context tokens are processed using blockwise-local attention, with the context segmented into blocks where each block is prefixed with an anchor block.
2. **Phase 2 - Query Processing and Token Generation**: The query and response tokens attend to all prior cached tokens through sequence-global attention.
Star Attention **improves the inference time by up to 11x** while **preserving 97-100% of accuracy**. The method is **compatible with most Transformer-based LLMs trained with global attention, operating seamlessly out-of-the-box without additional training/finetuning.** Furthermore, Star Attention is **orthogonal to other optimization methods**, including Flash Attention and KV cache compression techniques, allowing for potential combined enhancements.
This codebase contains the implementation of Star Attention in PyTorch using the [HuggingFace Transformers](https://github.com/huggingface/transformers) library, along with the code for launching inference with Star Attention on two benchmarks: RULER and BABILong.
<div align="center">
<table>
<thead>
<tr>
<th rowspan="2" style="text-align: center">Model</th>
<th rowspan="2" style="text-align: center">Seq. Len.<br>(K)</th>
<th rowspan="2" style="text-align: center">Block Size<br>(K)</th>
<th rowspan="2" style="text-align: center">Ring-Attn<br>Acc. (%)</th>
<th colspan="2" style="text-align: center">Star-Attn</th>
</tr>
<tr>
<th style="text-align: center">Δ Acc.</th>
<th style="text-align: center">Δ Speedup</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="4">meta-llama<br>Llama3.1-8B-Instruct</td>
<td style="text-align: center">16</td>
<td style="text-align: center">4</td>
<td style="text-align: center">92.22</td>
<td style="text-align: center">-0.94%</td>
<td style="text-align: center">1.1x</td>
</tr>
<tr>
<td style="text-align: center">32</td>
<td style="text-align: center">8</td>
<td style="text-align: center">87.53</td>
<td style="text-align: center">+1.17%</td>
<td style="text-align: center">1.2x</td>
</tr>
<tr>
<td style="text-align: center">64</td>
<td style="text-align: center">16</td>
<td style="text-align: center">84.79</td>
<td style="text-align: center">-1.42%</td>
<td style="text-align: center">1.8x</td>
</tr>
<tr>
<td style="text-align: center">128</td>
<td style="text-align: center">32</td>
<td style="text-align: center">76.31</td>
<td style="text-align: center">-1.90%</td>
<td style="text-align: center">2.7x</td>
</tr>
<tr>
<td rowspan="3">meta-llama<br>Llama-3.1-70B-Instruct</td>
<td style="text-align: center">16</td>
<td style="text-align: center">4</td>
<td style="text-align: center">95.09</td>
<td style="text-align: center">-2.71%</td>
<td style="text-align: center">1.7x</td>
</tr>
<tr>
<td style="text-align: center">32</td>
<td style="text-align: center">8</td>
<td style="text-align: center">94.61</td>
<td style="text-align: center">-2.55%</td>
Excerpt of 13,922 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:dc45772fad6d3d7e, topic:llm-inference
matched fp:dc45772fad6d3d7e, topic:large-language-models