Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICML 2025] "SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator"
| Date | Stars |
|---|---|
| 2026-07-31 | 572 |
| 2026-08-06 | 572 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<!-- <div align="center">
# **SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator**
### An Easy-to-Use Native Sparse Attention Baseline Method
---
</div> -->
<div align="center">
<h1 align="center">
<strong>🚀 SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator</strong>
</h1>
<h3 align="center">✨ An Easy-to-Use <u><strong>Native Sparse Attention</strong></u> Baseline Method</h3>
<h4 align="center">
<a href="https://sepllm.github.io" target="_blank">
<img src="https://cdn.jsdelivr.net/npm/simple-icons@v9/icons/github.svg"
alt="GitHub"
width="20"
height="20"
style="vertical-align: middle; margin-right: 8px;"/>
sepllm.github.io
</a>
</h4>
---
</div>
Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and inference speed, due to their quadratic complexity. In this work, we have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens. This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss. Guided by this insight, we introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens. Additionally, we implement efficient kernels for training acceleration. Experimental results across training-free, training-from-scratch, and post-training settings demonstrate SepLLM's effectiveness. Notably, using the Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in streaming settings, SepLLM effectively processes sequences of up to 4 million tokens or more while maintaining consistent language modeling capabilities.
# News
- :star2: [2025/07] The portable [`SepCache`](#33-sepcache) is available on [HuggingFace](https://huggingface.co/transformers-community/sep_cache)'s **transformers' official repo** now !! It is a plug-and-play `Cache` class, and we also provide sample code for monkey patching, **which supports Llama 3.1 series now**. Note that the HuggingFace's [`SepCache`](#33-sepcache) version otherwise needs `transformers>=4.53.0,<4.54.0`, *i.e.*, the new `transformers`. See [Transformers Community](https://huggingface.co/transformers-community) for detailed usage. :rocket::rocket::rocket:
- :star2: [2025/07] The portable [`SepCache`](#33-sepcache) is available on [HuggingFace](https://huggingface.co/transformers-community/sep_cache) now !! It is a plug-and-play `Cache` class, and we also provide sample code for monkey patching, **which supports Llama 3.1 series now**. Note that the HuggingFace's `SepCache` version otherwise needs `transformers>=4.53.0,<4.54.0`, *i.e.*, the new `transformers`. We are working on integrating [`SepCache`](#33-sepcache) into HuggingFace's [Transformers Community](https://huggingface.co/transformers-community). Stay tuned! :rocket::rocket::rocket:
- :star2: [2025/06] We are working on integrating [`SepCache`](#33-sepcache) into HuggingFace's [transformers](https://github.com/huggingface/transformers). Stay tuned! :rocket::rocket::rocket:
- :star2: [2025/06] [`SepCache`](#33-sepcache) is released, which is an efficient, portable, and easy-to-use *Cache* class for [transformers](https://github.com/huggingface/transformers).
- :star2: [2025/06] SepLLM's trained [checkpoint samples](https://huggingface.co/Gausson/models) have beenExcerpt of 86,757 characters
Read on GitHub
155
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ba81da6adc2bb675, topic:large-language-models