Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
| Date | Stars |
|---|---|
| 2026-07-31 | 1517 |
| 2026-08-02 | 1518 |
| 2026-08-06 | 1579 |
Today
+61 stars today
This week
— stars this week
This month
— stars this month
Momentum
244.0
growth rate 0.00%/day
<div align="center" id="redknottop"> <img src="RedKnot_Logo.png" alt="RedKnot logo" width="600" margin="10px"></img> **Head-Classified KV Reuse + Elastic Sparsity for Long-Context LLM Inference** [](https://github.com/sgl-project/sglang) [](./LICENSE) []() </div> -------------------------------------------------------------------------------- ## About **RedKnot** is a long-context inference acceleration integration built on top of [SGLang](https://github.com/sgl-project/sglang). Its core idea is: **not every attention head needs the full KV, and not every token needs to go through the full FFN**. RedKnot achieves this through: - **Head Classification**: each `(layer, kv_head)` is categorized into one of four classes — `global / local / retrieval / dense` — and the KV storage and reuse strategy is decided per class; - **Offline KV Reuse + RoPE Relocation**: KV for reusable segments is stored offline; at serving time only the necessary tokens are selectively recomputed, and RoPE relocation guarantees numerical alignment; - **Elastic Sparsity / Sparse FFN**: token-selective FFN based on attention importance, skipping the feed-forward computation for low-contribution tokens; - **SegPagedAttention Runtime**: per-head page table + segmented KV store, allowing different head classes to have different visible windows; - **DeepSeek-V4** and the **full Qwen 3.5 series** will be fully updated and adapted in the next release; only the base version is open-sourced for now. While staying at **near-lossless accuracy** (in some scenarios even better than the dense baseline), it reduces long-context prefill **FLOPs by roughly 50%–70%** and delivers **1.35x–2.2x TTFT speedup** (the gains grow larger as context length increases). - **Paper**: **RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention** — Yang Liu, ZhaoKai Luo, HuaYi Jin, ZhiYong Wang, RuoZhou He, BoYu Wang, Guanjie Chen, Junhao Hu (<https://arxiv.org/abs/2606.06256>) > This repository is built on SGLang and retains all of SGLang's high-performance serving capabilities (RadixAttention, zero-overhead scheduler, PD disaggregation, continuous batching, quantization, etc.). RedKnot is integrated as an attention-layer extension under `python/sglang/srt/layers/attention/redknot/`. ## Ascend Adaptation Huawei Cloud is actively adapting RedKnot for Ascend NPUs. Ascend support is currently work in progress; compatibility notes, deployment instructions, and performance results will be updated as the adaptation matures. ## Key Ideas | Mechanism | Description | Code Location | |---|---|---| | Head classification config | `global / local / retrieval / dense` four-class strategy + JSON loading | `redknot/head_config.py`, `head_profiler.py` | | Offline KV cache + RoPE relocation | Segment-level offline KV storage and relocation (numerical alignment) | `redknot/offline_cache.py`, `rope_helper.py` | | Head-aware attention recovery | FlashAttention-2 / FA-3 bucketed attention | `redknot/ops_flash.py`, `ops_flash3.py` | | Sparse FFN (Elastic Sparsity) | Token-selective FFN, skipping computation by importance | `redknot/sparse_ffn.py` | | SegPagedAttention runtime | Per-head page table + segmented KV store | `redknot/segpaged.py`, `segpaged_v2/` | | DeepSeek-V4 MLA integration | Reuse indexer top-k for selective recomputation | `redknot/deepseek_v4_mla.py`, `dsv4_offline_reuse.py` | | PD KV transfer / head-aware scheduling | Head-class KV shard transfer and capacity model | `redknot/pd_transfer.py`, `scheduler.py` | See `python/sglang/srt/layers/attention/redknot/ROADMAP.md` for a more detailed phase plan. ## Getting Started RedKnot reuses SGLang's installation flow: ```bash # Install (devel
Excerpt of 9,655 characters
Read on GitHub
3
2
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e43f035c6d04fa8d, llm:Repository description: "Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention" (language: Python).
matched fp:e43f035c6d04fa8d, llm:Repository description: "Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention" (language: Python).
matched fp:e43f035c6d04fa8d, llm:Repository description: "Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention" (language: Python).