Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
| Date | Stars |
|---|---|
| 2026-07-31 | 400 |
| 2026-08-05 | 400 |
| 2026-08-06 | 400 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference [[paper](https://arxiv.org/abs/2406.10774)] [[poster](./assets/quest_poster.pdf)] [[slides](./assets/quest_slides.pdf)]   ## News - [2024/10] 🔥 We released Quest support for the **Llama-3.1** and **Mistral-v0.3** model family! Check out our example [here](/scripts/passkey.sh). ## TL;DR Quest is an efficient long-context LLM inference framework that leverages **query-aware sparsity** in KV cache to reduce memory movement during attention and thus boost throughput. ## Abstract As the demand for long-context large language models (LLMs) increases, models with context windows of up to 128k or 1M tokens are becoming increasingly prevalent. However, long-context LLM inference is challenging since the inference speed decreases significantly as the sequence length grows. This slowdown is primarily caused by loading a large KV cache during self-attention. Previous works have shown that a small portion of critical tokens will dominate the attention outcomes. However, we observe the criticality of a token highly depends on the query. To this end, we propose Quest, a query-aware token criticality estimation algorithm. Quest keeps track of the minimal and maximal Key values in KV cache pages and estimates the criticality of a given page using Query vectors. By only loading the Top-K critical KV cache pages for attention, Quest significantly speeds up self-attention without sacrificing accuracy. We show that Quest can achieve up to 7.03× self-attention speedup, which reduces inference latency by 2.23× while performing well on tasks with long dependencies with negligible accuracy loss. ## Installation 1. Clone this repo (also clone submodules) ``` git clone --recurse-submodules https://github.com/mit-han-lab/quest cd quest ``` 2. Install dependency libraries ``` conda create -yn quest python=3.10 conda activate quest # Quest pip install -e . # Flash-Attention pip install ninja packaging pip install flash-attn==2.6.3 --no-build-isolation # Install CMake (with version >= 3.26.4) conda install cmake # build libraft cd kernels/3rdparty/raft ./build.sh libraft ``` 3. Compile kernel benchmarks (Optional). Remember to configure env variables for CUDA (Check the [tutorial](https://faculty.cc.gatech.edu/~hyesoon/spr09/installcuda.html)). ``` cd kernels mkdir build && cd build cmake .. make -j ``` 4. Build end-to-end operators with PyBind ``` # This will automatically build and link the operators cd quest/ops bash setup.sh ``` ## Accuracy Evaluation Our evaluations are based on [LongChat-7B-v1.5-32K](https://huggingface.co/lmsys/longchat-7b-v1.5-32k?clone=true) and [Yarn-Llama2-7B-128K](https://huggingface.co/NousResearch/Yarn-Llama-2-7b-128k) models, which are capable of handling long-context text generations. We evaluate both passkey retrieval and LongBench benchmarks. We provide several scripts to reproduce our results in the paper: To get the Passkey Retrieval results, please modify and execute: ``` bash scripts/passkey.sh ``` To reproduce the LongBench results, please modify and execute: ``` bash scripts/longbench.sh ``` To evaluate the perplexity result of PG-19, please execute: ``` bash scripts/ppl_eval.sh ``` ## Efficiency Evaluation Kernels and end-to-end effiency are evaluated on NVIDIA Ada6000 and RTX4090 GPUs with CUDA version of 12.4. We provide several scripts to reproduce our results in the paper: ### Kernel-level Efficiency We also release the unit tests and benchmarks used for kernel implementations. Correctness of kernel is verified by unit tests in `kernels/src/test`, while performance is evaluated by NVBench in `kernels/src/bench`. We also test the correctness of PyBind operators in `quest/tests` with PyTorch results via PyTest. To test the correctness of kernels, please execute: ``` cd kernels/build ./test_batch_decode # or any other operator ``` Or utilize PyTest: ``` cd ques
Excerpt of 8,019 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9a952846f2df1058, desc:sparsity