Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
| Date | Stars |
|---|---|
| 2026-07-31 | 852 |
| 2026-08-04 | 852 |
| 2026-08-06 | 852 |
| 2026-08-13 | 853 |
| 2026-08-20 | 854 |
| 2026-09-01 | 854 |
| 2026-09-03 | 855 |
| 2026-09-06 | 856 |
| 2026-09-10 | 857 |
| 2026-09-11 | 858 |
| 2026-09-13 | 858 |
| 2026-09-15 | 859 |
| 2026-09-16 | 861 |
| 2026-09-20 | 861 |
Today
— stars today
This week
+3 stars this week
This month
+7 stars this month
Momentum
0.0
growth rate 0.35%/day
# OmniServe: Unified and Efficient Inference Engine for Large-Scale LLM Serving
**[Paper (QServe)](https://arxiv.org/abs/2405.04532) | [Paper (LServe)](https://arxiv.org/abs/2502.14866) | [Website (QServe)](https://hanlab.mit.edu/projects/qserve) | [Website (LServe)](https://hanlab.mit.edu/projects/lserve)**
OmniServe aims to revolutionize large-scale LLM serving by unifying and optimizing key advancements in both low-bit quantization and long-context processing. OmniServe integrates the innovations from [QServe](https://arxiv.org/abs/2405.04532), which boosts efficiency with W4A8KV4 quantization and reduces dequantization overheads, and [LServe](https://arxiv.org/abs/2502.14866), which accelerates long-context LLM inference through unified sparse attention and hierarchical KV cache management. OmniServe delivers a comprehensive solution for scalable and cost-effective LLM deployment. This unified system addresses the dual challenges of computational complexity and memory overhead, achieving significant speedups in both prefill and decoding stages, while also maximizing GPU throughput and minimizing infrastructure costs.
## News
- [2025/02] 🔥 **OmniServe is now publicly available!** OmniServe has integrated optimizations from both QServe and LServe into one single LLM inference framework. Experience efficient and accurate inference for both [long-context](#lserve-efficient-long-sequence-llm-serving-with-unified-sparse-attention) and [quantized](#qserve-w4a8kv4-quantization-and-system-co-design-for-efficient-llm-serving) LLMs with OmniServe now!
- [2025/02] 🏆 Both **QServe and LServe have been accepted by MLSys 2025**!
- [2024/12] 🔥 QServe has been integrated into NVIDIA [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/README.md#w4aint8-quantization-qserve)!
- [2024/05] 🔥 QServe is publicly released! Check our paper [here](https://arxiv.org/abs/2405.04532).
## Key Features
**OmniServe** is a unified, flexible, and efficient LLM serving system designed to support modern large language models and multi-modal language models. With configurable quantization precisions and hybrid sparse attention patterns, OmniServe integrates the strengths of [QServe](#qserve-w4a8kv4-quantization-and-system-co-design-for-efficient-llm-serving) and [LServe](#lserve-efficient-long-sequence-llm-serving-with-unified-sparse-attention), enabling efficient processing of both large-batch and long-context inputs, significantly reducing LLM serving costs while maintaining high response quality.
## Contents
- [Installation](#installation)
- [OmniServe Model Zoo](#omniserve-model-zoo)
- [QServe: ***W4A8KV4*** Quantization and System Co-design for Efficient LLM Serving](#qserve-w4a8kv4-quantization-and-system-co-design-for-efficient-llm-serving)
- [Introduction](#introduction)
- [Usage and Examples](#usage-and-examples)
- [Results](#results)
- [Accuracy Evaluation](#accuracy-evaluation)
- [Efficiency Benchmarks](#efficiency-benchmarks)
- [LServe: Efficient Long-Sequence LLM Serving with ***Unified Sparse Attention***](#lserve-efficient-long-sequence-llm-serving-with-unified-sparse-attention)
- [Introduction](#introduction-1)
- [Usage and Examples](#usage-and-examples-1)
- [Results](#results-1)
- [Accuracy Evaluation](#accuracy-evaluation-1)
- [Efficiency Benchmarks](#efficiency-benchmarks-1)
- [Reference](#reference)
- [Team](#team)
- [Related Projects](#related-projects)
- [Acknowledgement](#acknowledgement)
## Installation
1. Clone this repository and navigate to the corresponding folder:
```bash
git clone https://github.com/mit-han-lab/OmniServe
cd OmniServe
```
2. Install OmniServe
2.1 LLM setup tutorial
If you hope to serve text-only LLMs, please follow the tutorial below:
```bash
conda create -n OmniServe python=3.10 -y
conda activate OmniServe
pip install --upgrade pip # enable PEP 660 support
conda install -c nvidia cuda-toolkit -y # This is optional if you prefer to use Excerpt of 33,468 characters
Read on GitHub
Shang Yang · MIT · United States
17
Haotian (Ken) Tang · Meta · United States
5
Yujun Lin · Research Scientist at @NVIDIA · United States
2
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:1ceea6d7403a12e2, desc:quantization