Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A low-latency & high-throughput serving engine for LLMs
| Date | Stars |
|---|---|
| 2026-07-24 | 512 |
| 2026-07-25 | 512 |
| 2026-07-28 | 512 |
| 2026-07-30 | 512 |
| 2026-07-31 | 511 |
| 2026-08-06 | 511 |
Today
— stars today
This week
-1 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Sarathi-Serve Sarathi-Serve is a high througput and low-latency LLM serving framework. Please refer to our [OSDI'24 paper](https://www.usenix.org/conference/osdi24/presentation/agrawal) for more details. ## Setup ### Setup CUDA Sarathi-Serve has been tested with CUDA 12.3 on H100 and A100 GPUs. ### Clone repository ```sh git clone [email protected]:microsoft/sarathi-serve.git ``` ### Create mamba environment Setup mamba if you don't already have it, ```sh wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh bash Mambaforge-Linux-x86_64.sh # follow the instructions from there ``` Create a Python 3.10 environment, ```sh mamba create -p ./env python=3.11 ``` ### Install Sarathi-Serve ```sh pip install -e . ``` ## Reproducing Results Refer to readmes in individual folders corresponding to each figure in `osdi-experiments`. ## Citation If you use our work, please consider citing our paper: ``` @article{agrawal2024taming, title={Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve}, author={Agrawal, Amey and Kedia, Nitin and Panwar, Ashish and Mohan, Jayashree and Kwatra, Nipun and Gulavani, Bhargav S and Tumanov, Alexey and Ramjee, Ramachandran}, journal={Proceedings of 18th USENIX Symposium on Operating Systems Design and Implementation, 2024, Santa Clara}, year={2024} } ``` ## Acknowledgment This repository originally started as a fork of the [vLLM project](https://vllm-project.github.io/). Sarathi-Serve is a research prototype and does not have complete feature parity with open-source vLLM. We have only retained the most critical features and adopted the codebase for faster research iterations.
Excerpt of 1,713 characters
Read on GitHub
19
Microsoft Open Source · Microsoft · United States
5
3
1
1
1
1
1
1
Ashish Panwar · Indian Institute of Science · India
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:729af88f6f74ade4, topic:transformer, topic:llama
matched fp:729af88f6f74ade4, topic:llm-inference, desc:serving engine, desc:high-throughput
matched fp:729af88f6f74ade4, topic:pytorch