Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Efficient and easy multi-instance LLM serving
| Date | Stars |
|---|---|
| 2026-07-31 | 564 |
| 2026-08-06 | 564 |
| 2026-08-14 | 563 |
| 2026-08-18 | 564 |
| 2026-08-27 | 563 |
| 2026-09-06 | 562 |
| 2026-09-15 | 561 |
| 2026-09-16 | 562 |
| 2026-09-24 | 561 |
| 2026-09-30 | 561 |
| 2026-10-22 | 561 |
| 2026-10-30 | 560 |
| 2026-11-04 | 560 |
Today
— stars today
This week
-1 stars this week
This month
-1 stars this month
Momentum
0.0
growth rate 0.00%/day
<h1 align="center">
Llumnix
</h1>
<h3 align="center">
Efficient and easy <i>multi-instance</i> LLM serving
</h3>
<p align="center">
| <a href="https://github.com/AlibabaPAI/llumnix/blob/main/docs"><b>Documentation</b></a> | <a href="https://arxiv.org/abs/2406.03243"><b>Paper</b></a> | <a href="https://llumnix.slack.com"><b>Developer Slack</b></a> |
</p>
---
## About the Llumnix V1 Upgrage (Mar. 2026)
We have refactored and upgraded Llumnix to Llumnix v1, a new architecture designed to be more modular and cloud-native, and better suited for production environments. We will keep iterating Llumnix v1 in the future. See the [new repository](https://github.com/llumnix-project/llumnix) for details!
This repository, i.e., the Ray-based architecture (Llumnix v0), is a better choice for local deployments and quick prototyping and experimentation of scheduling ideas.
---
## 🔥 Latest News
- [2025.5] We created a developer [slack](https://llumnix.slack.com) for discussion and collaboration.
- [2025.1] We updated vLLM to version v0.6.3.post1.
- [2024.11] Llumnix v0.1.0 launched!
- [2024.7] We officially released the first version of Llumnix.
- [2024.6] We released our OSDI '24 [research paper](https://arxiv.org/abs/2406.03243) on arxiv.
## 🚀 Why Llumnix
Llumnix is a cross-instance request scheduling layer built on top of LLM inference engines such as [vLLM](https://github.com/vllm-project/vllm).
Llumnix provides optimized multi-instance serving performance in terms of:
- *Low latency*
- **Reduced time-to-first-token** (TTFT) and queuing delays with less memory fragmentation
- **Reduced time-between-tokens** (TBT) and preemption stalls with better load balancing
- *High throughput*
- Integration with state-of-the-art inference engines
- Support for techniques like prefill-decode disaggregation
Llumnix achieves this with:
- Dynamic, fine-grained, KV-cache-aware scheduling
- Continuous **rescheduling** across instances
- Enabled by a KV cache migration mechanism with near-zero overhead
- Exploited for continuous load balancing, de-fragmentation, and prefill-decode disaggregation
Llumnix is easy to use with:
- Minimal code changes required for vanilla vLLM deployments
- Seamless integration with existing multi-instance deployment platforms
- Fault tolerance, elasticity, and high service availability
- Extensibility to more inference engines and scheduling policies
## Getting Started
Llumnix provides two entrypoints `api_server` and `serve` for deploying Llumnix. The `api_server` entrypoint provides a compatible deployment method with the default single-instance vLLM. By contrast, using the `serve` entrypoint, user can easily deploy Llumnix via Ray job submission API.
If you are already utilizing vLLM for multi-instance LLM serving deployments, simply replace the vLLM serving deployment command `python -m entrypoints.vllm.api_server ...` for each instance with the command provided below:
```
python -m llumnix.entrypoints.vllm.api_server \
--host $HOST \
--port $PORT \
...
```
During the serving deployment execution, Llumnix will automatically configure itself and serve as the request scheduling layer on top of the multiple vLLM engine instances.
For deploying Llumnix using the `serve` module, please refer to [QuickStart](./docs/Quickstart.md).
Visit our [documentation](./docs/) to get started:
- [Quick Start](./docs/Quickstart.md)
- [Supported Models](./docs/Supported_Models.md)
- [Fault Tolerance](./docs/Fault_Tolerance.md)
- [Simulator](./docs/Simulator.md)
- [Prefill-decode Disaggregation](./docs/Prefill-decode_Disaggregation.md)
## Performance
We evaluate the performance of the KV-cache-aware load-balancing scheduler and migration mechanism of Llumnix with 16 Qwen2.5-7B instances (each using an A10-24GB GPU) and 16 Llama2-13B instances (each using an A800-80GB GPU).
We use Poisson distributions with different request rates to generate request arrivals. For the input/output lengths of reqExcerpt of 5,502 characters
Read on GitHub
106
61
31
5
4
4
3
2
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:118c78a58a3d7117, llm:Repository description: 'Efficient and easy multi-instance LLM serving' (no topics).
matched fp:118c78a58a3d7117, llm:Repository description: 'Efficient and easy multi-instance LLM serving' (no topics).