Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A framework for serving and evaluating LLM routers - save LLM costs without compromising quality
| Date | Stars |
|---|---|
| 2026-07-31 | 5286 |
| 2026-08-02 | 5286 |
| 2026-08-04 | 5295 |
| 2026-08-06 | 5295 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# RouteLLM
RouteLLM is a framework for serving and evaluating LLM routers.
[ [Blog](http://lmsys.org/blog/2024-07-01-routellm/) ] [ [Paper](https://arxiv.org/abs/2406.18665) ]
<p align="center">
<img src="assets/router.png" width="50%" />
</p>
Our core features include:
- Drop-in replacement for OpenAI's client (or launch an OpenAI-compatible server) to route simpler queries to cheaper models.
- Trained routers are provided out of the box, which we have shown to **reduce costs by up to 85%** while maintaining **95% GPT-4 performance** on widely-used benchmarks like MT Bench.
- Benchmarks also demonstrate that these routers achieve the same performance as commercial offerings while being **>40% cheaper**.
- Easily extend the framework to include new routers and compare the performance of routers across multiple benchmarks.
## Installation
**From PyPI**
```
pip install "routellm[serve,eval]"
```
**From source**
```
git clone https://github.com/lm-sys/RouteLLM.git
cd RouteLLM
pip install -e .[serve,eval]
```
## Quickstart
Let's walkthrough replacing an existing OpenAI client to route queries between LLMs instead of using only a single model.
1. First, let's replace our OpenAI client by initializing the RouteLLM controller with the `mf` router. By default, RouteLLM will use the best-performing config:
```python
import os
from routellm.controller import Controller
os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"
# Replace with your model provider, we use Anyscale's Mixtral here.
os.environ["ANYSCALE_API_KEY"] = "esecret_XXXXXX"
client = Controller(
routers=["mf"],
strong_model="gpt-4-1106-preview",
weak_model="anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1",
)
```
Above, we pick `gpt-4-1106-preview` as the strong model and `anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1` as the weak model, setting the API keys accordingly. You can route between different model pairs or providers by updating the model names as described in [Model Support](#model-support).
Want to route to local models? Check out [Routing to Local Models](examples/routing_to_local_models.md).
2. Each routing request has a *cost threshold* that controls the tradeoff between cost and quality. We should calibrate this based on the types of queries we receive to maximize routing performance. As an example, let's calibrate our threshold for 50% GPT-4 calls using data from Chatbot Arena.
```
> python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5 --config config.example.yaml
For 50.0% strong model calls for mf, threshold = 0.11593
```
This means that we want to use `0.11593` as our threshold so that approximately 50% of all queries (those that require GPT-4 the most) will be routed to it (see [Threshold Calibration](#threshold-calibration) for details).
3. Now, let's update the `model` field when we generate completions to specify the router and threshold to use:
```python
response = client.chat.completions.create(
# This tells RouteLLM to use the MF router with a cost threshold of 0.11593
model="router-mf-0.11593",
messages=[
{"role": "user", "content": "Hello!"}
]
)
```
That's it! Now, requests with be routed between the strong and weak model depending on what is required, **saving costs while maintaining a high quality of responses**.
Depending on your use case, you might want to consider using a different model pair, modifying the configuration, or calibrating the thresholds based on the types of queries you receive to improve performance.
### Server & Demo
Instead of using the Python SDK, you can also launch an OpenAI-compatible server that will work with any existing OpenAI client, using similar steps:
```
> export OPENAI_API_KEY=sk-XXXXXX
> export ANYSCALE_API_KEY=esecret_XXXXXX
> python -m routellm.openai_server --routers mf --strong-model gpt-4-1106-preview --weak-model anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:6060 (Press CExcerpt of 14,103 characters
Read on GitHub
162
Tianhao Wu
3
2
2
Ikko Eltociear Ashimine · Japan
1
Wei-Lin Chiang
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:21aee0f9b72155c0, llm:description: 'A framework for serving and evaluating LLM routers - save LLM costs without compromising quality'
matched fp:21aee0f9b72155c0, llm:description: 'A framework for serving and evaluating LLM routers - save LLM costs without compromising quality'
matched fp:21aee0f9b72155c0, llm:description: 'A framework for serving and evaluating LLM routers - save LLM costs without compromising quality'