Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A fast inference library for running LLMs locally on modern consumer-class GPUs
| Date | Stars |
|---|---|
| 2026-07-31 | 4597 |
| 2026-08-02 | 4597 |
| 2026-08-04 | 4601 |
| 2026-08-06 | 4601 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Note
**This project is archived for now**. Development continues on [ExLlamaV3](https://github.com/turboderp-org/exllamav3).
# ExLlamaV2
ExLlamaV2 is an inference library for running local LLMs on modern consumer GPUs.
The official and recommended backend server for ExLlamaV2 is [TabbyAPI](https://github.com/theroyallab/tabbyAPI/),
which provides an OpenAI-compatible API for local or remote inference, with extended features like HF model
downloading, embedding model support and support for HF Jinja2 chat templates.
See the [wiki](https://github.com/theroyallab/tabbyAPI/wiki/1.-Getting-Started) for help getting started.
## New in v0.1.0+:
- ExLlamaV2 now supports paged attention via [Flash Attention](https://github.com/Dao-AILab/flash-attention) 2.5.7+
- New generator with dynamic batching, smart prompt caching, K/V cache deduplication and simplified API
## Dynamic generator
The dynamic generator supports all inference, sampling and speculative decoding features of the previous two
generators, consolidated into one API (with the exception of FP8 cache, though the Q4 cache mode is supported and
performs better anyway, see [here](doc/qcache_eval.md).)
The generator is explained in detail [here](doc/dynamic.md).
- Single generation:
```python
output = generator.generate(prompt = "Hello, my name is", max_new_tokens = 200)
```
- Batched generation:
```python
outputs = generator.generate(
prompt = [
"Hello, my name is",
"Once upon a time,",
"Large language models are",
],
max_new_tokens = 200
)
```
- Streamed generation with `asyncio`:
```python
job = ExLlamaV2DynamicJobAsync(
generator,
input_ids = tokenizer.encode("You can lead a horse to water"),
banned_strings = ["make it drink"],
gen_settings = ExLlamaV2Sampler.Settings.greedy(),
max_new_tokens = 200
)
async for result in job:
text = result.get("text", "")
print(text, end = "")
```
See the full, updated examples [here](https://github.com/turboderp/exllamav2/tree/master/examples).
## Performance
Some quick tests to compare performance with ExLlama V1. There may be more performance optimizations in the future,
and speeds will vary across GPUs, with slow CPUs still being a potential bottleneck:
| Model | Mode | Size | grpsz | act | 3090Ti | 4090 |
|------------|--------------|-------|-------|-----|---------|-------------|
| Llama | GPTQ | 7B | 128 | no | 181 t/s | **205** t/s |
| Llama | GPTQ | 13B | 128 | no | 110 t/s | **114** t/s |
| Llama | GPTQ | 33B | 128 | yes | 44 t/s | **48** t/s |
| OpenLlama | GPTQ | 3B | 128 | yes | 259 t/s | **296** t/s |
| CodeLlama | EXL2 4.0 bpw | 34B | - | - | 44 t/s | **50** t/s |
| Llama2 | EXL2 3.0 bpw | 7B | - | - | 217 t/s | **257** t/s |
| Llama2 | EXL2 4.0 bpw | 7B | - | - | 185 t/s | **211** t/s |
| Llama2 | EXL2 5.0 bpw | 7B | - | - | 164 t/s | **179** t/s |
| Llama2 | EXL2 2.5 bpw | 70B | - | - | 33 t/s | **38** t/s |
| TinyLlama | EXL2 3.0 bpw | 1.1B | - | - | 656 t/s | **770** t/s |
| TinyLlama | EXL2 4.0 bpw | 1.1B | - | - | 602 t/s | **700** t/s |
## How to
To install from the repo you'll need the CUDA Toolkit and either gcc on Linux or (Build Tools for) Visual Studio
on Windows). Also make sure you have an appropriate version of [PyTorch](https://pytorch.org/get-started/locally/), then run:
```sh
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .
python test_inference.py -m <path_to_model> -p "Once upon a time,"
# Append the '--gpu_split auto' flag for multi-GPU inference
```
A simple console chatbot is included. Run it with:
```sh
python examples/chat.py -m <path_to_model> -mode llamExcerpt of 9,282 characters
Read on GitHub
1.3k
Brian · United States
31
Sinan
22
13
12
11
Lyrcaxis
6
6
Saifeddine ALOUI · CEA · France
5
5
5
4
4
3
3
3
3
3
3
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:79a5c93a08fa20d4, llm:description: 'A fast inference library for running LLMs locally on modern consumer-class GPUs' (repo description)
matched fp:79a5c93a08fa20d4, llm:description: 'A fast inference library for running LLMs locally on modern consumer-class GPUs' (repo description)
matched fp:79a5c93a08fa20d4, llm:description: 'A fast inference library for running LLMs locally on modern consumer-class GPUs' (repo description)