Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.
| Date | Stars |
|---|---|
| 2026-07-24 | 394 |
| 2026-07-25 | 395 |
| 2026-07-28 | 395 |
| 2026-07-30 | 395 |
| 2026-07-31 | 395 |
| 2026-08-06 | 395 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<p align="center">
<img src="docs/assets/hero.png" alt="quant.cpp" width="600">
</p>
<h3 align="center">quant.cpp</h3>
<p align="center"><b>Beyond RAG: load the whole document. On your laptop.</b></p>
<p align="center">
Chunking was a workaround for small context windows. We just made it unnecessary.<br>
6.4× KV compression brings full-document understanding to consumer hardware.<br>
<code>pip install quantcpp</code> — 17.6K lines of C, zero dependencies.
</p>
<table align="center">
<tr>
<td align="center"><b>10/10 on 12K tokens</b><br>RLV crosses the cliff</td>
<td align="center"><b>7/7 vs 0/7</b><br>Beyond RAG measured</td>
<td align="center"><b>6.4x compression</b><br>+3% PPL</td>
<td align="center"><b>128K context</b><br>on 16GB Mac</td>
<td align="center"><b>17.6K LOC</b><br>zero deps</td>
</tr>
</table>
<p align="center">
<a href="https://pypi.org/project/quantcpp/"><img src="https://img.shields.io/pypi/v/quantcpp.svg?label=PyPI&color=blue" alt="PyPI"></a>
<a href="https://pypi.org/project/quantcpp/"><img src="https://img.shields.io/pypi/pyversions/quantcpp.svg" alt="Python versions"></a>
<a href="https://github.com/quantumaikr/quant.cpp/releases/latest"><img src="https://img.shields.io/github/v/release/quantumaikr/quant.cpp?label=release" alt="Release"></a>
<a href="#"><img src="https://img.shields.io/badge/license-Apache%202.0-blue" alt="License"></a>
<a href="#"><img src="https://img.shields.io/badge/tests-35%20pass-brightgreen" alt="Tests"></a>
<br>
<a href="#"><img src="https://img.shields.io/badge/models-7%20verified-blue" alt="Models"></a>
<a href="https://quantumaikr.github.io/quant.cpp/"><img src="https://img.shields.io/badge/WASM_demo-192KB-purple" alt="WASM"></a>
<a href="https://quantumaikr.github.io/quant.cpp/guide/"><img src="https://img.shields.io/badge/guide-How_it_Works-blueviolet" alt="Guide"></a>
<a href="#"><img src="https://img.shields.io/badge/platforms-macOS%20%7C%20Linux%20%7C%20WASM-orange" alt="Platforms"></a>
</p>
---
## Quick Start
**Ollama-style CLI (v0.12.0+):**
```bash
pip install quantcpp
quantcpp pull qwen3 # download Qwen3-4B Q4_K_M (~2.5 GB)
quantcpp run qwen3 # interactive chat
quantcpp serve qwen3 -p 8080 # OpenAI-compatible HTTP server (SSE streaming)
quantcpp client "Hi" # streaming client → server on :8080
quantcpp list # show cached models
```
Recommended default: **Qwen3-4B** (4B params, MMLU 73, 4.5 tok/s on M3). Best speed AND quality — the Q4 NEON fused dot path makes it 2.4x faster than Phi-3.5-mini despite a larger vocab. Other aliases: `phi3.5`, `smollm2`, `llama3.2:1b`. Auto-pulls on first `run` / `serve`.
The `serve` subcommand exposes `POST /v1/chat/completions` (OpenAI-compatible) on port 8080 — clients pass `"stream": true` for SSE streaming, or omit it for a single JSON response. Built-in `quantcpp client` supports both modes (default: streaming, `--no-stream` for single response).
**One-shot question:**
```bash
quantcpp run qwen3 "What is gravity?"
```
**Python API (3 lines):**
```python
from quantcpp import Model
m = Model.from_pretrained("Qwen3-4B")
print(m.ask("What is gravity?"))
```
Downloads on first use, cached at `~/.cache/quantcpp/`. No API key, no GPU. See [`docs/supported_models.md`](docs/supported_models.md) for the architecture support matrix and model selection guide. [Try in browser →](https://quantumaikr.github.io/quant.cpp/) · [**Interactive Guide →**](https://quantumaikr.github.io/quant.cpp/guide/)
---
## Key Result: FP32 Quality at 3x Compression
> **128 FP32 tokens + 4-bit everything else = FP32 quality, regardless of context length.**
Measured on **Llama 3.2 3B, 3970 tokens** (k128 = 3.2% FP32):
| Configuration | PPL | vs FP32 | KV Memory (32K) | Speed |
|---|---:|---:|---:|---:|
| FP32 (baseline) | 19.41 | — | 7.17 GB | baseline |
| **4-bit + progressive** | **19.39** | **-0.1%** | **2.33 GB** | **+13%** Excerpt of 66,475 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8b219445babfc469, topic:llm, topic:transformer
matched fp:8b219445babfc469, topic:llm-inference, desc:kv cache
matched fp:8b219445babfc469, topic:quantization, readme:4-bit