Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for LLM KV cache compression. 5x compression at 3-bit with 99.5% attention fidelity.
| Date | Stars |
|---|---|
| 2026-07-31 | 1037 |
| 2026-08-06 | 1038 |
Today
+1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# TurboQuant
A from-scratch PyTorch implementation of [TurboQuant](https://arxiv.org/abs/2504.19874) (ICLR 2026), Google's vector quantization algorithm for compressing LLM key-value caches. Tested on Windows with NVIDIA GPUs.
We implemented the paper's algorithm, found that its key innovation (QJL) actually hurts in practice, and built an improved version (V3) informed by findings from 8+ independent community implementations.
> **Correction (2026-03-30):** An earlier version of this README claimed "18/18 perfect generation at 5x compression." This was based on a [bugged test](https://github.com/tonbistudio/turboquant-pytorch/issues/14) where `residual_window=0` caused no compression to happen. The corrected results are below. Credit to [@barbel-bb](https://github.com/barbel-bb) for finding the bug.
## Results
### V3: Generation Test (the real test — does the model produce correct text?)
We hid a fact ("The secret project code name is AURORA-7749") in a long document and asked the model to find it. Results with **actual compression verified** (compressed token counts logged):
| Config | 2K ctx | 4K ctx | Compression (2K) |
|--------|--------|--------|-----------------|
| FP16 (baseline) | EXACT | EXACT | 1.0x |
| **K6/V4 rw=128** | **EXACT** | **EXACT** | **~2x** |
| **K8/V4 rw=128** | **EXACT** | **EXACT** | **~1.6x** |
| K4/V4 rw=128 | PARTIAL ("AURORA7749") | MISS | ~3x |
| K4/V4 rw=0 | MISS | MISS | ~3.4x |
| K4/V2 rw=0 | MISS | MISS | ~5x |
"EXACT" = output contains "AURORA-7749". "PARTIAL" = contains both "AURORA" and "7749" but not the exact string. "rw" = residual window (recent tokens kept in fp16).
**What works:** K6/V4 with a 128-token fp16 window gives ~2x real compression with perfect output at both context lengths. At 4-bit keys, the model finds the needle at short context but garbles it slightly (drops the hyphen). At 3-bit keys, generation is broken.
**What doesn't work:** 3-4 bit compression without a residual window produces garbage, same as V2. High attention score similarity (99.5%+) does not guarantee working generation.
### V3: Attention Score Accuracy (8K context)
These results are valid — they test compression directly on captured KV tensors, not through V3Cache:
| Config | Compression | Cosine Similarity | Top-1 Match | Top-5 Match |
|--------|-----------|------------------|-------------|-------------|
| V3 K4/V2 | 5.1x | **0.9996** | **94%** | **97%** |
| V3 K4/V2 + protected layers | 3.6x | **0.9997** | **99%** | **100%** |
| V2 3-bit (MSE+QJL) | 5.0x | 0.9945 | 86% | 94% |
| V2 4-bit (MSE+QJL) | 3.8x | 0.9983 | 86% | 96% |
V3 gets better attention score accuracy than V2 by removing QJL. However, high attention scores alone do not guarantee working text generation (see above).
## What Is K4/V2?
The KV cache stores two types of vectors: **Keys** (K) and **Values** (V).
- **Keys** decide which words the model pays attention to — this needs precision
- **Values** are the content that gets averaged together — errors cancel out naturally
**K4/V2** means keys get 4 bits, values get 2 bits. The average is 3 bits — same as uniform 3-bit — but allocated where it matters. This gives dramatically better results than uniform allocation at the same bit budget.
## How It Works
### The Core: Random Rotation + Lloyd-Max Quantization
Each vector is multiplied by a random orthogonal matrix, which makes every coordinate follow a predictable bell-curve distribution. We then apply an **optimal scalar quantizer** (Lloyd-Max) to each coordinate independently, rounding to the nearest precomputed centroid.
To quantize: normalize, rotate, round each coordinate, store indices + norm.
To dequantize: look up centroids, reverse the rotation, restore the norm.
### What About QJL? (The Paper's Stage 2)
The paper adds a second stage: QJL residual correction, which stores 1-bit sign information to make inner product estimates mathematically unbiased. We implemented this as V2.
**It doesn't work for KV cache.Excerpt of 10,563 characters
Read on GitHub
9
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:2a63455c5ddbea28, desc:kv cache