Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
| Date | Stars |
|---|---|
| 2026-07-31 | 1705 |
| 2026-08-02 | 1713 |
| 2026-08-06 | 1713 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# TurboQuant: KV Cache Compression for LLM Inference Implementation of TurboQuant KV cache compression (ICLR 2026, arXiv:2504.19874) with vLLM integration. Tested on dense and MoE architectures across RTX 3090 and RTX 5090 GPUs. ## Benchmark Results ### RTX 5090 (32GB) -- Qwen3.5-27B-AWQ (dense, 4-bit weights, TP=1) **Setup**: Single RTX 5090, vLLM 0.18.0, `gpu_memory_utilization=0.90`, 16 full-attention layers out of 64 total (rest are linear-attention). | Metric | Baseline (bf16 KV) | TurboQuant (3b key / 2b val) | |--------|-------------------|------------------------------| | Prefill tok/s (30k ctx) | 1,804 | 1,907 (+5.7%) | | Decode tok/s (30k ctx) | 1.264 | 1.303 (+3.1%) | | KV cache freed | -- | **30.0 GB** (across 4 GPUs) | | Max token capacity | 457,072 | **914,144** (2.0x) | | Peak activation memory | 644.6 MB | 599.2 MB (-7.0%) | ### 8x RTX 3090 (24GB each) -- Qwen3.5-35B-A3B MoE (pruned, 205 experts, TP=8) **Setup**: 8x RTX 3090, vLLM 0.18.0, `gpu_memory_utilization=0.92`, AMD EPYC 7443P 24-Core, 504GB RAM. Model has 10 full-attention layers + 30 linear-attention layers (40 total). TQ compresses only the 10 full-attention layers. #### Throughput & Latency (Baseline, bf16 KV) | Context | Prefill tok/s | Decode tok/s | TTFT (s) | Needles Found | |--------:|--------------:|-------------:|---------:|--------------:| | 1,000 | 7,127 | 129.7 | 0.14 | 4/5 | | 4,000 | 8,887 | 131.5 | 0.45 | 4/5 | | 8,000 | 9,684 | 131.1 | 0.83 | 4/5 | | 16,000 | 9,933 | 133.0 | 1.61 | 4/5 | | 32,000 | 9,761 | 116.7 | 3.28 | 4/5 | | 64,000 | 8,843 | 122.6 | 7.24 | 4/5 | | 100,000 | 8,479 | 106.8 | 11.79 | 4/5 | | 131,000 | 8,238 | 98.3 | 15.90 | 4/5 | - **Prefill** saturates around 10k tok/s, degrades gently to 8.2k at 131k context. - **Decode** drops from 133 to 98 tok/s at 131k (KV readback cost from full-attention layers). - **TTFT** scales linearly with context length (purely compute-bound). - **Needles** 4/5 found consistently at ALL context lengths -- the model reformats one answer. #### VRAM Breakdown (per GPU at 131k context) | Component | Size | |-----------|-----:| | Total VRAM | 24,576 MB | | Reserved (0.92 util) | 22,610 MB | | Model weights | ~6,750 MB | | KV cache pool | **9,035 MB** | | -- full_attention (10 layers) | 3,614 MB | | -- linear_attention (30 layers) | 5,421 MB | | CUDA overhead + graphs | ~6,825 MB | #### Baseline vs TurboQuant KV Cache | Context | Baseline KV/GPU | TQ KV/GPU | Savings/GPU | Savings % | |--------:|----------------:|----------:|------------:|----------:| | 8,000 | 55.7 MB | 38.5 MB | **17.2 MB** | 30.9% | | 32,000 | 191.5 MB | 132.3 MB | **59.3 MB** | 30.9% | | 64,000 | 374.3 MB | 258.5 MB | **115.8 MB** | 30.9% | | 100,000 | 578.1 MB | 399.2 MB | **178.8 MB** | 30.9% | | 131,000 | 755.7 MB | 521.9 MB | **233.8 MB** | 30.9% | - Savings are **30.9% of total KV** because TQ only compresses the 10 full-attention layers (40% of KV). - The 30 linear-attention layers (60% of KV) are **not compressible** by TQ. - On a **pure dense transformer**, savings would be **77%** (4.4x compression). #### Context Extension | | Tokens | Multiplier | |---|-------:|:----------:| | Baseline capacity | 1,411,680 | 1.0x | | With TQ | 2,043,808 | **1.45x** | Alternatively, freed VRAM supports **3 additional concurrent 131k-context requests**. #### Coherence & Quality | Test | Result | |------|--------| | Single needle (512-131k tokens) | **PASS** at all lengths | | 5-needle at near-max context | **5/5** retrieved | | 3-needle multi-fact coherence | **3/3** retrieved | | Golden ratio completion (all lengths) | **PASS**, perplexity 1.05-1.35 | | Math reasoning at max context | Coherent (model math error from pruning, not context) | #### TQ Quantization Quality (head_dim=256, measured on GPU) | Component | cos_sim | Notes | |-----------|--------:|-------| | TQ key compression (3-bit) | **1.000000** | Near-lossless | | TQ key compression (4-bit) | **1.000000** | Near-lossless | | Value quantization (2-bi
Excerpt of 10,116 characters
Read on GitHub
0xSero · Sybil Solutions · United States
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:73e8dd22062269db, desc:kv cache