Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
| Date | Stars |
|---|---|
| 2026-07-24 | 442 |
| 2026-07-25 | 442 |
| 2026-07-28 | 447 |
| 2026-07-30 | 447 |
| 2026-07-31 | 450 |
| 2026-08-06 | 450 |
Today
— stars today
This week
+3 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.67%/day
[](https://github.com/vllm-project/vllm)
[](https://opensource.org/licenses/Apache-2.0)
[](https://arxiv.org/abs/2606.03458)
[](https://huggingface.co/huawei-csl)
[](https://github.com/huawei-csl/KVarN/stargazers)
<p align="center">
<img src="imgs/logo_600.png" alt="KVarN" width="640">
</p>
> ⚡️ **Built for agentic and long-context workloads.**
> 💡 KVarN delivers **3-5x more KV-cache capacity** and **up to ~1.3x the throughput** of FP16, so you fit far longer contexts and serve more concurrent requests, with **FP16-level accuracy**.
> 🔌 **Calibration-free, plug-and-play with vLLM.** A native vLLM attention backend: add one flag, no model changes, no calibration.
> 🥊 **Up to ~2.4× TurboQuant throughput**, same capacity, **higher accuracy**.
---
## Why KVarN (Variance Normalized KV-Cache)?
> **kvarn** /kvɑːɳ/ · *noun* (Swedish)
>
> 1. A grinding apparatus used to reduce substances into smaller particles or
> powder, especially grains, seeds, spices, coffee beans, KV-caches.
KV-cache quantization usually comes with a catch. As the
[vLLM TurboQuant blog](https://vllm.ai/blog/2026-05-11-turboquant) shows, existing
methods buy extra KV-cache capacity but **give up throughput** (TurboQuant reports
**40 to 52% lower throughput** for 2.3-3.7x capacity), and aggressive low-bit
quantization also tends to **cost accuracy**. Losing both speed *and* quality is
the main reason KV-cache quantization is rarely turned on in production.
**KVarN is built to keep both.** On Qwen3-32B (AIME25, 16K-context burst, TP=2) it
matches FP16 accuracy and **beats its throughput** while delivering ~4× the KV-cache capacity:
<p align="center">
<img src="imgs/pareto_qwen3-32b.png" alt="KVarN vs FP16 vs TurboQuant: accuracy, throughput and capacity" width="660">
</p>
KVarN stays in the upper-right corner the blog's methods can't reach: **FP16-level
accuracy, FP16-or-better throughput, and several times the context.**
---
## Quickstart
KVarN ships as a vLLM fork. Install it like vLLM, then select the KVarN KV-cache dtype.
```bash
# 1. Clone
git clone https://github.com/huawei-csl/KVarN.git
cd KVarN
# 2. Install (uses the upstream precompiled wheel; KVarN kernels are Triton, JIT-compiled at runtime)
VLLM_USE_PRECOMPILED=1 pip install -e .
```
```python
from vllm import LLM, SamplingParams
llm = LLM(
model="Qwen/Qwen3-32B",
dtype="float16", # KVarN runs in float16
kv_cache_dtype="kvarn_k4v2_g128", # enable KVarN
block_size=128, # KVarN tile size
)
print(llm.generate("Explain KV-cache quantization in one sentence.",
SamplingParams(max_tokens=64))[0].outputs[0].text)
```
Serving works the same way:
```bash
vllm serve Qwen/Qwen3-32B --dtype float16 --kv-cache-dtype kvarn_k4v2_g128 --block-size 128
```
> **Note:** KVarN runs in `float16` compute. One vLLM block is one KVarN tile, so
> the tile / page size equals `--block-size`. Both **128** (default) and **64** are
> supported, selected by the matching preset (`kvarn_k4v2_g128` / `kvarn_k4v2_g64`).
> 128 is the design point; 64 gives finer quantization granularity at the cost of a
> little KV capacity (more per-tile scale overhead per token), at essentially the
> same throughput.
> **Tip (capacity):** KVarN realizes its full KV-cache capacity when there is room
> to amortize a small fixed decode workspace. On multi-GPU or generous
> `--gpu-memory-utilization` setups this is automatic. On a tight single-GPU budget,
> vLLM's CUDA-graph memory profileExcerpt of 10,363 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9264dd78c18c6b4d, topic:llm-inference, topic:vllm, desc:kv cache
matched fp:9264dd78c18c6b4d, topic:quantization, desc:quantization, readme:quantization
matched fp:9264dd78c18c6b4d, topic:agentic-ai, readme:agentic