Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.
| Date | Stars |
|---|---|
| 2026-07-31 | 361 |
| 2026-08-06 | 361 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center"> # 🚀 KVSplit **Differentiated KV Cache Quantization for Apple Silicon** [](https://github.com/dipampaul17/KVSplit/stargazers) [](LICENSE) []() <img src="./plots/kv_cache_memory_usage.png" alt="KV Cache Memory Usage" width="70%"> </div> ## 📌 Overview Run **larger context windows** and **heavier LLMs** on your Mac by applying different quantization precision to keys vs values in the attention mechanism's KV cache. KVSplit enables you to: - **Reduce memory usage by up to 72%** with minimal quality loss - **Run 2-3x longer contexts** in the same memory budget - **Maintain or improve inference speed** compared to FP16 - **Optimize for Apple Silicon** with full Metal support ## Key Findings | Configuration | VRAM @ 8K tokens | Tokens/sec | Perplexity Change | |---------------|-----------------|------------|-------------------| | FP16 (base) | 176.00 MB (100%)| 54,360 | -- | | K8V8 (8-bit) | 93.50 MB (47%) | 51,503 | +0.03% | | **K8V4** | **71.50 MB (41%)** | **57,438** | **+0.86%** | | K4V8 | 71.50 MB (41%) | 58,690 | +6.06% | | K4V4 (4-bit) | 49.50 MB (28%) | 55,193 | +6.15% | ### Memory Savings by Sequence Length | Configuration | 128 tokens | 2048 tokens | 4096 tokens | 8192 tokens | |---------------|------------|-------------|-------------|-------------| | FP16 (baseline) | 5.50 MB | 44.00 MB | 88.00 MB | 176.00 MB | | K8V8 (8-bit) | 2.92 MB | 23.38 MB | 46.75 MB | 93.50 MB | | K8V4 (mixed) | 2.23 MB | 17.88 MB | 35.75 MB | 71.50 MB | | K4V8 (mixed) | 2.23 MB | 17.88 MB | 35.75 MB | 71.50 MB | | K4V4 (4-bit) | 1.55 MB | 12.38 MB | 24.75 MB | 49.50 MB | ## Features - Independent quantization of keys and values in the KV cache - Optimized for Apple Silicon with Metal support - Comprehensive benchmarking suite with perplexity measurement - Memory usage and performance analysis tools - Publication-quality visualization tools - Easy setup and usage ## Prerequisites - macOS (tested on Apple Silicon) - Homebrew package manager - Xcode Command Line Tools ## ⚡ Flexible Installation ```bash # Clone the repository git clone https://github.com/dipampaul17/KVSplit.git cd kvsplit # Run the installer script chmod +x scripts/install_kvsplit.sh ./scripts/install_kvsplit.sh ``` The installer provides flexible options: ### 🐍 Python Setup Options - **Virtual Environment** (default): Creates a standalone Python environment in the project folder - **System Python**: Uses your existing Python installation instead of creating a virtual environment - **Skip Python Setup**: For users who prefer to manage their Python environment manually ### 🔄 llama.cpp Integration Options - **Standard Method** (default): Clones llama.cpp and applies the KV split patch - **Git Submodule Method**: Adds llama.cpp as a git submodule (ideal for advanced users or development) The installer will: - Set up the project structure with your preferred configuration - Configure llama.cpp with Metal support optimized for Apple Silicon - Enable differentiated KV cache quantization - Offer to download a small test model (optional) - Set up visualization tools based on your Python preferences ## 🏎️ Quick Comparison Want to see the benefits immediately? Run a quick comparison with your model: ```bash # Run quick comparison with different configurations python scripts/quick_compare.py --model models/your-model.gguf ``` This will show you a side-by-side comparison of FP16, K8V8, K8V4, K4V8, and K4V4 with memory usage, speed, and quality metrics. ## 📊 Impressive Results <div align="center"> <img src="./plots/memory_vs_quality.png" alt="Memory vs Quality" width="
Excerpt of 12,398 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:60136b9c29a0e0c4, topic:quantization, desc:quantization, desc:4-bit
matched fp:60136b9c29a0e0c4, topic:llama-cpp, desc:kv cache
matched fp:60136b9c29a0e0c4, topic:llm