Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
| Date | Stars |
|---|---|
| 2026-07-24 | 1115 |
| 2026-07-25 | 1115 |
| 2026-07-28 | 1120 |
| 2026-07-30 | 1120 |
| 2026-08-06 | 1120 |
| 2026-08-07 | 1129 |
| 2026-08-15 | 1131 |
| 2026-08-18 | 1135 |
| 2026-08-19 | 1138 |
| 2026-08-20 | 1138 |
| 2026-08-21 | 1139 |
| 2026-08-22 | 1140 |
| 2026-08-23 | 1140 |
| 2026-08-24 | 1141 |
| 2026-08-25 | 1143 |
| 2026-08-26 | 1143 |
| 2026-08-27 | 1144 |
| 2026-08-28 | 1145 |
| 2026-08-30 | 1145 |
| 2026-08-31 | 1145 |
| 2026-09-01 | 1192 |
| 2026-09-02 | 1257 |
| 2026-09-03 | 1303 |
| 2026-09-04 | 1323 |
| 2026-09-05 | 1332 |
| 2026-09-06 | 1341 |
| 2026-09-07 | 1353 |
| 2026-09-08 | 1362 |
| 2026-09-09 | 1363 |
| 2026-09-10 | 1365 |
| 2026-09-11 | 1366 |
| 2026-09-12 | 1367 |
| 2026-09-13 | 1369 |
| 2026-09-14 | 1371 |
| 2026-09-15 | 1386 |
| 2026-09-16 | 1389 |
| 2026-09-17 | 1395 |
| 2026-09-18 | 1406 |
| 2026-09-19 | 1419 |
| 2026-09-20 | 1441 |
Today
+22 stars today
This week
+72 stars this week
This month
+302 stars this month
Momentum
165.0
growth rate 5.26%/day
<div align="center">
<img src="https://raw.githubusercontent.com/ovg-project/kvcached/refs/heads/main/assets/logo-v2.svg" alt="kvcached logo" height="96" />
<br>
<br>
<p>
<a href="https://www.python.org/"><img alt="Python" src="https://img.shields.io/badge/Python-3.9%E2%80%933.13-blue"></a>
<img alt="Engines" src="https://img.shields.io/badge/Engines-SGLang%20%7C%20vLLM-blueviolet">
<a href="https://yifanqiao.notion.site/Solve-the-GPU-Cost-Crisis-with-kvcached-289da9d1f4d68034b17bf2774201b141"><img alt="Blog" src="https://img.shields.io/badge/Blog-Read-FF5722?logo=rss&logoColor=white&labelColor=555555"></a>
<a href="https://arxiv.org/abs/2508.08448"><img alt="arXiv: GPU OS vision" src="https://img.shields.io/badge/arXiv-GPU%20OS%20vision-b31b1b?logo=arxiv&logoColor=white&labelColor=555555"></a>
<br>
<a href="https://arxiv.org/abs/2505.04021"><img alt="arXiv: Multi LLM Serving" src="https://img.shields.io/badge/arXiv-Multi%20LLM%20Serving-b31b1b?logo=arxiv&logoColor=white&labelColor=555555"></a>
<a href="https://join.slack.com/t/ovg-project/shared_invite/zt-3fr01t8s7-ZtDhHSJQ00hcLHgwKx3Dmw"><img alt="Slack Join" src="https://img.shields.io/badge/Slack-Join-4A154B?logo=slack&logoColor=white&labelColor=555555"></a>
<a href="https://deepwiki.com/ovg-project/kvcached"><img alt="DeepWiki" src="https://img.shields.io/badge/DeepWiki-Docs-6B46C1?logo=book&logoColor=white&labelColor=555555"></a>
<a href="https://kvcached.org/"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-kvcached.org-0A66C2?logo=internetexplorer&logoColor=white&labelColor=555555"></a>
<a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-blue.svg"></a>
</p>
</div>
<h2 align="center">Make GPU Sharing Flexible and Easy </h2>
<p align="center">
<img src="https://raw.githubusercontent.com/ovg-project/kvcached/refs/heads/main/assets/ads.jpg" alt="Make GPU Sharing Flexible and Easy" width="500" />
</p>
kvcached (KV cache daemon) is a KV cache library for LLM serving/training on **shared GPUs**. By bringing OS-style **virtual memory** abstraction to LLM systems, it enables **elastic and demand-driven** KV cache allocation, improving GPU utilization under dynamic workloads.
kvcached achieves this by decoupling GPU virtual addressing from physical memory allocation for KV caches. It allows serving engines to initially reserve virtual memory only and later back it with physical GPU memory when the cache is actively used. This decoupling enables on-demand allocation and flexible sharing, bringing better GPU memory utilization under dynamic and mixed workloads. Check out more details in the [blog](https://yifanqiao.notion.site/Solve-the-GPU-Cost-Crisis-with-kvcached-289da9d1f4d68034b17bf2774201b141).
<!-- <p align="center">
<img src="https://raw.githubusercontent.com/ovg-project/kvcached/refs/heads/main/assets/vmm_v2.svg" alt="kvcached virtual memory model" width="600" />
</p> -->
<h3 align="left">Key Features</h3>
- **Elastic KV cache**: allocate and reclaim KV memory dynamically to match live load.
- **GPU virtual memory**: decouple logical KV from physical GPU memory via runtime mapping.
- **Memory control CLI**: enforce memory limits with kvcached CLI.
- **Frontend router and sleep mode**: route requests to the target models and put models to sleep when idle.
- **Support mainstream serving engines**: integrate with SGLang and vLLM.
- **Prefix caching**: support automatic prefix caching (APC) with a configurable memory bound. See [the example doc](examples/09_prefix_caching) for details.
## 📢 Updates
- **[2026-04]** kvcached is **featured by Red Hat** for running LLMs dynamically in production under limited resources! Red Hat's [Sardeenz](https://github.com/rh-aiservices-bu/sardeenz) builds on kvcached to provide dynamic multi-model serving with Kubernetes and OpenShift support. See the [blog post](https://www.redhat.com/en/blog/running-llms-dynamically-producExcerpt of 16,391 characters
Read on GitHub
77
35
34
20
9
8
8
4
4
2
Xinyuan Tong · University of Edinburgh · United Kingdom
2
2
2
1
1
Lancer · HUST · China
1
Amir Fathi · FastInfer Inc. · Canada
1
Lil2J
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e755a0770f5aa4e7, topic:llm-inference, topic:inference-engine, topic:llm-serving