Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Cache-Augmented Generation: A Simple, Efficient Alternative to RAG
| Date | Stars |
|---|---|
| 2026-07-24 | 1504 |
| 2026-07-25 | 1504 |
| 2026-07-28 | 1504 |
| 2026-07-30 | 1504 |
| 2026-08-06 | 1504 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Cache-Augmented Generation (CAG)
<img src="https://github.com/hhhuang/CAG/blob/main/overview.png?raw=true" width=600 alt="Overview of CAG">
Retrieval-Augmented Generation (RAG) has emerged as a powerful approach for enhancing language models by integrating external knowledge sources. However, RAG also introduces several challenges, including:
- **Retrieval Latency** – Delays caused by real-time retrieval steps.
- **Retrieval Errors** – Inaccuracies in selecting relevant documents.
- **System Complexity** – Increased architectural and maintenance overhead.
To address these limitations, we propose **Cache-Augmented Generation (CAG)**—an alternative paradigm that bypasses real-time retrieval. CAG leverages the extended context windows of modern large language models (LLMs) by preloading all relevant resources into the model’s context and caching its runtime parameters. During inference, the preloaded KV-cache enables the model to generate responses directly, eliminating the need for retrieval.
**Advantages of CAG**
- **Reduced Latency** – Eliminates real-time retrieval, enabling faster inference.
- **Improved Reliability** – Minimizes retrieval errors while maintaining context relevance.
- **Simplified Design** – Provides a streamlined, retrieval-free alternative to RAG, achieving comparable or superior results with lower complexity.
**Limitations of CAG**
- **Limited Knowledge Size** – CAG requires the entire knowledge source to fit within the context window, making it less suitable for tasks involving extremely large datasets.
- **Context Length Constraints** – The performance of LLMs may degrade with very long contexts ([reference](https://arxiv.org/pdf/2404.02060v2)).
Our [paper](https://arxiv.org/abs/2412.15605), which will be presented at [the ACM Web Conference 2025](https://www2025.thewebconf.org/) as a short paper, investigates the relationship between model performance and context length, providing insights into scenarios where CAG excels.
The limitations of CAG are rapidly being addressed by advancements in LLMs with longer context windows and improved capabilities for extracting relevant information from extended inputs. As these models continue to evolve, CAG is expected to handle increasingly complex applications, making it a practical and scalable alternative to traditional RAG.
---
## Installation
```bash
pip install -r ./requirements.txt
```
## Preparation
> [!IMPORTANT]
> download the required `squad` and `hotpotqa` datasets by curl script
> ```bash
> sh ./downloads.sh
> ```
> [!IMPORTANT]
> create `.env` file by `.env.template` and input the keys required
> ```bash
> cp ./.env.template ./.env
> ```
## Usage
- `rag.py` is for RAG Experiment
- `kvcache.py` is for CAG Experiment
## Parameter Usage -- kvcache.py
- `--kvcache`: "file"
- `--dataset`: "hotpotqa-train" or "squad-train"
- `--similarity` "bertscore"
- `--modelname`: "meta-llama/Llama-3.1-8B-Instruct"
- `--maxKnowledge`: "", int, select how many document in dataset, explanation in Note
- `--maxParagraph`: 100
- `--maxQuestion` int, max question number, explanation in Note
- `--randomSeed`: "", int, a random seed number
- `--output`: "", str, output filepath string
- `--usePrompt`, add this parameter if not using CAG knowledge cache acceleration
### Example -- kvcache.py
```bash
python ./kvcache.py --kvcache file --dataset "squad-train" --similarity bertscore \
--maxKnowledge 5 --maxParagraph 100 --maxQuestion 1000 \
--modelname "meta-llama/Llama-3.1-8B-Instruct" --randomSeed 0 \
--output "./result_kvcache.txt"
```
## Parameter Usage -- rag.py
- `--index`: "openai" or "bm25"
- `--dataset`: "hotpotqa-train" or "squad-train"
- `--similarity` "bertscore"
- `--maxKnowledge`: "", int, select how many document in dataset, explanation in Note
- `--maxParagraph`: 100
- `--maxQuestion` int, max question number, explanation in Note
- `--topk`: int, the similarity topk of retrieval
- `--modelname`: "meta-lExcerpt of 7,814 characters
Read on GitHub
16
16
5
4
3
3
1
1
1
1
1
1
Ikko Eltociear Ashimine · Japan
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9e91d5610b838702, topic:rag, readme:retrieval-augmented generation, readme:retrieval augmented
matched fp:9e91d5610b838702, topic:llm