Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
| Date | Stars |
|---|---|
| 2026-08-05 | 402 |
| 2026-08-18 | 403 |
| 2026-08-21 | 404 |
| 2026-08-22 | 405 |
| 2026-08-26 | 406 |
| 2026-08-27 | 407 |
| 2026-08-30 | 408 |
| 2026-08-31 | 408 |
| 2026-09-02 | 409 |
| 2026-09-03 | 409 |
| 2026-09-08 | 410 |
| 2026-09-13 | 411 |
| 2026-09-16 | 413 |
| 2026-09-23 | 414 |
| 2026-10-11 | 416 |
| 2026-10-12 | 416 |
| 2026-10-19 | 417 |
| 2026-10-20 | 418 |
| 2026-10-28 | 419 |
| 2026-11-04 | 420 |
Today
+1 stars today
This week
+1 stars this week
This month
+6 stars this month
Momentum
0.0
growth rate 0.24%/day
[English](./README.md) | [中文简体](./README.zh_CN.md)
<h1 align="center">
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
</h1>
<p align="center">
<a href="https://opensource.org/license/apache-2-0/">
<img alt="License: Apache" src="https://img.shields.io/badge/License-Apache2.0-green.svg">
</a>
<a href="https://github.com/IAAR-Shanghai/CRUD_RAG/issues">
<img alt="GitHub Issues" src="https://img.shields.io/github/issues/IAAR-Shanghai/CRUD_RAG?color=red">
</a>
<a href="https://arxiv.org/abs/2401.17043">
<img alt="arXiv Paper" src="https://img.shields.io/badge/Paper-arXiv-blue.svg">
</a></p>
# Introduction
This repository contains the official code of CRUD-RAG, a novel benchmark for evaluting the RAG systems. It includes the datasets we created for evaluating RAG systems, and a tutorial on how to run the experiments on our benchmark.
# Important Notes
- The prompts in this repository are **designed for use with ChatGPT**. For other models, we recommend selecting appropriate prompts. The 7B models are particularly sensitive to prompts, they can not understand complex prompts. So please exercise caution.
- The use of RAGQuestEval metric relies on GPT, we use GPT as question answer and generator.
- The first time you run the code, you need to build a vector index for the text(It takes about **3 hours**). This is a one-time process, so you don't need to repeat it later. Please make sure to omit the construct-index parameter when you use the code again.
- The models evaluated are primarily from 2023. **Since then, output styles have shifted significantly** (e.g., use of subheadings, icons, and flattering or ingratiating language), which are absent in our references. Consequently, bleu scores based on string matching may vary. For comparable results, we recommend carefully prompting for concise outputs.
# Project Structure
```bash
├── data # This folder comprises the datasets used for evaluation.
│ │
│ ├── crud
│ │ └── merged.json # The complete datasets.
│ │
│ ├── crud_split
│ │ └── split_merged.json # The dataset we used for experiments in the paper.
│ │
│ └── 80000_docs
│ │ └── documents_dup_part... # More than 80,000 news documents, which are used to build the retrieval database of the RAG system.
│ │
├── src
│ ├── configs # This folder comprises scripts used to initialize the loading parameters of the LLMs in RAG systems.
│ │
│ ├── datasets # This folder contains scripts used to load the dataset.
│ │
│ ├── embeddings # The embedding model used to build vector databases.
│ │
│ ├── llms # This folder contains scripts used to load the large language models.
│ │ ├── api_model.py # Call GPT-series models.
│ │ ├── local_model.py # Call a locally deployed model.
│ │ └── remote_model.py # Call the model deployed remotely and encapsulated into an API.
│ │
│ ├── metric # The evaluation metric we used in the experiments.
│ │ ├── common.py # bleu, rouge, bertScore.
│ │ └── quest_eval.py # RAGQuestEval. Note that using such metric requires calling a large language model such as GPT to answer questions, or modifying the code and deploying the question answering model yourself.
│ │
│ ├── prompts # The prompts we used in the experiments.
│ │
│ ├── quest_eval # Question answering dataset for RAGQuestEval metric.
│ │
│ ├── retrievers # The retriever used in RAG system.
│ │
│ └── tasks # The evaluation tasks.
│ ├── base.py
│ ├── continue_writing.py
│ ├── hallucinated_modified.py
│ ├── quest_answer.py
│ └── summary.py
```
# Quick Start
- Install dependency packages
```bash
pip install -r requirements.txt
```
- Start the milvus-lite service(vector database)
```bash
milvus-server
```
- Download the bge-base-zh-v1.5 model to the sentence-transformers/bge-base-zh-v1.5/ directory
- Modify config.py according to your need.
- Run quick_start.py
```bashExcerpt of 5,049 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:4811c7b6c2076036, topic:retrieval-augmented-generation, desc:retrieval-augmented generation, desc:retrieval augmented
matched fp:4811c7b6c2076036, topic:large-language-models
matched fp:4811c7b6c2076036, topic:benchmark