Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[EMNLP 2023] Enabling Large Language Models to Generate Text with Citations. Paper: https://arxiv.org/abs/2305.14627
| Date | Stars |
|---|---|
| 2026-07-31 | 523 |
| 2026-08-05 | 523 |
| 2026-08-06 | 523 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Enabling Large Language Models to Generate Text with Citations
<p align="center"><img src="https://github.com/princeton-nlp/ALCE/blob/main/assets/moose.png?raw=true" alt="ALCE" width="15%"><br>*: ALCE is pronounced as /elk/ as ALCE is the Latin word for elk (Europe) or moose (North America).
</p>
This repository contains the code and data for paper [Enabling Large Language Models to Generate Text with Citations](https://arxiv.org/abs/2305.14627).
In this paper, we propose ALCE, a benchmark for **A**utomatic **L**LMs' **C**itation Evaluation.
ALCE contains three datasets: ASQA, QAMPARI, and ELI5.
We provide automatic evaluation code of LLM generations around three dimensions: fluency, correctness, and citation quality.
This repository also includes code to reproduce the baselines in our paper.
<img src="https://github.com/princeton-nlp/ALCE/blob/main/assets/ALCE.png?raw=true" alt="ALCE" width="100%">
## Quick Links
- [Requirements](#requirements)
- [Data](#data)
- [Code Structure](#code-structure)
- [Reproducing Baselines](#reproducing-baselines)
- [Evaluation](#evaluation)
- [Human Evaluation](#human-evaluation)
- [Bug or Questions](#bug-or-questions)
- [Citation](#citation)
## Requirements
Please install the latest versions of PyTorch (`torch`), HuggingFace Transformers (`transformers`), HuggingFace Accelerate (`accelerate`), and the OpenAI API package (`openai`). This codebase is tested on
`torch==2.1.0.dev20230514+cu118`, `transformers==4.28.1`, `accelerate==0.17.1`, and `openai==0.27.4` with Python 3.9.7.
## Data
You can download datasets (along with retrieval results) by running the following command:
```bash
bash download_data.sh
```
All the data will be stored in `data/`. Our data included top-100 DPR/GTR retrieved results for ASQA and QAMPARI, and top-100 BM25 retrieved results for ELI5. We also provide reranked oracle retrieval results, where top-5 passages can achieve the same recall as the original top-100 recall.
### Retrieval
You can reproduce the passage retrieval step with the following command:
```bash
python retrieval.py --data {path/to/data} --retriever {bm25/gtr} --output_file {path/to/output}
```
There are additional packages required for the retrieval steps.
Specifically, you need to install `pyserini==0.21.0`(their github [repo](https://github.com/castorini/pyserini/tree/master) is helpful) and `sentence-transformers==2.2.2`.
For the BM25 retrieval over Common Crawl using Sphere, you must first download the index from the Sphere [repo](https://github.com/facebookresearch/Sphere), and set the environmental variable `BM25_SPHERE_PATH` to the path of the downloaded index.
Specifically, you can use the following command:
```bash
wget -P faiss_index https://dl.fbaipublicfiles.com/sphere/sphere_sparse_index.tar.gz
tar -xzvf faiss_index/sphere_sparse_index.tar.gz -C faiss_index
export BM25_SPHERE_PATH=$PWD/faiss_index
```
It's important to note that given the large size of the corpus, this step is extremely expensive and time-consuming. We found that larger CPU memory tends to help with the speed.
For GTR, we first build an index using the DPR wikipedia snapshot, which you can obtain using the download script from the DPR [repo](https://github.com/facebookresearch/DPR), and then setting the environmental variable `DPR_WIKI_TSV` to the path of the tsv file.
Specifically, you can use the following command:
```bash
wget https://dl.fbaipublicfiles.com/dpr/wikipedia_split/psgs_w100.tsv.gz
gzip -xzvf psgs_w100.tsv.gz
export DPR_WIKI_TSV=$PWD/psgs_w100.tsv
```
Then, you want to set `GTR_EMB` to the path of the GTR embeddings of the Wikipedia corpus, and running the retrieval script for the first time will automatically build and save the index.
Building the dense index can be expensive for GPU memory (we use 80GB GPUs for this) and time-consuming; the entire index will take about 31GB.
If you find this step to be too expensive, you can also download it using:
```bash
wget Excerpt of 8,124 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:fbfa76aef2187104, llm:Description: 'Enabling Large Language Models to Generate Text with Citations' (EMNLP 2023 paper). Repository name princeton-nlp/ALCE and paper link indicate NLP research for citation generation by LLMs.
