Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code and data for "Lost in the Middle: How Language Models Use Long Contexts"
| Date | Stars |
|---|---|
| 2026-07-31 | 387 |
| 2026-08-02 | 389 |
| 2026-08-06 | 389 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Lost in the Middle: How Language Models Use Long Contexts
This repository contains accompanying material for [Lost in the Middle: How
Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172).
## Table of Contents
- [Installation](#installation)
- [Multi-Document Question Answering Experiments](#multi-document-question-answering-experiments)
- [Multi-Document Question Answering Data](#multi-document-question-answering-data)
* [Generating new multi-document QA data.](#generating-new-multi-document-qa-data)
- [Key-Value Retrieval Experiments](#key-value-retrieval-experiments)
- [Key-Value Retrieval Data](#key-value-retrieval-data)
* [Generating new key-value retrieval data](#generating-new-key-value-retrieval-data)
- [Testing Prompting Templates](#testing-prompting-templates)
- [References](#references)
## Installation
1. Set up a conda environment
``` sh
conda create -n lost-in-the-middle python=3.9 --yes
conda activate lost-in-the-middle
```
2. Install package and requirements
``` sh
pip install -e .
```
3. (optional) set up pre-commit hooks for development.
``` sh
pre-commit install
```
## Multi-Document Question Answering Experiments
See [EXPERIMENTS.md](./EXPERIMENTS.md#multi-document-question-answering) for
instructions to run and evaluate models on the multi-document QA task.
## Multi-Document Question Answering Data
[`qa_data/`](./qa_data/) contains multi-document question answering data for the
oracle setting (1 input document, which is exactly the passage that answers the
question) and 10-, 20-, and 30-document settings (where 1 input passage answers
the question, and the other passages do not contain an NQ-annotated answer).
Each line of this gzipped file is in the following format:
``` sh
{
"question": "who got the first nobel prize in physics",
"answers": [
"Wilhelm Conrad Röntgen"
],
"ctxs": [
...
{
"id": <string id, e.g., "71445">,
"title": <string title of the wikipedia article that this passage comes from>,
"text": <string content of the passage>,
"score": <string relevance score, e.g. "1.0510446">,
"hasanswer": <boolean, whether any of the values in the `answers` key appears in the text>,
"original_retrieval_index": <int indicating the original retrieval index. for example, a value of 0 indicates that this was the top retrieved document>,
"isgold": <boolean, true or false indicating if this chunk is the gold answer from NaturalQuestions>
},
...
],
"nq_annotated_gold": {
"title": <string title of the wikipedia article containing the answer, as annotated in NaturalQuestions>,
"long_answer": "<string content of the paragraph element containing the answer, as annotated in NaturalQuestions>",
"chunked_long_answer": "<string content of the paragraph element containing the answer, randomly chunked to approximately 100 words>",
"short_answers": [
<string short answers, as annootated in NaturalQuestions>
]
}
}
```
### Generating new multi-document QA data.
1. First, download Contriever retrieval results for each of the queries:
``` sh
wget https://nlp.stanford.edu/data/nfliu/lost-in-the-middle/nq-open-contriever-msmarco-retrieved-documents.jsonl.gz
```
2. Then, to generate examples with 20 total documents with the relevant documents at positions 0, 4, 9, 14, and 19, run:
``` sh
for gold_index in 0 4 9 14 19; do
python -u ./scripts/make_qa_data_from_retrieval_results.py \
--input-path nq-open-contriever-msmarco-retrieved-documents.jsonl.gz \
--num-total-documents 20 \
--gold-index ${gold_index} \
--output-path qa_data/nq-open-20_total_documents_gold_at_${gold_index}.jsonl.gz
done
```
## Key-Value Retrieval Experiments
See [EXPERIMENTS.md](./EXPERIMENTS.md#key-value-retrieval) for
instructions to run and evaluate models on the key-value retrieval task.
## Key-Value Retrieval Data
[`kv_retrieval_data/`](./kv_retrieval_data/) contains multi-documeExcerpt of 6,405 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9acad13e57c8c6dc, llm:Repository description: "Code and data for 'Lost in the Middle: How Language Models Use Long Contexts'" — implements experiments analyzing LMs' use of long context.
matched fp:9acad13e57c8c6dc, llm:Repository description: "Code and data for 'Lost in the Middle: How Language Models Use Long Contexts'" — implements experiments analyzing LMs' use of long context.
matched fp:9acad13e57c8c6dc, llm:Repository description: "Code and data for 'Lost in the Middle: How Language Models Use Long Contexts'" — implements experiments analyzing LMs' use of long context.