Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Doing simple retrieval from LLM models at various context lengths to measure accuracy
| Date | Stars |
|---|---|
| 2026-07-31 | 2355 |
| 2026-08-04 | 2358 |
| 2026-08-06 | 2358 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Needle In A Haystack
> Pressure-test LLM long-context retrieval. Now in v2.
`niah` runs a sweep of `(context length × needle depth)` cells against any
configured model, scores each response, and writes one result row per cell to
a JSONL file. Built-in tasks include single-fact lookup, multi-fact recall,
single-UUID retrieval, and **UUID-chain hops** for testing multi-step
reasoning over long contexts.
Supported providers out of the box: **OpenAI**, **Anthropic**, **Cohere**.
Adding more is a small plugin.
---
## Quick start
```bash
pip install needlehaystack
niah demo --fake # no API key, ~1s, proves the install works
```
That runs a 2 × 3 sweep (2 context lengths × 3 depths = 6 cells) against an
in-process fake model and writes `results.jsonl`. Inspect the exact context any
cell saw:
```bash
niah reconstruct results.jsonl --row 0
```
### Demo against a real model
```bash
echo "OPENAI_API_KEY=sk-..." > .env # niah auto-loads .env
niah demo # default: gpt-4o-mini, ~$0.01
```
Or pick another provider:
```bash
niah demo --provider anthropic # needs ANTHROPIC_API_KEY
niah demo --provider cohere # needs COHERE_API_KEY
```
That's it. The demo uses sensible defaults (gpt-4o-mini, the bundled Paul
Graham essays haystack, a single-fact needle, 6 cells) so you can see real
output before learning anything about config files.
---
## Custom runs
Once the demo works, drop the `demo` command and drive your own sweep with two
small YAML files. You point `niah` at **one run config** that references **one
model config**. Full examples live in [`configs/`](./configs).
```bash
niah validate my-run.yaml # parse + resolve, no model calls
niah run my-run.yaml # actually run the sweep, append to JSONL
```
### Run config (`configs/runs/uuid_chain.example.yaml`)
```yaml
run_name: "uuid-chain-opus"
model: "anthropic-opus-4-medium" # resolved against configs/models/
task:
type: "uuid_chain"
chain_length: 5
haystack:
type: "files"
path: "PaulGrahamEssays"
sweep:
context_lengths: {min: 2000, max: 32000, num: 8, scale: "linear"}
depth_percents: {min: 0, max: 100, num: 11, scale: "sigmoid"}
seeds: [1, 2, 3]
runner:
concurrency: 2
retries: 2
resume: true
store:
type: "jsonl"
path: "results/uuid-chain-opus.jsonl"
```
### Model config (`configs/models/anthropic-opus-4-medium.yaml`)
```yaml
id: "anthropic-opus-4-medium"
runtime:
sdk: "anthropic-python"
api: "messages"
client:
api_key_env: "ANTHROPIC_API_KEY"
request:
model: "claude-opus-4"
max_tokens: 120000
thinking:
type: "adaptive"
output_config:
effort: "medium"
pricing:
input: 5.00 # USD per 1M input tokens
output: 25.00 # USD per 1M output tokens
```
Anything under `request:` is forwarded verbatim to the SDK, so adding new
provider-specific knobs (`thinking`, `reasoning_effort`, `top_p`, …) doesn't
require a code change.
---
## Built-in tasks
| `task.type` | What it does |
|----------------|-------------|
| `single` | One fact placed at one depth; exact-match scored. |
| `multi` | N facts spread evenly through the context; fractional score. |
| `uuid` | One fresh UUID at one depth; model must repeat it. |
| `uuid_chain` | Chain of `A → B → C → …` links spread through the context. The question asks "what is the value associated with A?" **without** revealing the chain structure — the model has to discover the hops on its own. |
Tasks are a small Protocol — see [`needlehaystack/tasks/`](./needlehaystack/tasks).
Adding your own is one file and a registry call; nothing in the runner needs to
change.
```python
from needlehaystack.tasks import register_task
class MyCustomTask:
name = "my_task"
inserter_name = "single_depth"
def generate_needle(self, seed): ...
def insert(self, ctx, needle, depth): ...
def question(self, needle): ...
def score(self, response, needle): ...
Excerpt of 9,515 characters
Read on GitHub
29
26
Lance Martin
19
4
Lazaro Hurtado · Microsoft · United States
4
1
1
1
Ikko Eltociear Ashimine · Japan
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:7ca218e4f05f3697, llm:Repository description: 'Doing simple retrieval from LLM models at various context lengths to measure accuracy'
matched fp:7ca218e4f05f3697, llm:Repository description: 'Doing simple retrieval from LLM models at various context lengths to measure accuracy'
matched fp:7ca218e4f05f3697, llm:Repository description: 'Doing simple retrieval from LLM models at various context lengths to measure accuracy'