Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Reproducible, flexible LLM evaluations
| Date | Stars |
|---|---|
| 2026-07-31 | 390 |
| 2026-08-02 | 390 |
| 2026-08-12 | 391 |
| 2026-08-18 | 392 |
| 2026-08-23 | 394 |
| 2026-08-25 | 394 |
| 2026-08-26 | 395 |
| 2026-08-28 | 394 |
| 2026-09-08 | 395 |
| 2026-09-15 | 395 |
| 2026-09-17 | 395 |
| 2026-09-19 | 395 |
| 2026-09-20 | 395 |
Today
— stars today
This week
— stars this week
This month
+3 stars this month
Momentum
0.0
growth rate 0.00%/day
## Open Language Model Evaluation System (OLMES)
The OLMES (Open Language Model Evaluation System) repository is used within [Ai2](https://allenai.org)'s Open
Language Model efforts to evaluate base and
instruction-tuned LLMs on a range of tasks.
<details>
<summary>more details</summary>
The repository includes code to faithfully reproduce the evaluation results:
* **OLMo 3:** TBD Title ([TBD Citation](...))
* **OLMo 2:** 2 OLMo 2 Furious ([Team OLMo et al, 2024](https://arxiv.org/abs/2501.00656))
* **TÜLU 3:** Pushing Frontiers in Open Language Model Post-Training ([Lambert et al, 2024](https://www.semanticscholar.org/paper/T/%22ULU-3%3A-Pushing-Frontiers-in-Open-Language-Model-Lambert-Morrison/5ca8f14a7e47e887a60e7473f9666e1f7fc52de7))
* **OLMES:** A Standard for Language Model Evaluations ([Gu et al, 2024](https://www.semanticscholar.org/paper/c689c37c5367abe4790bff402c1d54944ae73b2a))
* **OLMo:** Accelerating the Science of Language Models ([Groeneveld et al, 2024](https://www.semanticscholar.org/paper/ac45bbf9940512d9d686cf8cd3a95969bc313570))
The code base uses helpful features from the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
by Eleuther AI, with a number of modifications and enhancements, including:
* Support deep configurations for variants of tasks
* Record more detailed data about instance-level predictions (logprobs, etc)
* Custom metrics and metric aggregations
* Integration with external storage options for results
</details>
### Setup
```sh
git clone https://github.com/allenai/olmes.git
cd olmes
# To install with uv:
uv sync
uv sync --group gpu # for vLLM support
# To install with pip:
pip install -e .
pip install -e ".[gpu]" # for vLLM support
```
## Usage
To evaluate a model and task (or task suite):
```bash
olmes \
--model allenai/OLMo-2-0425-1B \
--task arc_challenge::olmes \
--output-dir workspace
```
This will launch the standard [OLMES](https://www.semanticscholar.org/paper/c689c37c5367abe4790bff402c1d54944ae73b2a)
version of [ARC Challenge](https://www.semanticscholar.org/paper/88bb0a28bb58d847183ec505dda89b63771bb495)
(which uses a curated 5-shot example, trying both multiple-choice and cloze formulations, and reporting
the max) with [OLMo 2 1B](https://huggingface.co/allenai/OLMo-2-0425-1B), storing the output in `workspace`.
### More options
**Multiple tasks** can be specified after `--task`:
```bash
olmes \
--model allenai/OLMo-2-0425-1B \
--task arc_challenge::olmes hellaswag::olmes \
--output-dir workspace
```
**Inspect tasks.** You can sanity check using `--inspect`, which shows a sample prompt (and does 5-instance eval with a small `pythia`):
```bash
olmes --task arc_challenge:mc::olmes --inspect
```
**Dry run.** You can inspect the launch command with `--dry-run`:
```bash
olmes \
--model allenai/OLMo-2-0425-1B \
--task mmlu::olmes \
--output-dir workspace \
--dry-run
```
For a full list of arguments run `olmes --help`.
## Running Eval Suites
We include full suites for our releases in [`task_suites.py`](oe_eval/configs/task_suites.py):
### Olmo 3 eval suite
To run all tasks specified in the [OLMo 3](.) technical report:
**Base Model Evaluation**
```bash
# Run the base easy evaluation (for evaluating small-scale experiments)
olmes \
--model allenai/Olmo-3-1025-7B \
--task \
olmo3:base_easy:code_bpb \
olmo3:base_easy:math_bpb \
olmo3:base_easy:qa_rc \
olmo3:base_easy:qa_bpb \
--output-dir workspace
# Run the base main evaluation
olmes \
--model allenai/Olmo-3-1025-7B \
--task \
olmo3:base:stem_qa_mc \
olmo3:base:nonstem_qa_mc \
olmo3:base:gen \
olmo3:base:math \
olmo3:base:code \
olmo3:base:code_fim \
--output-dir workspace
# Run the base held-out evaluation
olmes \
--model allenai/Olmo-3-1025-7B \
--task \
olmo3:heldout \
--output-dir workspaExcerpt of 9,854 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:88682c566061ce29, desc:reproducible