Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A simple toolkit for benchmarking LLMs on mathematical reasoning tasks. 🧮✨
| Date | Stars |
|---|---|
| 2026-07-31 | 278 |
| 2026-08-04 | 278 |
| 2026-08-06 | 278 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# LLM Math Evaluation Harness
A unified, precise, and extensible toolkit to benchmark LLMs on various mathematical tasks 🧮✨.
> 🔴🚀 **Important Notice**: We've identified variances above 5% in results from diverse math evaluation frameworks. To ensure fair and standardized comparisons across research, our toolkit strives to harmonize evaluation methods, promoting consistent and reliable math evaluation.
> 🌟 **In Practice**: Esteemed projects like [ToRA](https://github.com/microsoft/ToRA) (ICLR'24) and [DeepSeek-Coder](https://github.com/deepseek-ai/DeepSeek-Coder/tree/main/Evaluation/PAL-Math) have leveraged this suite!
### Features:
- **Models**: Seamless compatibility with models from Hugging Face 🤗 and [vLLM](https://github.com/vllm-project/vllm).
- **Datasets**: An extensive array of datasets including `minerva_math`, `math`, `math_oai`, `gsm8k`, `gsm_hard`, `svamp`, `asdiv`, `mawps`, `tabmwp`, `finqa`, `theorem_qa`, `bbh`, `mmlu_stem`, `sat_math`, `mathqa`, `hungarian_exam`.
- **Prompts**: Diverse prompting paradigms, from Direct to Chain-of-Thought (CoT), Program-of-Thought (PoT/PAL), and [Tool-Integrated Reasoning (ToRA)](https://github.com/microsoft/ToRA).
## 🚀 Getting Started
### ⚙️ Environment Setup
#### Option 1: Conda
```
conda create -n math_eval python=3.10
conda activate math_eval
```
#### Option 2: Docker
We suggest using vLLM docker directly:
```
docker run --network host --cap-add=SYS_ADMIN --privileged -d \
--entrypoint '' --name vllm \
--runtime nvidia --gpus all \
--security-opt apparmor:unconfined \
--ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
-v /mnt:/mnt \
-p 8000:8000 \
vllm/vllm-openai:latest \
sleep infinity
```
### Install
```
git clone https://github.com/ZubinGou/math-evaluation-harness.git
cd math-evaluation-harness
pip install -r requirements.txt
```
### ⚖️ Evaluation
1. Configure model and data settings in `scripts/run_math_eval.sh`, and set the `PROMPT_TYPE` variable accordingly:
- For base models, choose from: `direct`, `cot`, `pal`, or `tool-integrated`.
- For SFT models, your options include: `tora`, `wizard_zs`, `deepseek-math`, etc.
- To add new models, update the `construct_prompt` function in `utils.py` to include your new prompt template.
2. Run the script:
```bash
bash scripts/run_eval.sh $PROMPT_TYPE $MODEL_NAME_OR_PATH
```
## 📊 Results
### Base Models (CoT)
> PROMPT_TYPE=cot
| Model | Size | Data | Uniq. Token | Train Token | GSM8K | MATH[^1] | SVAMP | ASDiv | MAWPS | TAB[^2] | MQA | MMLU STEM | SAT | AVG |
|---------------------------------------------------------------|--------------------------|--------|--------------|------------|-------|----------------|-------|-------|-------|-------|------|-----------|----------------|------|
| **1-2B Base Models** | | | | | | | | | | | | | | |
| [Tinyllama](https://huggingface.co/Tinyllama/Tinyllama-1.1B-intermediate-step-1431k-3T) | 1.1B | - | - | - | 2.9 | 3.2 | 11.0 | 18.1 | 20.4 | 12.5 | 14.6 | 16.1 | 21.9 | 13.4 |
| [Phi-1.5](https://huggingface.co/microsoft/phi-1_5) | 1.3B | - | - | - | 32.4 | 4.2 | 43.4 | 53.1 | 66.2 | 24.4 | 14.3 | 21.8 | 18.8 | 31.0 |
| [Qwen1.5](https://huggingface.co/Qwen/Qwen1.5-1.8B) | 1.8B | - | - | - | 36.1 | 6.8 | 48.5 | 63.6 | 79.0 | 29.2 | 25.1 | 31.3 | 40.6 | 40.0 |
| [Gemma](https://huggingface.co/google/gemma-2b) | 2.0B | - | - | - | 18.8 | 11.4 | 38.0 | 56.6 | 72.5 |Excerpt of 10,146 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:3d6db80879030a97, llm:Repository description: 'A simple toolkit for benchmarking LLMs on mathematical reasoning tasks.' Language: Python. Purpose: benchmarking LLMs on math reasoning.
matched fp:3d6db80879030a97, llm:Repository description: 'A simple toolkit for benchmarking LLMs on mathematical reasoning tasks.' Language: Python. Purpose: benchmarking LLMs on math reasoning.