Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Automatically evaluate your LLMs in Google Colab
| Date | Stars |
|---|---|
| 2026-07-31 | 695 |
| 2026-08-06 | 695 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center">
<h1>🧐 LLM AutoEval</h1>
<p>
🐦 <a href="https://twitter.com/maximelabonne">Follow me on X</a> •
🤗 <a href="https://huggingface.co/mlabonne">Hugging Face</a> •
💻 <a href="https://mlabonne.github.io/blog">Blog</a> •
📙 <a href="https://github.com/PacktPublishing/Hands-On-Graph-Neural-Networks-Using-Python">Hands-on GNN</a>
</p>
<p><em>Simplify LLM evaluation using a convenient Colab notebook.</em></p>
<a href="https://colab.research.google.com/drive/1Igs3WZuXAIv9X0vwqiE90QlEPys8e8Oa?usp=sharing"><img src="img/colab.svg" alt="Open In Colab"></a></center>
</div>
<br/>
<p align="center">
<img src='img/llmautoeval.png'>
</p>
## 🔍 Overview
LLM AutoEval **simplifies the process of evaluating LLMs** using a convenient [Colab notebook](https://colab.research.google.com/drive/1Igs3WZuXAIv9X0vwqiE90QlEPys8e8Oa?usp=sharing). You just need to specify the name of your model, a benchmark, a GPU, and press run!
### Key Features
* Automated setup and execution using [RunPod](https://runpod.io?ref=9nvk2srl).
* Customizable evaluation parameters for tailored benchmarking.
* Summary generation and upload to [GitHub Gist](https://gist.github.com/) for easy sharing and reference.
[View a sample summary here.](https://gist.github.com/mlabonne/1d33c86824b3a11d2308e36db1ba41c1)
*Note: This project is in the early stages and primarily designed for personal use. Use it carefully and feel free to contribute.*
## ⚡ Quick Start
### Evaluation
* **`MODEL_ID`**: Enter the model id from Hugging Face.
* **`BENCHMARK`**:
* `nous`: List of tasks: AGIEval, GPT4ALL, TruthfulQA, and Bigbench (popularized by [Teknium](https://github.com/teknium1) and [NousResearch](https://github.com/NousResearch)). This is recommended.
* `lighteval`: This is a [new library](https://github.com/huggingface/lighteval) from Hugging Face. It allows you to specify your tasks as shown in the readme. Check the list of [recommended tasks](https://github.com/huggingface/lighteval/blob/main/examples/tasks/recommended_set.txt) to see what you can use (e.g., HELM, PIQA, GSM8K, MATH, etc.)
* `openllm`: List of tasks: ARC, HellaSwag, MMLU, Winogrande, GSM8K, and TruthfulQA (like the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)). It uses the [vllm](https://docs.vllm.ai/) implementation to enhance speed (note that the results will not be identical to those obtained without using vllm). "mmlu" is currently missing because of a problem with vllm.
* **`LIGHTEVAL_TASK`**: You can select one or several tasks as specified in the [readme](https://github.com/huggingface/lighteval?tab=readme-ov-file#usage) or in the list of [recommended tasks](https://github.com/huggingface/lighteval/blob/main/examples/tasks/recommended_set.txt).
### Cloud GPU
* **`GPU`**: Select the GPU you want for evaluation (see prices [here](https://www.runpod.io/console/gpu-cloud)). I recommend using beefy GPUs (RTX 3090 or higher), especially for the Open LLM benchmark suite.
* **`Number of GPUs`**: Self-explanatory (more cost-efficient than bigger GPUs if you need more VRAM).
* **`CONTAINER_DISK`**: Size of the disk in GB.
* **`CLOUD_TYPE`**: RunPod offers a community cloud (cheaper) and a secure cloud (more reliable).
* **`REPO`**: If you made a fork of this repo, you can specify its URL here (the image only runs `runpod.sh`).
* **`TRUST_REMOTE_CODE`**: Models like Phi require this flag to run them.
* **`PRIVATE_GIST`**: (W.I.P.) Make the Gist with the results private (true) or public (false).
* **`DEBUG`**: The pod will not be destroyed at the end of the run (not recommended).
### Tokens
Tokens use Colab's Secrets tab. Create two secrets called "runpod" and "github" and add the corresponding tokens you can find as follows:
* **`RUNPOD_TOKEN`**: Please consider using my [referral link](https://runpod.io?ref=9nvk2srl) if you don't have an account yet. You can create your token [here](https://www.runpod.io/coExcerpt of 7,024 characters
Read on GitHub
Maxime Labonne · Liquid AI · United Kingdom
58
huggingface · Belgium
7
3
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9cbb85561c1d37aa, llm:Repository description: "Automatically evaluate your LLMs in Google Colab" (mlabonne/llm-autoeval). Language: Python.
matched fp:9cbb85561c1d37aa, llm:Repository description: "Automatically evaluate your LLMs in Google Colab" (mlabonne/llm-autoeval). Language: Python.
matched fp:9cbb85561c1d37aa, llm:Repository description: "Automatically evaluate your LLMs in Google Colab" (mlabonne/llm-autoeval). Language: Python.