Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A framework for the evaluation of autoregressive code generation language models.
| Date | Stars |
|---|---|
| 2026-07-31 | 1055 |
| 2026-08-01 | 1056 |
| 2026-08-06 | 1056 |
| 2026-08-12 | 1054 |
| 2026-08-19 | 1055 |
| 2026-08-21 | 1057 |
| 2026-08-23 | 1057 |
| 2026-08-25 | 1058 |
| 2026-08-27 | 1058 |
| 2026-09-08 | 1059 |
| 2026-09-12 | 1060 |
| 2026-09-15 | 1061 |
| 2026-09-16 | 1062 |
| 2026-09-20 | 1062 |
Today
— stars today
This week
+2 stars this week
This month
+5 stars this month
Momentum
0.0
growth rate 0.19%/day
<h1 align="center">Code Generation LM Evaluation Harness</h1>
<h4 align="center">
<p>
<a href="#features">Tasks</a> |
<a href="#setup">Usage</a> |
<a href="#implementing-new-tasks">Contribution</a> |
<a href="#documentation">Documentation</a> |
<a href="https://huggingface.co/bigcode">BigCode</a>
<p>
</h4>
<h3 align="center">
<img style="float: middle; padding: 10px 10px 10px 10px;" width="50" height="50" src="https://user-images.githubusercontent.com/44069155/191557209-6219acb8-a766-448c-9bd6-284d22b1e398.png" /></a>
</h3>
## Features
This is a framework for the evaluation of code generation models. This work is inspired from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) for evaluating language models in general. We welcome contributions to fix issues, enhance features and add new benchmarks. You can find contribution guides in [`docs/guide.md`](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/guide.md) and [`CONTRIBUTING.md`](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/CONTRIBUTING.md) and more documentation in [`docs/README.md`](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/README.md).
Below are the features and tasks of this framework:
- Features:
- Any autoregressive model available on [Hugging Face hub](https://huggingface.co/) can be used, but we recommend using code generation models trained specifically on Code such as [SantaCoder](https://huggingface.co/bigcode/santacoder), [InCoder](https://huggingface.co/facebook/incoder-6B) and [CodeGen](https://huggingface.co/Salesforce/codegen-16B-mono).
- We provide Multi-GPU text generation with `accelerate` and Dockerfiles for evaluating on Docker containers for security and reproducibility.
- Tasks:
- 7 code generation **Python** tasks (with unit tests): [HumanEval](https://huggingface.co/datasets/openai_humaneval), [HumanEval+](https://huggingface.co/datasets/evalplus/humanevalplus), [InstructHumanEval](https://huggingface.co/datasets/codeparrot/instructhumaneval), [APPS](https://huggingface.co/datasets/codeparrot/apps), [MBPP](https://huggingface.co/datasets/mbpp), [MBPP+](https://huggingface.co/datasets/evalplus/mbppplus), and [DS-1000](https://github.com/HKUNLP/DS-1000/) for both completion (left-to-right) and insertion (FIM) mode.
- [HumanEvalPack](https://huggingface.co/datasets/bigcode/humanevalpack) extends HumanEval to **3** scenarios across **6** languages via human translations and was released with [OctoPack](https://arxiv.org/abs/2308.07124).
- [MultiPL-E](https://github.com/nuprl/MultiPL-E) evaluation suite (HumanEval translated into **18** programming languages).
- [Recode](https://github.com/amazon-science/recode/tree/main) applied to the HumanEval benchmark. It evaluates the robustness of code-generation models.
- [Pal](https://github.com/reasoning-machines/pal) Program-aided Language Models evaluation for grade school math problems : [GSM8K](https://huggingface.co/datasets/gsm8k) and [GSM-HARD](https://huggingface.co/datasets/reasoning-machines/gsm-hard). These problems are solved by generating reasoning chains of text and code.
- Code to text task from [CodeXGLUE](https://huggingface.co/datasets/code_x_glue_ct_code_to_text) (zero-shot & fine-tuning) for 6 languages: **Python, Go, Ruby, Java, JavaScript and PHP.** Documentation translation task from [CodeXGLUE](https://huggingface.co/datasets/code_x_glue_tt_text_to_text).
- [CoNaLa](https://huggingface.co/datasets/neulab/conala) for **Python** code generation (2-shot setting and evaluation with BLEU score).
- [Concode](https://huggingface.co/datasets/code_x_glue_tc_text_to_code) for **Java** code generation (2-shot setting and evaluation with BLEU score).
- 3 multilingual downstream classification tasks: [Java Complexity prediction](https://huggingface.co/datasets/codeparrot/cExcerpt of 14,543 characters
Read on GitHub
Loubna Ben Allal · @huggingface · France
424
Niklas
193
41
39
Ben Lipkin · MIT · United States
38
22
Thomas Wolf · @huggingface
13
11
Arjun Guha · @nuprl · Morocco
10
7
Jiawei Liu · University of Illinois Urbana-Champaign
7
Manan Dey · Loading.. · India
6
4
4
3
Terry Yue Zhuo
3
3
3
2
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a3fa095ed6c6e083, name:evaluation harness