Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A benchmark to evaluate language models on questions I've previously asked them to solve.
| Date | Stars |
|---|---|
| 2026-07-31 | 1063 |
| 2026-08-03 | 1063 |
| 2026-08-06 | 1063 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Yet Another Applied LLM Benchmark
This is a benchmark I made, for me, to test how well language models perform
on tasks I care about. I know I care about them because each test is directly
derived from something I've asked a LLM to perform for me in the last year.
For example, there are tests in this benchmark that evaluate the ability of a model to:
- convert a python function to an equivalent-but-faster c function;
- decompile python bytecode to functional source code;
- explain the functionality of minified javascript;
- identify the encoding format (in this case, uuencoded) of some data;
- write a parser from a BNF-like grammar;
- convert some english sentences to SQL queries; or,
- write some bash oneliners
There are two defining features of this benchmark that make it interesting:
1. I've implemented a simple dataflow domain specific language to make it easy for
me (or anyone else!) to add new tests that realistically evaluate model capabilities.
2. Directly as a result of this, I've written nearly 100 tests for different
situations I've actually encountered when working with LLMs as assistants.
For example, here's a complete test to evaluate if a model can write a "hello world" program.
```python
'Write a "hello world" program in python' >> LLMRun() >> PythonRun() >> SubstringEvaluator("hello world")
```
I first ask the model to write hello world, then run that program (in a docker
container, seamlessly behind the scenes), and then check if stdout contains "hello world".
(You should read the `>>` operator as "and then do". So "a >> b" means "do a, and then do b".)
More interestingly, here's a test case that first asks a model to produce code that
draws a flag. To evaluate it, I run the code, partially judge it with another
language model, and then fully evaluate the output by comparing to a reference solution.
```python
"Write a C program that draws an american flag to stdout." >> LLMRun() >> CRun() >> \
LLMRun("What flag is shown in this image?") >> \
(SubstringEvaluator("United States") | SubstringEvaluator("USA") | SubstringEvaluator("America"))
```
This DSL makes it easy for me to evaluate significantly more diverse and
more sophisticated behavior than any other evaluation benchmark I'm aware of.
This is helpful for determining whether or not models are capable of performing tasks I actually care about.
## Results
I've evaluated a few models on this benchmark. Here's how they perform:
* o1-mini: 62% passed
* Claude 3.5 Sonnet: 56% passed
* GPT 4o: 48% passed
* Gemini 1.5 Pro: 43% passed
* Claude 3 Opus: 42% passed
* GPT 4o Mini: 36% passed
* Mistral Large: 28% passed
* GPT 3.5: 26% passed
A complete evaluation grid is available [here](https://nicholas.carlini.com/writing/2024/evaluation_examples/index.html).
## What this is not
A serious academic benchmark.
In more words: this is not meant to try to rigorously evaluate the capabilities of
models on any particular task. It's not meant to be something you can use to decide
which model is more capable, more knowledgeable, more factual, less biased, less
harmful, more aligned, more helpful, or anything else.
The questions are not optimally prompt-engineered. It is entirely
possible---and indeed likely!---that a better phrasing of some of the questions
would allow the model to give a better answer.
But I am lazy.
I do not want to remind the model it is AN EXPERT IN PYTHON
and tell it that I'll give it a $100,000 tip for giving the right answer
OR I WILL MURDER A KITTEN but please pause....take a deep breath....and think step
by step by step before answering.
(Or whatever the current incantation is people use to get models to work best.)
I just want to type my question and get the right answer.
So this benchmark tests for that,
on types of questions I've actually cared about having answered.
### Failing a question doesn't mean much
As a result of my (often intentional) lack of prompt engineering,
when a model fails a question, you woExcerpt of 16,716 characters
Read on GitHub
Nicholas Carlini
58
Viswa · Amazon · United States
4
3
2
2
2
Anton Sokolchenko · Sauce Labs · Germany
1
1
Alessio Fanelli · @Kernel-Labs-AI · United States
1
minakokojima · Google · China
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:381b470688362216, llm:Description: 'A benchmark to evaluate language models on questions I've previously asked them to solve.' Language: Python. No topics.
matched fp:381b470688362216, llm:Description: 'A benchmark to evaluate language models on questions I've previously asked them to solve.' Language: Python. No topics.