Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
DAWNBench: An End-to-End Deep Learning Benchmark and Competition
| Date | Stars |
|---|---|
| 2026-07-24 | 262 |
| 2026-07-25 | 262 |
| 2026-07-28 | 262 |
| 2026-07-30 | 262 |
| 2026-07-31 | 262 |
| 2026-08-06 | 262 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# DAWNBench Submission Instructions
Thank you for the interest in DAWNBench!
To add your model to our leaderboard, open a Pull Request with title `<Model name> || <Task name> || <Author name>`
([example PR](https://github.com/stanford-futuredata/dawn-bench-entries/pull/1)), with JSON
(and TSV where applicable) result files in the format outlined below.
## Tasks
* [CIFAR10 Training](#cifar10-training)
* [CIFAR10 Inference](#cifar10-inference)
* [ImageNet Training](#imagenet-training)
* [ImageNet Inference](#imagenet-inference)
* [SQuAD Training](#squad-training)
* [SQuAD Inference](#squad-inference)
## CIFAR10 Training
### Task Description
We evaluate image classification performance on the [CIFAR10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html).
For training, we have two metrics:
- **Training Time:** Train an image classification model for the CIFAR10 dataset. Report the time needed to train
a model with test set accuracy of at least 94%
- **Cost:** On public cloud infrastructure, compute the total time needed to reach a test set accuracy of
94% or greater, as outlined above. Multiply the time taken (in hours) by the cost of the instance per hour, to obtain the
total cost of training the model
Including cost is optional and will only be calculated if the `costPerHour` field is included in the JSON file.
Submissions that only aim for time aren't restricted to public cloud infrastructure.
### JSON Format
Results for the CIFAR10 training tasks can be reported using a JSON file with the following fields,
- `version`: DAWNBench competition version (currently v1.0)
- `author`: Author name
- `authorEmail`: Author email
- `framework`: Framework on which training / inference was performed
- `codeURL`: [Optional] URL pointing to code for model
- `model`: Model name
- `hardware`: A short description of the hardware on which model training was performed. If relevant,
please specify Cloud provider and instance type to make results more reproducible
- `costPerHour`: [Optional] Reported in USD ($). Cost of instance per hour
- `timestamp`: Date of submission in format `yyyy-mm-dd`
- `logFilename`: [Optional] URL pointing to training logs
- `misc`: [Optional] JSON object of other miscellaneous notes, such as learning rate schedule, optimization algorithm,
framework version, etc.
In addition, report training progress at the end of every epoch in a TSV with the following format,
```epoch\thours\ttop1Accuracy```
We will compute time to reach a test set accuracy of 94% by reading off the first entry in the above TSV
with a top-1 test set accuracy of at least 94%.
JSON and TSV files are named `[author name]_[model name]_[hardware tag]_[framework].json`, similar to
`dawn_resnet56_1k80-gc_tensorflow.[json|tsv]`. Put the JSON and TSV files in the `CIFAR10/train/` sub-directory.
### Example JSON and TSV
#### JSON
```JSON
{
"version": "v1.0",
"author": "Stanford DAWN",
"authorEmail": "[email protected]",
"framework": "TensorFlow",
"codeURL": "https://github.com/stanford-futuredata/dawn-benchmark/tree/master/tensorflow",
"model": "ResNet 56",
"hardware": "1 K80 / 30 GB / 8 CPU (Google Cloud)",
"costPerHour": 0.90,
"timestamp": "2017-08-14",
"misc": {}
}
```
#### TSV
```TSV
epoch hours top1Accuracy
1 0.07166666666666667 33.57
2 0.1461111111111111 52.51
3 0.21805555555555556 61.71
4 0.2902777777777778 69.46
5 0.3622222222222222 71.47
6 0.43416666666666665 69.64
7 0.5061111111111111 75.81
```
<br/>
## CIFAR10 Inference
### Task Description
We evaluate image classification performance on the [CIFAR10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html).
For inference, we have two metrics:
- **Latency:** Use a model that has a test set accuracy of 94% or greater. Measure the total time needed to classify
all 10,000 images in the CIFAR10 test set _one-at-a-time_, and then dExcerpt of 19,056 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8220aec688ab1fae, topic:inference
matched fp:8220aec688ab1fae, topic:training