Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Minimal and annotated implementations of key ideas from modern deep learning research.
| Date | Stars |
|---|---|
| 2026-07-31 | 1336 |
| 2026-08-01 | 1336 |
| 2026-08-06 | 1339 |
Today
+3 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
## Beyond NanoGPT: Go From LLM Beginner to AI Researcher!
<p align="center">
<span style="display: inline-block; text-align: center; margin: 0 10px;">
<img src="https://github.com/user-attachments/assets/b2943618-d5ed-468d-b792-d1cf4e0d6c6a" style="width: 350px; height: auto;" />
</span>
</p>
**Beyond-NanoGPT** is the minimal and educational repo aiming to **bridge between nanoGPT and research-level deep learning.**
This repo includes annotated and from-scratch implementations of almost 100 crucial modern techniques in frontier deep learning, aiming to help newcomers learn enough to start running experiments of their own.
The repo implements everything from KV caching and speculative decoding for LLMs to
architectures like vision transformers and MLP-mixers; from attention variants like linear or multi-latent attention to generative models like denoising diffusion models and flow matching algorithms; from landmark RL papers like PPO, A3C, and AlphaZero to
systems fundamentals like GPU communication algorithms and data/tensor parallelism.
**Because everything is implemented by-hand, the code comments explain the especially subtle details often glossed over both in papers and production codebases.**
<p align="center">
<span style="display: inline-block; text-align: center; margin: 0 10px;">
<a href="https://github.com/user-attachments/assets/e49fad0a-f51b-4771-a59a-f5d6a969f8ed">
<img src="https://github.com/user-attachments/assets/e49fad0a-f51b-4771-a59a-f5d6a969f8ed" />
</a>
<div style="text-align: center; max-width: 600px; margin-top: 8px;">
<sub>
A glimpse of some plots you can make! <br />
<b>(Left)</b> Language model speedups from
<code>attention-variants/linear_attention.ipynb</code>,<br />
<b>(Center)</b> Samples from a small denoising diffusion model trained on MNIST in
<code>generative-models/train_ddpm.py</code>,<br />
<b>(Right)</b> Reward over time for a small MLP policy on CartPole in
<code>rl/fundamentals/train_ppo.py</code>.
</sub>
</div>
</span>
</p>
`LESSONS.md` documents some of the things I've learned in the months spent writing this codebase.
## Quickstart
1. **Clone the Repo:**
```bash
git clone https://github.com/tanishqkumar/beyond-nanogpt.git
```
2. **Get Minimal Dependencies:**
```bash
pip install torch numpy torchvision wandb tqdm transformers datasets diffusers matplotlib pillow jupyter gym
```
3. **Start learning!**
The code is meant for you to read carefully, hack around with, then re-implement yourself from scratch and compare to.
You can just run `.py` files with vanilla Python in the following way.
```bash
cd architectures/
python train_dit.py
```
or for instance
```bash
cd rl/fundamentals/
python train_reinforce.py --verbose --wandb
```
Everything is written to be run on a single GPU. The code is self-documenting with comments for intuition and elaborating
on subtleties I found tricky to implement.
Arguments are specified at the bottom of each file.
Jupyter notebooks are meant to be stepped through.
## Current Implementations and Roadmap
Asterisks (*) denote particularly tricky implementations.
### Architectures
- [x] Basic Transformer `language-models/transformer.py` and `train_naive.py` [[paper]](https://arxiv.org/abs/1706.03762)
- [x] Vision Transformer (ViT) `architectures/train_vit.py` [[paper]](https://arxiv.org/abs/2010.11929)
- [x] Diffusion Transformer (DiT) `architectures/train_dit.py` [[paper]](https://arxiv.org/abs/2212.09748)
- [x] Recurrent Neural Network (RNN) `architectures/train_rnn.py` [[paper]](https://arxiv.org/abs/1506.00019)
- [x] Residual Networks (ResNet) `architectures/train_resnet.py` [[paper]](https://arxiv.org/abs/1512.03385)
- [x] MLP-Mixer `architectures/train_mlp_mixer.py` [[paper]](https://arxiv.org/abs/2105.01601)
- [x] LSTM `architectures/train_lstm.py` [[paper]](httExcerpt of 11,859 characters
Read on GitHub
369
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9c5c21b87dd21f9e, llm:description: 'Minimal and annotated implementations of key ideas from modern deep learning research.' (repository description)
matched fp:9c5c21b87dd21f9e, llm:description: 'Minimal and annotated implementations of key ideas from modern deep learning research.' (repository description)
matched fp:9c5c21b87dd21f9e, llm:description: 'Minimal and annotated implementations of key ideas from modern deep learning research.' (repository description)