Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A complete GPT language model (training and inference) in ~600 lines of pure C#, zero dependencies
| Date | Stars |
|---|---|
| 2026-07-24 | 397 |
| 2026-07-25 | 397 |
| 2026-07-28 | 397 |
| 2026-07-30 | 397 |
| 2026-07-31 | 397 |
| 2026-08-07 | 397 |
| 2026-09-01 | 398 |
| 2026-09-02 | 399 |
| 2026-09-17 | 401 |
| 2026-09-20 | 401 |
Today
— stars today
This week
+2 stars this week
This month
+4 stars this month
Momentum
0.0
growth rate 0.50%/day
# MicroGPT.cs
[](https://github.com/milanm/AutoGrad-Engine/actions/workflows/build.yml)
[](LICENSE)
[](https://dotnet.microsoft.com/)
A complete [GPT](https://en.wikipedia.org/wiki/Generative_pre-trained_transformer) language model (training and inference) in pure C# with zero dependencies.
Faithful port of [Andrej Karpathy's microgpt.py](https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95).
## What is this?
This is the exact same algorithm that powers ChatGPT, in ~600 lines of code across 4 files (plus extensive comments explaining every piece). No PyTorch, no TensorFlow, no NuGet packages. Just plain C# and math.
It trains a tiny GPT model on a list of human names, then generates new ones that sound real but never existed.
**This is not production code.** It's an educational tool. It processes one number at a time, whereas real implementations process millions in parallel on GPUs. But every conceptual piece of a real GPT is here.
**New to ML?** Start with the [Prerequisites guide](PREREQUISITES.md) — it covers all the math and ML concepts you need, from scratch.
## Project Structure
| File | Responsibility |
|---|---|
| `Value.cs` | Autograd engine — wraps scalars with automatic gradient tracking |
| `Tokenizer.cs` | Character-level tokenizer with `Encode()`/`Decode()` |
| `NeuralOps.cs` | Stateless neural-net building blocks: `Linear`, `Softmax`, `RMSNorm` |
| `Program.cs` | GPT model, training loop (`Train`), and generation (`Generate`) |
| `ValueTests.cs` | 25 tests — numerical gradient checking, ops correctness, roundtrips |
## Architecture
```mermaid
flowchart TD
subgraph LOOP["TRAINING LOOP (× num_steps)"]
A["'emma'"] -->|Tokenizer| B["[BOS, e, m, m, a, EOS]"]
subgraph GPT["GPT MODEL"]
C["Token Embedding + Position Embedding → x"]
subgraph TF["Transformer Layer (× n_layer)"]
E["RMSNorm → Multi-Head Attention\n(Q·K/√d → softmax → V)\n+ Residual Connection"]
F["RMSNorm → MLP\n(expand → ReLU² → compress)\n+ Residual Connection"]
E --> F
end
C --> TF
TF --> G["Linear (weight-tied with Token Embedding)"]
end
B --> C
G --> H["Softmax → Probabilities"]
H --> I["Cross-Entropy Loss"]
I --> J["Backward() ← Value autograd engine\n(chain rule through computation graph)"]
J --> K["Adam Optimizer\n(update all parameters)"]
end
```
## Quick Start
```bash
cd src/AutogradEngine
dotnet run
```
Or with custom settings:
```bash
dotnet run -- --n_embd 32 --n_layer 2 --num_steps 2000
```
### Running Tests
The autograd engine is verified with numerical gradient checking — the same technique PyTorch uses in `torch.autograd.gradcheck`:
```bash
dotnet test
```
### CLI Arguments
| Argument | Default | What it does |
|---|---|---|
| `--n_embd` | 16 | Size of each token's vector representation |
| `--n_layer` | 1 | Number of transformer layers |
| `--block_size` | 8 | Maximum sequence length (tokens the model can "see") |
| `--num_steps` | 1000 | Number of training iterations |
| `--n_head` | 4 | Number of attention heads |
| `--learning_rate` | 0.01 | How aggressively to update parameters |
| `--seed` | 42 | Random seed for reproducibility |
## What you should see
```
vocab size: 28, num docs: 32033
num params: 3648
step 1 / 1000 | loss 3.3327
step 2 / 1000 | loss 3.3090
...
step 1000 / 1000 | loss 2.1844
--- generation ---
sample 0: jayede
sample 1: kal
sample 2: mede
sample 3: si
sample 4: ren
```
The loss starts around 3.3 (random guessing on 28 characters = ln(28) ≈ 3.33) and drops over training. The generated names aren't real, but they follow English-like patterns.
## How GPT Works — A Developer's Guide
IfExcerpt of 15,597 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:cdb187dcccd32ceb, topic:transformer, topic:gpt, topic:language-model
matched fp:cdb187dcccd32ceb, topic:deep-learning, topic:neural-network, name:autograd