Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A straightforward method for training your LLM, from downloading data to generating text.
| Date | Stars |
|---|---|
| 2026-07-24 | 8633 |
| 2026-07-25 | 8653 |
| 2026-07-28 | 8653 |
| 2026-07-30 | 8653 |
| 2026-07-31 | 8795 |
| 2026-08-06 | 8795 |
Today
— stars today
This week
+142 stars this week
This month
— stars this month
Momentum
127.0
growth rate 1.64%/day
<div align="center">
<!-- omit in toc -->
# Train LLM From Scratch
   [](https://fareedkhan-dev.github.io/train-llm-from-scratch/)
**I am Looking for a PhD position in AI**. [GitHub](https://github.com/FareedKhan-dev)
</div>
I implemented a transformer model from scratch using PyTorch, based on the paper [Attention is All You Need](https://arxiv.org/abs/1706.03762). You can use my scripts to train your own **billion** or **million** parameter LLM using a single GPU.
This started as a pretraining tutorial. It now goes all the way from raw text to an aligned, reasoning style model, with every algorithm hand written in plain PyTorch (no `trl`, no `peft`, no `transformers`). The whole journey is one idea repeated: turn text into numbers, predict the next token, then keep changing the data and the loss until the model does what we want.
Here is the path we will walk, end to end:
```
raw text -> tokens -> a Transformer -> next-token loss -> a base model
base model -> SFT -> Reward Model -> {PPO, DPO} -> GRPO -> evaluation and chat
```
Below is the output of a trained 13 million parameter LLM, just so you can see where the small end of this starts:
```
In ***1978, The park was returned to the factory-plate that
the public share to the lower of the electronic fence that
follow from the Station's cities. The Canal of ancient Western
nations were confined to the city spot. The villages were directly
linked to cities in China that revolt that the US budget and in
Odambinais is uncertain and fortune established in rural areas.
```
<!-- omit in toc -->
## Table of Contents
- [Who this is for](#who-this-is-for)
- [Prerequisites and Training Time](#prerequisites-and-training-time)
- [Setup](#setup)
- [Code Structure](#code-structure)
- [Step 1: Preparing the Data](#step-1-preparing-the-data)
- [Step 2: The Model, Built From Small Pieces](#step-2-the-model-built-from-small-pieces)
- [Multi Layer Perceptron (MLP)](#multi-layer-perceptron-mlp)
- [Single Head Attention](#single-head-attention)
- [Multi Head Attention](#multi-head-attention)
- [The Transformer Block](#the-transformer-block)
- [The Full Transformer](#the-full-transformer)
- [Step 3: Pretraining the Base Model](#step-3-pretraining-the-base-model)
- [Step 4: Generating Text](#step-4-generating-text)
- [Step 5: Post-Training, Turning a Base Model Into an Assistant](#step-5-post-training-turning-a-base-model-into-an-assistant)
- [SFT (Supervised Fine-Tuning)](#sft-supervised-fine-tuning)
- [The Reward Model](#the-reward-model)
- [DPO, ORPO and KTO](#dpo-orpo-and-kto)
- [PPO](#ppo)
- [GRPO / RLVR](#grpo--rlvr)
- [Step 6: Evaluation](#step-6-evaluation)
- [Step 7: Talking to the Model](#step-7-talking-to-the-model)
- [The Streamlit Control Panel](#the-streamlit-control-panel)
- [The Documentation Site](#the-documentation-site)
- [Run the Whole Thing](#run-the-whole-thing)
- [What's Next](#whats-next)
## Who this is for
I tried to write this so one page works for very different readers:
- If you are a **student**, read top to bottom. Every block of code comes after a plain explanation of what it does and why, and most blocks are followed by the output you should expect.
- If you are a **developer**, the commands and file paths are all here. You can copy, run, and read the referenced source files directly.
- If you are a **researcher**, the post-training half is the interesting part: SFT, a Bradley-Terry reward model, PPO with GAE, DPO/ORPO/KTO, and GRPO, all from scratch on the same small Transformer, trained on real public datasets.
EvExcerpt of 38,133 characters
Read on GitHub
Fareed Khan · Pakistan
37
19
4
2
1
Schleiferdyne Systems · Schleiferdyne Systems
1
Ikko Eltociear Ashimine · Japan
1
1
1
RoomWithOutRoof · @p-to-q @scanapi @Spark-Lab-Global · Singapore
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:811aedaee69360f7, topic:large-language-models, topic:llm
matched fp:811aedaee69360f7, topic:training, readme:pretraining