Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Minimalistic large language model 3D-parallelism training
| Date | Stars |
|---|---|
| 2026-08-05 | 2775 |
| 2026-08-06 | 2775 |
| 2026-08-08 | 2778 |
| 2026-08-09 | 2779 |
| 2026-08-11 | 2780 |
| 2026-08-12 | 2782 |
| 2026-08-13 | 2783 |
| 2026-08-15 | 2783 |
| 2026-08-17 | 2784 |
| 2026-08-18 | 2786 |
| 2026-08-19 | 2790 |
| 2026-08-20 | 2791 |
| 2026-08-21 | 2792 |
| 2026-08-24 | 2793 |
| 2026-08-25 | 2794 |
| 2026-08-26 | 2798 |
| 2026-08-27 | 2800 |
| 2026-08-28 | 2802 |
| 2026-08-29 | 2804 |
| 2026-08-30 | 2805 |
| 2026-09-02 | 2806 |
| 2026-09-03 | 2808 |
| 2026-09-04 | 2807 |
| 2026-09-05 | 2808 |
| 2026-09-06 | 2808 |
| 2026-09-07 | 2809 |
| 2026-09-08 | 2810 |
| 2026-09-09 | 2812 |
| 2026-09-10 | 2815 |
| 2026-09-11 | 2816 |
| 2026-09-12 | 2819 |
| 2026-09-13 | 2821 |
| 2026-09-14 | 2823 |
| 2026-09-15 | 2824 |
| 2026-09-16 | 2825 |
| 2026-09-18 | 2825 |
| 2026-09-23 | 2827 |
| 2026-09-27 | 2826 |
| 2026-09-29 | 2827 |
| 2026-09-30 | 2827 |
| 2026-10-02 | 2828 |
| 2026-10-03 | 2828 |
| 2026-10-04 | 2828 |
| 2026-10-05 | 2829 |
| 2026-10-06 | 2831 |
| 2026-10-07 | 2833 |
| 2026-10-08 | 2834 |
| 2026-10-10 | 2834 |
| 2026-10-11 | 2836 |
| 2026-10-12 | 2836 |
| 2026-10-13 | 2840 |
| 2026-10-14 | 2841 |
| 2026-10-15 | 2843 |
| 2026-10-19 | 2843 |
| 2026-10-20 | 2844 |
| 2026-10-22 | 2844 |
| 2026-10-23 | 2845 |
| 2026-10-24 | 2846 |
| 2026-10-26 | 2847 |
| 2026-10-27 | 2848 |
| 2026-10-29 | 2849 |
| 2026-10-30 | 2850 |
| 2026-11-01 | 2851 |
| 2026-11-02 | 2851 |
| 2026-11-03 | 2853 |
| 2026-11-04 | 2854 |
Today
+1 stars today
This week
+6 stars this week
This month
+25 stars this month
Momentum
10.0
growth rate 0.21%/day
<h1 align="center">⚡️ Nanotron</h1>
<p align="center">
<a href="https://github.com/huggingface/nanotron/releases">
<img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/nanotron.svg">
</a>
<a href="https://github.com/huggingface/nanotron/blob/master/LICENSE">
<img alt="License" src="https://img.shields.io/github/license/huggingface/nanotron.svg?color=green">
</a>
</p>
<h4 align="center">
<p>
<a href="#installation">Installation</a> •
<a href="#quick-start">Quick Start</a> •
<a href="#features">Features</a> •
<a href="#benchmarks">Benchmarks</a> •
<a href="CONTRIBUTING.md">Contributing</a>
<p>
</h4>
<h3 align="center">
<a href="https://huggingface.co/nanotron"><img style="float: middle; padding: 10px 10px 10px 10px;" width="60" height="55" src="https://huggingface.co/datasets/huggingface/brand-assets/resolve/main/hf-logo.png" /></a>
</h3>
<h3 align="center">
<p>Pretraining models made easy
</h3>
Nanotron is a library for pretraining transformer models. It provides a simple and flexible API to pretrain models on custom datasets. Nanotron is designed to be easy to use, fast, and scalable. It is built with the following principles in mind:
- **Simplicity**: Nanotron is designed to be easy to use. It provides a simple and flexible API to pretrain models on custom datasets.
- **Performance**: Optimized for speed and scalability, Nanotron uses the latest techniques to train models faster and more efficiently.
📚 **Check out our [Ultrascale Playbook](https://huggingface.co/spaces/nanotron/ultrascale-playbook)** - A comprehensive guide to efficiently scale LLM training with Nanotron!
📝 **AI generated docs thanks to [DeepWiki](https://deepwiki.com/huggingface/nanotron)**
## Installation
To run the code in this project, first create a Python virtual environment using e.g. `uv`:
```shell
uv venv nanotron --python 3.11 && source nanotron/bin/activate && uv pip install --upgrade pip
```
> [!TIP]
> For Hugging Face cluster users, add `export UV_LINK_MODE=copy` to your `.bashrc` to suppress cache warnings from `uv`
Next, install Pytorch:
```shell
uv pip install torch --index-url https://download.pytorch.org/whl/cu124
```
Then install the core dependencies with:
```shell
uv pip install -e .
```
To run the example scripts, install the remaining dependencies as follows:
```shell
uv pip install datasets transformers datatrove[io] numba wandb
# Fused kernels
uv pip install ninja triton "flash-attn>=2.5.0" --no-build-isolation
```
Next, log into your Hugging Face and Weights and Biases accounts as follows:
```shell
huggingface-cli login
wandb login
```
Finally, check whether your system has Git LFS installed so that you can load and push models/datasets to the Hugging Face Hub:
```shell
git-lfs --version
```
If it isn't installed, run:
```shell
sudo apt-get install git-lfs
```
## Quick Start
### Training a tiny Llama model
The following command will train a tiny Llama model on a single node of 8 x H100s in about 10 minutes:
```shell
CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc_per_node=8 run_train.py --config-file examples/config_tiny_llama.yaml
```
The model will be saved in the `checkpoints` directory as specified in the config file.
> [!NOTE]
> You can use `examples/config_tiny_llama.py` to generate your own training config
For detailed instructions on training your first model, check out our [Your First Training guide](docs/your-first-training.md). For multi-node training with Slurm, see our [Multi-Node Training guide](docs/multi-node-training.md).
### Run generation from your checkpoint
```shell
torchrun --nproc_per_node=1 run_generate.py --ckpt-path checkpoints/{checkpoint_number}/ --tp 1 --pp 1
```
Increase the value of `--tp` (tensor parallel) to accelerate generation with multiple GPUs and use a larger value of `--pp` (pipeline parallel) for very large models.
### Debugging with VSCode
To debExcerpt of 7,382 characters
Read on GitHub
Nouamane Tazi · @huggingface · France
454
357
Ferdinand Mom · @Huggingface · France
175
Thomas Wolf · @huggingface
59
elie · @PrimeIntellect-ai · France
48
AleHC · École polytechnique fédérale de Lausanne (EPFL)
40
Antoni-Joan Solergibert · ETH AI Center · Switzerland
28
27
20
13
Guillaume LEGENDRE
7
Sam Foreman · @argonne-lcf · Israel
5
Pauline Bailly-Masson
4
4
Loubna Ben Allal · @huggingface · France
4
3
3
@huggingface · Switzerland
2
Luc Georges · @huggingface · France
2
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8322e86149cb5d66, llm:Repository description: 'Minimalistic large language model 3D-parallelism training' (Hugging Face nanotron). Language: Python. Purpose: training LLMs with 3D parallelism.
matched fp:8322e86149cb5d66, llm:Repository description: 'Minimalistic large language model 3D-parallelism training' (Hugging Face nanotron). Language: Python. Purpose: training LLMs with 3D parallelism.
matched fp:8322e86149cb5d66, llm:Repository description: 'Minimalistic large language model 3D-parallelism training' (Hugging Face nanotron). Language: Python. Purpose: training LLMs with 3D parallelism.