Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
The pure and clear PyTorch Distributed Training Framework.
| Date | Stars |
|---|---|
| 2026-07-24 | 276 |
| 2026-07-25 | 276 |
| 2026-07-28 | 276 |
| 2026-07-30 | 276 |
| 2026-08-06 | 276 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center">
<img src="./images/logo.png" width="300px">
**The pure and clear PyTorch Distributed Training Framework.**
</div>
* [Introduction](#introduction)
* [Requirements and Usage](#requirements-and-usage)
* [Dependency](#dependency)
* [Dataset](#dataset)
* [Basic Usage](#basic-usage)
* [Slurm Cluster Usage](#slurm-cluster-usage)
* [Baselines](#baselines)
* [Zombie processes problem](#zombie-processes-problem)
* [Acknowledgments](#acknowledgments)
* [Citation](#citation)
## Introduction
Distribuuuu is a Distributed Classification Training Framework powered by native PyTorch.
Please check [tutorial](./tutorial/) for detailed **Distributed Training** tutorials:
- Single Node Single GPU Card Training [[snsc.py](./tutorial/snsc.py)]
- Single Node Multi-GPU Cards Training (with DataParallel) [[snmc_dp.py](./tutorial/snmc_dp.py)]
- Multiple Nodes Multi-GPU Cards Training (with DistributedDataParallel)
- torch.distributed.launch [[mnmc_ddp_launch.py](./tutorial/mnmc_ddp_launch.py)]
- torch.multiprocessing [[mnmc_ddp_mp.py](./tutorial/mnmc_ddp_mp.py)]
- Slurm Workload Manager [[mnmc_ddp_slurm.py](./tutorial/mnmc_ddp_slurm.py)]
- ImageNet training example [[imagenet.py](./tutorial/imagenet.py)]
For the complete training framework, please see [distribuuuu](./distribuuuu/).
## Requirements and Usage
### Dependency
- Install **PyTorch>= 1.6** (has been tested on **1.6, 1.7.1, 1.8** and **1.8.1**)
- Install other dependencies: ``pip install -r requirements.txt``
### Dataset
Download the ImageNet dataset and move validation images to labeled subfolders, using the script [valprep.sh](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh).
<details>
<summary>Expected datasets structure for ILSVRC</summary>
```
ILSVRC
|_ train
| |_ n01440764
| |_ ...
| |_ n15075141
|_ val
| |_ n01440764
| |_ ...
| |_ n15075141
|_ ...
```
Create a directory containing symlinks:
``` bash
mkdir -p /path/to/distribuuuu/data
```
Symlink ILSVRC:
``` bash
ln -s /path/to/ILSVRC /path/to/distribuuuu/data/ILSVRC
```
</details>
### Basic Usage
Single Node with one task
``` bash
# 1 node, 8 GPUs
python -m torch.distributed.launch \
--nproc_per_node=8 \
--nnodes=1 \
--node_rank=0 \
--master_addr=localhost \
--master_port=29500 \
train_net.py --cfg config/resnet18.yaml
```
Distribuuuu use [yacs](https://github.com/rbgirshick/yacs), a elegant and lightweight package to define and manage system configurations.
You can setup config via a yaml file, and overwrite by other opts. If the yaml is not provided, the default configuration file will be used, please check [distribuuuu/config.py](./distribuuuu/config.py).
```bash
python -m torch.distributed.launch \
--nproc_per_node=8 \
--nnodes=1 \
--node_rank=0 \
--master_addr=localhost \
--master_port=29500 \
train_net.py --cfg config/resnet18.yaml \
OUT_DIR /tmp \
MODEL.SYNCBN True \
TRAIN.BATCH_SIZE 256
# --cfg config/resnet18.yaml parse config from file
# OUT_DIR /tmp overwrite OUT_DIR
# MODEL.SYNCBN True overwrite MODEL.SYNCBN
# TRAIN.BATCH_SIZE 256 overwrite TRAIN.BATCH_SIZE
```
<details>
<summary>Single Node with two tasks</summary>
```bash
# 1 node, 2 task, 4 GPUs per task (8GPUs)
# task 1:
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch \
--nproc_per_node=4 \
--nnodes=2 \
--node_rank=0 \
--master_addr=localhost \
--master_port=29500 \
train_net.py --cfg config/resnet18.yaml
# task 2:
CUDA_VISIBLE_DEVICES=4,5,6,7 python -m torch.distributed.launch \
--nproc_per_node=4 \
--nnodes=2 \
--node_rank=1 \
--master_addr=localhost \
--master_port=29500 \
train_net.py --cfg config/resnet18.yaml
```
</details>
<details>
<summary>Multiple Nodes Training</summary>
```bash
# 2 node, 8 GPUs per node (16GPUs)
# node 1:
python -m torch.distributed.launch \
--nproc_per_node=8 \
--nnoExcerpt of 9,713 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:47e3ea5c7507c3b3, topic:pytorch, topic:training, desc:training framework