Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Efficient Training (including pre-training and fine-tuning) for Big Models
| Date | Stars |
|---|---|
| 2026-07-31 | 624 |
| 2026-08-01 | 624 |
| 2026-08-06 | 624 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center">
<h1><img src="docs/logo.png" height="28px" /> BMTrain</h1>
**Efficient Training for Big Models**
<p align="center">
<a href="#overview">Overview</a> • <a href="#documentation">Documentation</a> • <a href="#install">Installation</a> • <a href="#usage">Usage</a> • <a href="#performance">Performance</a> • <a href="./README-ZH.md" target="_blank">简体中文</a>
<br>
</p>
<p align="center">
<a href='https://bmtrain.readthedocs.io/en/latest/?badge=latest'>
<img src='https://readthedocs.org/projects/bmtrain/badge/?version=latest' alt='Documentation Status' />
</a>
<a href="https://github.com/OpenBMB/BMTrain/releases">
<img alt="GitHub release (latest by date including pre-releases)" src="https://img.shields.io/github/v/release/OpenBMB/BMTrain?include_prereleases">
</a>
<a href="https://github.com/OpenBMB/BMTrain/blob/main/LICENSE">
<img alt="GitHub" src="https://img.shields.io/github/license/OpenBMB/BMTrain">
</a>
</p>
</div>
## What's New
- 2024/02/26 **BMTrain** [1.0.0](https://github.com/OpenBMB/BMTrain/releases/tag/v1.0.0) released. Code refactoring and Tensor parallel support. See the detail in [update log](docs/UPDATE_1.0.0.md)
- 2023/08/17 **BMTrain** [0.2.3](https://github.com/OpenBMB/BMTrain/releases/tag/v0.2.3) released. See the [update log](docs/UPDATE_0.2.3.md).
- 2022/12/15 **BMTrain** [0.2.0](https://github.com/OpenBMB/BMTrain/releases/tag/0.2.0) released. See the [update log](docs/UPDATE_0.2.0.md).
- 2022/06/14 **BMTrain** [0.1.7](https://github.com/OpenBMB/BMTrain/releases/tag/0.1.7) released. ZeRO-2 optimization is supported!
- 2022/03/30 **BMTrain** [0.1.2](https://github.com/OpenBMB/BMTrain/releases/tag/0.1.2) released. Adapted to [OpenPrompt](https://github.com/thunlp/OpenPrompt)and [OpenDelta](https://github.com/thunlp/OpenDelta).
- 2022/03/16 **BMTrain** [0.1.1](https://github.com/OpenBMB/BMTrain/releases/tag/0.1.1) has publicly released the first stable version, which fixes many bugs that were in the beta version.
- 2022/02/11 **BMTrain** [0.0.15](https://github.com/OpenBMB/BMTrain/releases/tag/0.0.15) has publicly released the first beta version.
<div id="overview"></div>
## Overview
BMTrain is an efficient large model training toolkit that can be used to train large models with tens of billions of parameters. It can train models in a distributed manner while keeping the code as simple as stand-alone training.
<div id="documentation"></div>
## Documentation
Our [documentation](https://bmtrain.readthedocs.io/en/latest/index.html) provides more information about the package.
<div id="install"></div>
## Installation
- From pip （recommend） : ``pip install bmtrain``
- From source code: download the package and run ``pip install .``
Installing BMTrain may take a few to ten minutes, as it requires compiling the c/cuda source code at the time of installation.
We recommend compiling BMTrain directly in the training environment to avoid potential problems caused by the different environments.
<div id="usage"></div>
## Usage
### Step 1: Initialize BMTrain
Before you can use BMTrain, you need to initialize it at the beginning of your code. Just like using the distributed module of PyTorch requires the use of **init_process_group** at the beginning of the code, using BMTrain requires the use of **init_distributed** at the beginning of the code.
```python
import bmtrain as bmt
bmt.init_distributed(
seed=0,
# ...
)
```
**NOTE:** Do not use PyTorch's distributed module and its associated communication functions when using BMTrain.
### Step 2: Enable ZeRO Optimization
To enable ZeRO optimization, you need to make some simple replacements to the original model's code.
* `torch.nn.Module` -> `bmtrain.DistributedModule`
* `torch.nn.Parameter` -> `bmtrain.DistributedParameter`
And wrap the transformer blocks with `bmtrain.Block`.
Here is an example.
**Original**
```python
import torch
class MyModule(torch.nn.Module):
def __init__(self):
super().__iniExcerpt of 13,696 characters
Read on GitHub
ZGY · OpenBMB · China
191
MayDomine
111
William · Tsinghua University
92
15
13
8
4
DingDing · @tsinghua
3
xiaocj · Tsinghua University · China
1
1
Ethan (Yusheng) Su · RadixArk | AMD | Tsinghua University · United States
1
Dennis Huang · EnjoyCloud Software
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:06d7ad6a44a50e7f, desc:fine-tuning, desc:fine tuning