Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A collection of phenomenons observed during the scaling of big foundation models, which may be developed into consensus, principles, or laws in the future
| Date | Stars |
|---|---|
| 2026-07-31 | 284 |
| 2026-08-05 | 284 |
| 2026-08-20 | 283 |
| 2026-09-13 | 284 |
| 2026-09-20 | 284 |
Today
— stars today
This week
— stars this week
This month
+1 stars this month
Momentum
0.0
growth rate 0.00%/day
# BM-Principles
🌟 The big models have proven their potential to lead to artificial general intelligence. However 😕, due to their rapid development, people have not fully grasped the principles of understanding and training big models. Therefore, in order to learn about big models together, we have decided to collect **new phenomena observed on the big models** and summarize them in this repository 📚 in the form of short entries. We hope this collection of phenomena observed during the scaling of big models may form future consensuses, principles, or patterns 📝.
The repository focuses on two aspects:
- **How**: How to train powerful big models? 🚀
- **What**: What properties are interesting for big models? 🤔
The repo is far from exclusive currently. Let's work together to improve it! 💪
## How: how to train a powerful big model.
1. **Scaling of Computation**
1. **Training loss decreases predictably.**
- Training loss can be written as a smooth function of model parameters and computation.
<img src="https://github.com/OpenBMB/BMPrinciples/blob/main/figs/H1.1.loss_descrease.png" alt="" width="400" height="250">
> [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361)
> [Scaling Laws for Autoregressive Generative Modeling](https://arxiv.org/abs/2010.14701)
2. **Computational-optimal language model.**
- Given a fixed computational budget, if we train an excessively large model, we can only iterate for a very limited number of steps. On the other hand, if we train a model that is too small, the limit of the loss will not be as good as that of a larger model. Therefore, there exists an *optimal model size*, *optimal training compute*, and *optimal tokens*.
- From previous experience, it's roughly $20 * N$, where $N$ is the number of model parameters.
<img src="https://github.com/OpenBMB/BMPrinciples/blob/main/figs/H1.2.computational_optimal.png" alt="" width="350" height="250">
> [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556)
3. **LLM doesn't converge at tokens of optimal computation.**
- LLM might continue to improve the loss after optimal tokens.
- From Llama-7b and Llama-13b's training loss, we can see that continue to improve after 140 B and 260 B parameters.
<img src="https://github.com/OpenBMB/BMPrinciples/blob/main/figs/H1.3.llm_doesnt_converge.png" alt="" width="350" height="250">
> [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
`
2. **Optimal Hyperparameters.**
1. **The best batch size is a function of loss.**
- To reach a certain loss, a large batch size requires more computation, a small batch size requires more training steps (i.e., times). The best batch size is a trade-off.
- Each diagonal line formed by the points represents a training process. The horizontal axis represents the training steps, the vertical axis represents the number of processed tokens, and the color depth represents the loss. The optimal batch size can be considered as the inflection point of each contour line of loss.
<img src="https://github.com/OpenBMB/BMPrinciples/blob/main/figs/H2.1.best_batch_size.png" alt="" width="660" height="200">
> [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361)
2. **Large batch size allows a large learning rate,**
1. Generally, a larger batch size allows a larger learning rate. And the larger learning rate has faster convergence.
> [Don't decay the learning rate, increase the batch size](https://arxiv.org/abs/1711.00489)
3. **Cosine scheduler is prevalent.**
- Cosine scheduler is the prevalent one, which is better than Noam with the same peak learning rate. Noam decreases more sharply.
- Below is our experiment for CPM.
<img src="https://github.com/OpenBMB/BMPrinciples/blob/main/figs/H2.3.Excerpt of 9,939 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:fd7046cd465d8892, llm:Repository description: "A collection of phenomenons observed during the scaling of big foundation models, which may be developed into consensus, principles, or laws in the future"
matched fp:fd7046cd465d8892, llm:Repository description: "A collection of phenomenons observed during the scaling of big foundation models, which may be developed into consensus, principles, or laws in the future"