Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Understanding the Difficulty of Training Transformers
| Date | Stars |
|---|---|
| 2026-07-24 | 332 |
| 2026-07-25 | 332 |
| 2026-07-28 | 332 |
| 2026-07-30 | 332 |
| 2026-08-06 | 332 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
#### News: we re-implmented Admin as a plug-in-and-play toolkit!
#### [Read more about `admin-torch`](https://github.com/microsoft/admin-torch)
----------------------------------
[](https://opensource.org/licenses/Apache-2.0) [](https://paperswithcode.com/sota/machine-translation-on-wmt2014-english-french?p=very-deep-transformers-for-neural-machine)
<h2 align="center">Admin</h2>
<h5 align="center">Understanding the Difficulty of Training Transformers</h5>
Guided by our analyses, we propose **Ad**aptive **M**odel **In**itialization (Admin), which successfully stabilizes previously-diverged Transformer training and achieves better performance, **without introducing additional hyper-parameters**. The design of Admin is half-precision friendly and can be **reparameterized into the original Transformer**.
In our experiments, Admin [easily stabilized the training of 200L Transformer](https://github.com/LiyuanLucasLiu/Transformer-Clinic/blob/master/nmt-experiments/wmt14_en-de.md).
We are in an early-release beta. Expect some adventures and rough edges.
## Table of Contents
- [Introduction](#introduction)
- [Amplification Effect](#dependency-and-amplification-effect)
- [Quick Start](#quick-start-guide)
- [Citation](#citation)
## Introduction
<h5 align="center"><i>What complicates Transformer training?</i></h5>
In our study, we go beyond gradient vanishing and identify an __amplification effect__ that substantially influences Transformer training.
Specifically, for each layer in a multi-layer Transformer, heavy dependency on its residual branch makes training unstable, yet light dependency leads to sub-optimal performance.
## Dependency and Amplification Effect
Our analysis starts from the observation that Pre-LN is more robust than Post-LN, whereas Post-LN typically leads to a better performance.
As shown in Figure 1, we find these two variants have different layer dependency patterns.
<p align="center"><img width="60%" src="img/6_layer_dependency.png"/></p>
With further exploration, we find that for a N-layer residual network, after updating its parameters W to W\*, its outputs change is proportion to the dependency on residual branches.
<p align="center"><img width="60%" src="img/output_change.png"/></p>
Intuitively, since a larger output change indicates a more unsmooth loss surface, the large dependency complicates training.
Moreover, we propose Admin (**ad**aptive **m**odel **in**itialization), which starts the training from the area with a smoother surface.
More details can be found in our [paper](https://arxiv.org/abs/2004.08249).
## Quick Start Guide
Our implementation is based on the fairseq package (`python 3.6, torch 1.5/1.6` are recommended). It can be installed by:
```
git clone https://github.com/LiyuanLucasLiu/Transforemr-Clinic.git
cd fairseq
pip install --editable .
```
The guidance for reproducing our results is available at:
- [WMT'14 De-En](nmt-experiments/wmt14_en-de.md)
- [WMT'14 De-Fr](nmt-experiments/wmt14_en-fr.md)
- [IWSLT'14 En-De](nmt-experiments/iwslt14_de-en.md)
Specifically, our implementation requires to first set ```--init-type adaptive-profiling``` and use one GPU for this profiling stage, then set ```--init-type adaptive``` and start training.
## Citation
Please cite the following papers if you found our model useful. Thanks!
>Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han (2020). Understanding the Difficulty of Training Transformers. Proc. 2020 Conf. on Empirical Methods in Natural Language Processing (EMNLP'20).
```
@inproceedings{liu2020admin,
title={Understanding the Difficulty of Training Transformers},
author = {Liu, Liyuan and Liu, Xiaodong and Gao, Jianfeng and Chen, Weizhu and Han, Jiawei},
booktitleExcerpt of 4,520 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:d83f8a7fee7f1219, topic:transformer