Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". Improve existing models like BERT.
| Date | Stars |
|---|---|
| 2026-07-24 | 252 |
| 2026-07-25 | 252 |
| 2026-07-28 | 252 |
| 2026-07-30 | 252 |
| 2026-08-06 | 252 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# TUPE (Transformer with Untied Positional Encoding)
Implementation for the paper [Rethinking Positional Encoding in Language Pre-training](https://arxiv.org/abs/2006.15595).
<p align="center">
<img src="tupe.png" width="400">
</p>
## Brief Introduction
This repo is to demonstrate TUPE (Transformer with Untied Positional Encoding). The algorithm details could be found in our paper. TUPE can outperform other baselines on GLUE benchmark by a large margin. In particular, it can achieve a higher score than baselines while only using 30% pre-training computational costs.
<p align="center">
<img src="exp.png" width="600">
</p>
Due to limited computational resources, we use the most widely-used pre-training model, BERT-Base, for verification. However, please note that our method could be used for larger (and better) Transformer-based models, like RoBERTa, ELECTRA and UniLM, and further improve them. Besides, since the modification is simple and easy, you can easily apply TUPE in your models.
Our implementation is based on [fairseq](https://github.com/pytorch/fairseq), with several changes:
1. update [`fairseq/modules/transformer_sentence_encoder.py`](fairseq/modules/transformer_sentence_encoder.py) and [`fairseq/modules/multihead_attention.py`](fairseq/modules/multihead_attention.py) for untied positional encoding.
2. some other minor changes to support `max-epoch` with `warmup-ratio` in finetune, instead of setting different `total-num-update` and `warmup-updates` for different tasks.
## Requirements and Installation
More details see [fairseq](https://github.com/pytorch/fairseq). Briefly,
* [PyTorch](http://pytorch.org/)
* Python version >= 3.5
* NVIDIA's [apex](https://github.com/NVIDIA/apex) library with the `--cuda_ext` installation option, for mixed precision training
* You may need [NCCL](https://github.com/NVIDIA/nccl) for multi-node distributed training
**Installing from source**
To install TUPE from source and develop locally:
```bash
git clone https://github.com/guolinke/TUPE
cd TUPE
pip install --editable .
```
## Getting Started
### Data Pre-Processing
The pre-processing relies on [mosesdecoder](https://github.com/moses-smt/mosesdecoder), you can run the following script to pull it.
```bash
cd TUPE
git submodule update --init
```
#### Pretraining Data
Refer to the steps in [`preprocess/pretrain/process.sh`](preprocess/pretrain/process.sh).
#### Downstream Data
Refer to the steps in [`preprocess/glue/process.sh`](preprocess/glue/process.sh).
### Pre-Training
```bash
DATA_DIR=./path_to_your_data/
SAVE_DIR=./your_own_save_path/
TOTAL_UPDATES=1000000
WARMUP_UPDATES=10000
PEAK_LR=0.0001
MAX_POSITIONS=512
MAX_SENTENCES=16
UPDATE_FREQ=1
SEED=your_seed
python train.py $DATA_DIR --fp16 --num-workers 16 --ddp-backend=c10d \
--task masked_lm --criterion masked_lm --arch bert_base \
--sample-break-mode complete --tokens-per-sample $MAX_POSITIONS \
--optimizer adam --adam-betas '(0.9, 0.999)' --adam-eps 1e-6 --clip-norm 1.0 \
--lr-scheduler polynomial_decay --lr $PEAK_LR --warmup-updates $WARMUP_UPDATES --total-num-update $TOTAL_UPDATES \
--dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
--max-sentences $MAX_SENTENCES --update-freq $UPDATE_FREQ --seed $SEED \
--mask-prob 0.15 \
--embedding-normalize \
--max-update $TOTAL_UPDATES --log-format simple --log-interval 100 \
--keep-updates-list 100000 300000 600000 1000000 \
--save-interval-updates 25000 --keep-interval-updates 3 --no-epoch-checkpoints --skip-invalid-size-inputs-valid-test \
--save-dir $SAVE_DIR --rel-pos
```
The above setting is for 16 V100 GPUs, and the batch size is 256 (`n_gpu * MAX_SENTENCES * UPDATE_FREQ`). You may need to change `MAX_SENTENCES` or `UPDATE_FREQ` according to your environment. To disable relative position, you can remove `--rel-pos` .
### Fine-Tuning
```bash
DATA_DIR=./path_to_your_downstream_data
SAVE_DIR=./path_to_your_save_dir
BERT_MODEL_PATH=./path_to_your_checkpExcerpt of 6,548 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:b3cce7f7cb8b947a, topic:transformer, topic:language-model
matched fp:b3cce7f7cb8b947a, topic:pretraining, readme:distributed training, readme:pretraining