Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Multi-Task Deep Neural Networks for Natural Language Understanding
| Date | Stars |
|---|---|
| 2026-07-24 | 2259 |
| 2026-07-25 | 2259 |
| 2026-07-28 | 2259 |
| 2026-07-30 | 2259 |
| 2026-07-31 | 2260 |
| 2026-08-06 | 2259 |
Today
-1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
[](https://opensource.org/licenses/MIT) [](https://github.com/namisan/mt-dnn) **New Release** <br/> We released Adversarial training for both LM pre-training/finetuning and f-divergence. Large-scale Adversarial training for LMs: [ALUM code](https://github.com/namisan/mt-dnn/tree/v0.2/alum). <br/> Hybrid Neural Network Model for Commonsense Reasoning: [HNN code](https://github.com/namisan/mt-dnn/tree/v0.2/hnn) <br/> If you want to use the old version, please use following cmd to clone the code: <br/> ```git clone -b v0.1 https://github.com/namisan/mt-dnn.git ``` **Update** <br/> We regret to inform you that due the policy changing, it no longer provides a public storage for model sharing. We are working hard to find a solution. # Multi-Task Deep Neural Networks for Natural Language Understanding This PyTorch package implements the Multi-Task Deep Neural Networks (MT-DNN) for Natural Language Understanding, as described in: Xiaodong Liu\*, Pengcheng He\*, Weizhu Chen and Jianfeng Gao<br/> Multi-Task Deep Neural Networks for Natural Language Understanding<br/> [ACL 2019](https://aclweb.org/anthology/papers/P/P19/P19-1441/) <br/> \*: Equal contribution <br/> Xiaodong Liu, Pengcheng He, Weizhu Chen and Jianfeng Gao<br/> Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding <br/> [arXiv version](https://arxiv.org/abs/1904.09482) <br/> Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao and Jiawei Han <br/> On the Variance of the Adaptive Learning Rate and Beyond <br/> [ICLR 2020](https://openreview.net/pdf?id=rkgz2aEKDr) <br/> Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao and Tuo Zhao <br/> SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization <br/> [ACL 2020](https://aclanthology.org/2020.acl-main.197/) <br/> Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, Jianfeng Gao<br/> The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding <br/> [ACL 2020](https://aclanthology.org/2020.acl-demos.16/) <br/> Hao Cheng and Xiaodong Liu and Lis Pereira and Yaoliang Yu and Jianfeng Gao<br/> Posterior Differential Regularization with f-divergence for Improving Model Robustness <br/> [NAACL 2021](https://aclanthology.org/2021.naacl-main.85/) <br/> ## Quickstart ### Setup Environment #### Install via pip: 1. python3.6 </br> Reference to download and install : https://www.python.org/downloads/release/python-360/ 2. install requirements </br> ```> pip install -r requirements.txt``` #### Use docker: 1. Pull docker </br> ```> docker pull allenlao/pytorch-mt-dnn:v1.3``` 2. Run docker </br> ```> docker run -it --rm --runtime nvidia allenlao/pytorch-mt-dnn:v1.3 bash``` </br> Please refer to the following link if you first use docker: https://docs.docker.com/ ### Train a toy MT-DNN model 1. Download data </br> ```> sh download.sh``` </br> Please refer to download GLUE dataset: https://gluebenchmark.com/ 2. Preprocess data </br> ```> sh experiments/glue/prepro.sh``` 3. Training </br> ```> python train.py``` **Note that we ran experiments on 4 V100 GPUs for base MT-DNN models. You may need to reduce batch size for other GPUs.** <br/> ### GLUE Result reproduce 1. MTL refinement: refine MT-DNN (shared layers), initialized with the pre-trained BERT model, via MTL using all GLUE tasks excluding WNLI to learn a new shared representation. </br> **Note that we ran this experiment on 8 V100 GPUs (32G) with a batch size of 32.** + Preprocess GLUE data via the aforementioned script + Training: </br> ```>scripts\run_mt_dnn.sh``` 2. Finetuning: finetune MT-DNN to each of the GLUE tasks to get task-s
Excerpt of 13,393 characters
Read on GitHub
418
77
14
9
Liyuan Liu · Thinking Machine Lab
5
3
2
2
2
1
1
1
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:32b2bf855599adde, topic:deep-learning, topic:pytorch, readme:pre-training
matched fp:32b2bf855599adde, topic:nlp, topic:named-entity-recognition