Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Repository of code for the tutorial on Transfer Learning in NLP held at NAACL 2019 in Minneapolis, MN, USA
| Date | Stars |
|---|---|
| 2026-07-24 | 723 |
| 2026-07-25 | 723 |
| 2026-07-28 | 723 |
| 2026-07-30 | 723 |
| 2026-07-31 | 724 |
| 2026-08-06 | 724 |
Today
— stars today
This week
+1 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.14%/day
# Code repository accompanying NAACL 2019 tutorial on "Transfer Learning in Natural Language Processing" The tutorial was given on June 2 at NAACL 2019 in Minneapolis, MN, USA by [Sebastian Ruder](http://ruder.io/), [Matthew Peters](https://www.linkedin.com/in/petersmatthew), [Swabha Swayamdipta](http://www.cs.cmu.edu/~sswayamd/index.html) and [Thomas Wolf](http://thomwolf.io/). Here is the [webpage](https://naacl2019.org/program/tutorials/) of NAACL tutorials for more information. The slides for the tutorial can be found here: https://tinyurl.com/NAACLTransfer. A Google Colab notebook with all the code for the tutorial can be found here: https://tinyurl.com/NAACLTransferColab. The present repository can also be accessed with the following short url: https://tinyurl.com/NAACLTransferCode ## Abstract The classic supervised machine learning paradigm is based on learning in isolation a single predictive model for a task using a single dataset. This approach requires a large number of training examples and performs best for well-defined and narrow tasks. Transfer learning refers to a set of methods that extend this approach by leveraging data from additional domains or tasks to train a model with better generalization properties. Over the last two years, the field of Natural Language Processing (NLP) has witnessed the emergence of several transfer learning methods and architectures, which significantly improved upon the state-of-the-art on a wide range of NLP tasks. These improvements together with the wide availability and ease of integration of these methods are reminiscent of the factors that led to the success of pretrained word embeddings and ImageNet pretraining in computer vision, and indicate that these methods will likely become a common tool in the NLP landscape as well as an important research direction. We will present an overview of modern transfer learning methods in NLP, how models are pre-trained, what information the representations they learn capture, and review examples and case studies on how these models can be integrated and adapted in downstream NLP tasks. ## Overview This codebase tries to present in the simplest and most compact way a few of the major Transfer Learning techniques, which have emerged over the past years. The code in this repository does not attempt to be state-of-the-art. However, effort has been made to achieve reasonable performance and with some modifications to be competitive with the current state of the art. Special effort has been made to - ensure the present code can be used as easily as possible, in particular by hosting pretrained models and datasets; - keep the present codebase as compact and self-contained as possible to make it easy to manipulate and understand. Currently the codebase comprises: - [`pretraining_model.py`](./pretraining_model.py): a transformer model with a GPT-2-like architecture as the basic pretrained model; - [`pretraining_train.py`](./pretraining_train.py): a pretraining script to train this model with a language modeling objective on a selection of large datasets (WikiText-103, SimpleBooks-92) using distributed training if available; - [`finetuning_model.py`](./finetuning_model.py): several architectures based on the transformer model for fine-tuning (with a classification head on top, with adapters); - [`finetuning_train.py`](./finetuning_train.py): a fine-tuning script to fine-tune these architectures on a classification task (IMDb). ## Installation To use this codebase, simply clone the Github repository and install the requirements like this: ```bash git clone https://github.com/huggingface/naacl_transfer_learning_tutorial cd naacl_transfer_learning_tutorial pip install -r requirements.txt ``` ## Pre-training To pre-train the transformer, run the `pretraining_train.py` script like this: ```bash python ./pretraining_train.py ``` or using distributed training like this (for a 8 GPU server): ```bash python -m torch.distributed.laun
Excerpt of 5,571 characters
Read on GitHub
Thomas Wolf · @huggingface
68
Bharat Raghunathan · United States
1
Sebastian Ruder · @facebook · Germany
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:bf23fd339994d1af, topic:tutorial, name:tutorial, desc:tutorial
matched fp:bf23fd339994d1af, topic:nlp, readme:natural language processing