Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT, ChatGPT, LLMs, and transfer learning.
| Date | Stars |
|---|---|
| 2026-07-24 | 1145 |
| 2026-07-25 | 1145 |
| 2026-07-28 | 1145 |
| 2026-07-30 | 1145 |
| 2026-07-31 | 1145 |
| 2026-08-06 | 1145 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Awesome Transformer & Transfer Learning in NLP [](https://awesome.re)
This repository contains a hand-curated list of great machine (deep) learning resources for Natural Language Processing (NLP) with a focus on Generative Pre-trained Transformer (GPT), Bidirectional Encoder Representations from Transformers (BERT), attention mechanism, Transformer architectures/networks, ChatGPT, and transfer learning in NLP.
<p align="center" width="100%">
<img src="https://user-images.githubusercontent.com/145605/206787465-bdfae6e0-c850-46fc-808d-a51c97644a9e.png#gh-dark-mode-only" width="40%" alt="Transformer (BERT encoder)" />
</p>
<p align="center" width="100%">
<img src="https://user-images.githubusercontent.com/145605/79639176-9ca33d80-81bc-11ea-8cde-f7ff68ee2042.png#gh-light-mode-only" width="40%" alt="Transformer (BERT encoder)" />
</p>
<p align="center" width="100%">
<sup>Transformer (<a href="https://web.archive.org/web/20201217063603/https://peltarion.com/knowledge-center/documentation/modeling-view/build-an-ai-model/blocks/bert-encoder">Source</a>)</sup>
</p>
# Table of Contents
<details>
<summary><b>Expand Table of Contents</b></summary>
- [Papers](#papers)
- [Articles](#articles)
- [BERT and Transformer](#bert-and-transformer)
- [Attention Mechanism](#attention-mechanism)
- [Transformer Architecture](#transformer-architecture)
- [Generative Pre-Training Transformer (GPT)](#generative-pre-training-transformer-gpt)
- [ChatGPT](#chatgpt)
- [Large Language Model (LLM)](#large-language-model-llm)
- [Transformer Reinforcement Learning](#transformer-reinforcement-learning)
- [Additional Reading](#additional-reading)
- [Educational](#educational)
- [Tutorials](#tutorials)
- [AI Safety](#ai-safety)
- [Videos](#videos)
- [BERTology](#bertology)
- [Attention and Transformer Networks](#attention-and-transformer-networks)
- [Official BERT Implementations](#official-bert-implementations)
- [Transformer Implementations By Communities](#transformer-implementations-by-communities)
- [PyTorch and TensorFlow](#pytorch-and-tensorflow)
- [PyTorch](#pytorch)
- [Keras](#keras)
- [TensorFlow](#tensorflow)
- [Chainer](#chainer)
- [Other](#other)
- [Transfer Learning in NLP](#transfer-learning-in-nlp)
- [Books](#books)
- [Other Resources](#other-resources)
- [Tools](#tools)
- [Tasks](#tasks)
- [Named-Entity Recognition (NER)](#named-entity-recognition-ner)
- [Classification](#classification)
- [Text Generation](#text-generation)
- [Question Answering (QA)](#question-answering-qa)
- [Knowledge Graph](#knowledge-graph)
</details>
---
## Papers
1. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
2. [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le and Ruslan Salakhutdinov.
- Uses smart caching to improve the learning of long-term dependency in Transformer. Key results: state-of-art on 5 language modeling benchmarks, including ppl of 21.8 on One Billion Word (LM1B) and 0.99 on enwiki8. The authors claim that the method is more flexible, faster during evaluation (1874 times speedup), generalizes well on small datasets, and is effective at modeling short and long sequences.
2. [Conditional BERT Contextual Augmentation](https://arxiv.org/abs/1812.06705) by Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han and Songlin Hu.
3. [SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering](https://arxiv.org/pdf/1812.03593) by Chenguang Zhu, Michael Zeng and Xuedong Huang.
4. [Language Models are Unsupervised Multitask Learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by Alec Radford, Excerpt of 104,835 characters
Read on GitHub
255
1
Malte Pietsch · deepset · Germany
1
Amit Chaudhary
1
Alex Strick van Linschoten · ZenML · Netherlands
1
1
1
1
Peter Thaleikis · @bringyourownideas
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a2725f02117ea022, topic:transformer, topic:llama, topic:language-model
matched fp:a2725f02117ea022, topic:nlp, topic:natural-language-processing, readme:natural language processing
matched fp:a2725f02117ea022, topic:awesome, topic:awesome-list, desc:curated list