Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.
| Date | Stars |
|---|---|
| 2026-07-24 | 1109 |
| 2026-07-25 | 1109 |
| 2026-07-28 | 1109 |
| 2026-07-30 | 1109 |
| 2026-08-06 | 1109 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
## The Original Transformer (PyTorch) :computer: = :rainbow: This repo contains PyTorch implementation of the original transformer paper (:link: [Vaswani et al.](https://arxiv.org/abs/1706.03762)). <br/> It's aimed at making it **easy to start playing and learning** about transformers. <br/> ## Table of Contents * [What are transformers?](#what-are-transformers) * [Understanding transformers](#understanding-transformers) * [Machine translation](#machine-translation) * [Setup](#setup) * [Usage](#usage) * [Hardware requirements](#hardware-requirements) ## What are transformers Transformers were originally proposed by Vaswani et al. in a seminal paper called [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf). You probably heard of transformers one way or another. **GPT-3 and BERT** to name a few well known ones :unicorn:. The main idea is that they showed that you don't have to use recurrent or convolutional layers and that simple architecture coupled with attention is super powerful. It gave the benefit of **much better long-range dependency modeling** and the architecture itself is highly **parallelizable** (:computer::computer::computer:) which leads to better compute efficiency! Here is how their beautifully simple architecture looks like: <p align="center"> <img src="data/readme_pics/transformer_architecture.PNG" width="350"/> </p> ## Understanding transformers This repo is supposed to be a learning resource for understanding transformers as the original transformer by itself is not a SOTA anymore. For that purpose the code is (hopefully) well commented and I've included the `playground.py` where I've visualized a couple of concepts which are hard to explain using words but super simple once visualized. So here we go! ### Positional Encodings Can you parse this one in a glimpse of the eye? <p align="left"> <img src="data/readme_pics/positional_encoding_formula.PNG"/> </p> Neither can I. Running the `visualize_positional_encodings()` function from `playground.py` we get this: <p align="center"> <img src="data/readme_pics/positional_encoding_visualized.jpg"/> </p> Depending on the position of your source/target token you "pick one row of this image" and you add it to it's embedding vector, that's it. They could also be learned, but it's just more fancy to do it like this, obviously! :nerd_face: ### Custom Learning Rate Schedule Similarly can you parse this one in `O(1)`? <p align="left"> <img src="data/readme_pics/lr_formula.PNG"/> </p> Noup? So I thought, here it is visualized: <p align="center"> <img src="data/readme_pics/custom_learning_rate_schedule.PNG"/> </p> It's super easy to understand now. Now whether this part was crucial for the success of transformer? I doubt it. But it's cool and makes things more complicated. :nerd_face: (`.set_sarcasm(True)`) *Note: model dimension is basically the size of the embedding vector, baseline transformer used 512, the big one 1024* ### Label Smoothing First time you hear of label smoothing it sounds tough but it's not. You usually set your target vocabulary distribution to a `one-hot`. Meaning 1 position out of 30k (or whatever your vocab size is) is set to 1. probability and everything else to 0. <p align="center"> <img src="data/readme_pics/label_smoothing.PNG" width="700"/> </p> In label smoothing instead of placing 1. on that particular position you place say 0.9 and you evenly distribute the rest of the "probability mass" over the other positions (that's visualized as a different shade of purple on the image above in a fictional vocab of size 4 - hence 4 columns) *Note: Pad token's distribution is set to all zeros as we don't want our model to predict those!* Aside from this repo (well duh) I would highly recommend you go ahead and read [this amazing blog](https://jalammar.github.io/illustrated-transformer/) by Jay Alammar! ## Machine translation Transformer was originally trained for the NMT (neural machine translation)
Excerpt of 16,987 characters
Read on GitHub
Aleksa Gordić · ex-DeepMind, ex-Microsoft · United States
89
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:59366abd33346a88, topic:deep-learning, topic:pytorch
matched fp:59366abd33346a88, topic:transformer, readme:transformer architecture