Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Original transformer paper: Implementation of Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems. 2017.
| Date | Stars |
|---|---|
| 2026-07-24 | 250 |
| 2026-07-25 | 251 |
| 2026-07-28 | 251 |
| 2026-07-30 | 251 |
| 2026-08-06 | 251 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Attention Is All You Need Paper Implementation
This is my from-scratch implementation of the original transformer architecture from the following paper: [Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems. 2017.](https://arxiv.org/abs/1706.03762)
<a href=https://arxiv.org/pdf/1706.03762.pdf>
<p align="center">
<img width="540" height="700" src="https://user-images.githubusercontent.com/57716666/143226016-e184793b-a0c8-443c-b2c0-8f034eb82d93.jpg">
</p>
</a>
## Table of Contents
* [About](#about)
* [Setup](#setup)
* [Environment](#environment)
* [Pretrained Models](#pretrained-models)
* [Usage](#usage)
* [Training](#training)
* [Inference](#inference)
* [Data](#data)
* [Architecture](#architecture)
* [Weights and Biases Logs](#weights-and-biases-logs)
* [Citation](#citation)
* [License](#license)
# About
*"We propose a new simple network architecture, the Transformer,
based solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. " - Abstract*
Transformers came to be a groundbreaking advance in neural network architectures which revolutionized what we can do with NLP and beyond. To name a few applications consider the [application of BERT to Google search](https://blog.google/products/search/search-language-understanding-bert/) and GPT to [Github Copilot](https://copilot.github.com/). Those architectures are upgrades on the original transformer architecture described in this seminal paper. **The goal of this repository is to provide an implementation that is easy to follow and understand while reading the paper. Setup is easy and everything is runnable on CPU for learning purposes.**
✔️ Highly customizable configuration and training loop
✔️ Runnable on CPU and GPU
✔️ W&B integration for detailed logging of every metric
✔️ Pretrained models and their training details
✔️ Gradient Accumulation
✔️ Label smoothing
✔️ BPE and WordLevel Tokenizers
✔️ Dynamic Batching
✔️ Batch Dataset Processing
✔️ Bleu-score calculation during training
✔️ Documented dimensions for every step of the architecture
✔️ Shown progress of translation for an example after every epoch
✔️ Tutorial notebook
# Setup
## Environment
Using Miniconda/Anaconda:
1. `cd path_to_repo`
2. `conda env create`
3. `conda activate attention-is-all-you-need-paper`
***Note**: Depending on your GPU you might need to switch cudatoolkit to version 10.2*
## Pretrained Models
To download the pretrained model and tokenizer run:
```
python scripts/download_pretrained.py
```
***Note**: If prompted about wandb setting select option 3*
You can see all the information and results for pretrained models at this [project link](https://wandb.ai/bkoch4142/attention-is-all-you-need-paper/runs/1rbhz2as?workspace=user-bkoch4142).
# Usage
## Training
Before starting training you can either **choose a configuration out of available ones or create your own** inside a single file `src/config.py`. The available parameters to customize, sorted by categories, are:
- **Run** 🚅:
- RUN_NAME - Name of a training run
- RUN_DESCRIPTION - Description of a training run
- RUNS_FOLDER_PTH - Saving destination of a training run
- **Data** 🔡:
- DATASET_SIZE - Number of examples you want to include from WMT14 en-de dataset (max 4,500,000)
- TEST_PROPORTION - Test set proportion
- MAX_SEQ_LEN - Maximum allowed sequence length
- VOCAB_SIZE - Size of the vocabulary (good choice is dependant on the tokenizer)
- TOKENIZER_TYPE - 'wordlevel' or 'bpe'
- **Training** 🏋️♂️:
- BATCH_SIZE - Batch size
- GRAD_ACCUMULATION_STEPS - Over how many batches to accumulate gradients before optimizing the parameters
- WORKER_COUNT - Number of workers used in dataloaders
- EPOCHS - Number of epochs
- **Optimizer** 📉:
- BETAS - Adam beta parameter
- EPS - Adam eps parameter
- **Scheduler** ⏲️:
- N_WARMUP_STEPS - How many warmup steps to use in the scheduExcerpt of 8,209 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:298b906496de6433, topic:deep-learning, topic:pytorch, readme:from scratch implementation
matched fp:298b906496de6433, topic:transformer, readme:pretrained model, readme:transformer architecture