Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
This is a repository with the code for the ACL 2019 paper "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned" and the ACL 2021 paper "Analyzing Source and Target Contributions to NMT Predictions".
| Date | Stars |
|---|---|
| 2026-07-24 | 324 |
| 2026-07-25 | 324 |
| 2026-07-28 | 324 |
| 2026-07-30 | 324 |
| 2026-08-06 | 324 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
## The Story of Heads
This is the official repo for the following papers:
* (ACL 2019) [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://www.aclweb.org/anthology/P19-1580)
* (ACL 2021) [Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation](https://arxiv.org/pdf/2010.10907.pdf)
<img src="./resources/acl_empty.png" title="conf logo"/>
<img src="./resources/acl19_heads-min_pad.png"
title="paper logo" width="600"/>
In this README, we discuss the ACL 2019 heads paper. Read the official [blog post](https://lena-voita.github.io/posts/acl19_heads.html) for the details!
For the contributions paper, go to the [source_target_contributions](./source_target_contributions/) folder.
#### Bibtex
```
@inproceedings{voita-etal-2019-analyzing,
title = "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned",
author = "Voita, Elena and
Talbot, David and
Moiseev, Fedor and
Sennrich, Rico and
Titov, Ivan",
booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
month = jul,
year = "2019",
address = "Florence, Italy",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/P19-1580",
pages = "5797--5808",
}
```
Table of Contents
=================
* [Introduction](#introduction)
* [Pruning Attention Heads](#pruning-attention-heads)
* [Experiments](#experiments)
* [Requirements](#requirements)
* [Data preprocessing](#data-preprocessing)
* [Tokenization](#tokenization)
* [BPE-ization](#bpe-ization)
* [Model training](#model-training)
* [Notebooks: how to use a model](#notebooks-how-to-use-a-model)
* [Training config tour](#training-config-tour)
* [Data](#data)
* [Model](#model)
* [Problem (loss function)](#problem-loss-function)
* [Starting checkpoint](#starting-checkpoint)
* [Variables to optimize](#variables-to-optimize)
* [Batch size](#batch-size)
* [Other options](#other-options)
* [Comments](#comments)
# Introduction
In the paper, we:
* evaluate the importance of attention heads in Transformer,
* identify functions of the most important encoder heads,
* prune the vast majority of attention heads in Transformer without seriously affecting quality using a method based on stochastic gates and a differentiable relaxation of the L0 penalty,
* show which types of model attention are most sensitive to the number of attention heads and on which layers.
In this repo, we provide code and describe steps needed to reproduce our experiments with the L0 head pruning.
## Pruning Attention Heads
In the standard Transformer, results of different attention heads in a layer are concatenated:
```MultiHead(Q, K, V ) = Concat(head_i)W^O.```
We modify the original Transformer architecture by multiplying the representation computed by each `head_i` by a scalar gate `g_i`:
```MultiHead(Q, K, V ) = Concat(g_i * head_i)W^O.```
Unlike usual gates, `g_i` are parameters specific to heads and are independent of the input (i.e. the sentence). As we would like to disable less important heads completely, we would ideally apply `L0` regularization to the scalars `g_i`. The L0 norm equals the number of non-zero components and would push the model to switch off less important heads.
Unfortunately, the L0 norm is nondifferentiable and so cannot be directly incorporated as a regularization term in
the objective function. Instead, we use a stochastic relaxation.
Each gate `g_i` is a random variable drawn independently from a head-specific [Hard Concrete distribution](https://openreview.net/pdf?id=H1Y8hhg0b). The distributions have non-zero probability mass at 0 and 1; look at the illustration.
We use Excerpt of 15,591 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:1feb950474654590, topic:transformer, readme:transformer architecture