Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code for the ALiBi method for transformer language models (ICLR 2022)
| Date | Stars |
|---|---|
| 2026-07-31 | 559 |
| 2026-08-03 | 559 |
| 2026-08-06 | 559 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Train Short, Test Long: Attention with Linear Biases (ALiBi) Enables Input Length Extrapolation This repository contains the ALiBi code and models for our ICLR 2022 paper Train Short, Test Long. This file explains how to run our experiments on the WikiText-103 dataset. Paper: [here](https://ofir.io/train_short_test_long.pdf) Video: [here](https://www.youtube.com/watch?v=Pp61ShI9VGc) ```diff + ----------- + NEW STUFF: + ----------- ``` 1. There's a [new paper](https://arxiv.org/abs/2310.13017) showing that the position interpolation trick works with ALiBi (October 2023). 2. If you'd like to apply ALiBi to a bidirectional transformer (such as an encoder) model, you could use one of the methods mentioned [here](https://github.com/ofirpress/attention_with_linear_biases/issues/5). 3. There's a FAQ [below](https://github.com/ofirpress/attention_with_linear_biases/#faq) that answers questions regarding comparisons with TransformerXL and why I think ALiBi works well. <p align="center"> <img src=".github/ALiBi.jpeg" width="50%" height="50%"> </p> Attention with Linear Biases (ALiBi) is very simple! Instead of adding position embeddings at the bottom of the transformer stack (which we don't) we add a linear bias to each attention score, as depicted in the figure above. The 'm' hyperparam is head-specific and is not learned- it is set at the beginning of training. We have a function that automatically generates these m values given the number of heads in the model. ALiBi allows the model to be trained on, for example, 1024 tokens, and then do inference on 2048 (or much more) tokens without any finetuning. It's also able to improve performance, even when not extrapolating, in lower resource language modeling settings. The implementation is very simple. 0. Remove the position embeddings from the model: https://github.com/ofirpress/attention_with_linear_biases/blob/master/fairseq/models/transformer.py#L941 1. Set up the relative bias matrix, here: https://github.com/ofirpress/attention_with_linear_biases/blob/master/fairseq/models/transformer.py#L742 2. Add the bias matrix to the mask, which is then added in each attention score computation: https://github.com/ofirpress/attention_with_linear_biases/blob/master/fairseq/models/transformer.py#L1011 3. (This might not be necessary in other frameworks.) Move the mask computation to before the layer loop, to make the transformer a tiny bit faster: https://github.com/ofirpress/attention_with_linear_biases/blob/master/fairseq/models/transformer.py#L949 Thats it! ## FAQ: #### #### Why do you think ALiBi works? I'm not quite sure, but here are my thoughts- I think that using position embeddings (learned, sinusoidal or rotary) is not optimal. I think it leads to the LM 'overfitting' to those position embeddings, and not really understanding the concept of positionality. I have more details about this in [this twitter thread](https://twitter.com/OfirPress/status/1435690039925567489). #### How does ALiBi compare to the positional embedding method from TransformerXL? Good question! Although I don't think it has ever been thoroughly tested, the TransformerXL positioning method might also enable extrapolation. In a previous paper ([Shortformer](https://ofir.io/shortformer.pdf), Table 5) we've shown that that method leads to the attention mechanism being more than two times *slower* than the unmodified attention method. It also requires doubling the amount of memory that the attention sublayer uses. ALiBi runs just as quickly as unmodified attention and uses at most 100MB of extra memory. #### How can I apply ALiBi to bidirectional models like an encoder-only model (like BERT) or an encoder-decoder model (like T5)? See [this](https://github.com/ofirpress/attention_with_linear_biases/issues/5). #### If I want to extrapolate to longer sequences, can't I just apply a sliding window mask to a sinusoidal/learned position embedding model? Nope, that won't work (I've tried). With lear
Excerpt of 10,338 characters
Read on GitHub
143
36
Ning · The Influence Company · United States
35
Ofir Press
23
23
21
21
21
Louis M.
19
Yiming Wang · Johns Hopkins University · United States
17
Sam Shleifer · United States
16
Xu Song · JD.com · China
16
Xutai Ma · @facebookresearch
15
14
13
Joshua Meier
13
13
Elijah Rippeth
10
10
Matt Le · @facebookresearch · United States
10
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:78b5efe609476984, llm:Repository description: 'Code for the ALiBi method for transformer language models (ICLR 2022)'.
matched fp:78b5efe609476984, llm:Repository description: 'Code for the ALiBi method for transformer language models (ICLR 2022)'.
matched fp:78b5efe609476984, llm:Repository description: 'Code for the ALiBi method for transformer language models (ICLR 2022)'.