Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.
| Date | Stars |
|---|---|
| 2026-07-24 | 1215 |
| 2026-07-25 | 1214 |
| 2026-07-28 | 1214 |
| 2026-07-30 | 1214 |
| 2026-07-31 | 1215 |
| 2026-08-06 | 1215 |
Today
— stars today
This week
+1 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.08%/day
<div align="center">
<img src="feat_img.png"/>
</div>
# Self-attention building blocks for computer vision applications in PyTorch
Implementation of self attention mechanisms for computer vision in PyTorch with einsum and einops.
Focused on computer vision self-attention modules.
#### Install it via pip
```$ pip install self-attention-cv```
It would be nice to pre-install pytorch in your environment, in case you don't have a GPU. To run the tests from the terminal
```$ pytest``` you may need to run ``` export PYTHONPATH=$PATHONPATH:`pwd` ``` before.
## Related articles
- [How Attention works in Deep Learning](https://theaisummer.com/attention/)
- [How Transformers work in deep learning and NLP](https://theaisummer.com/transformer/)
- [How the Vision Transformer (ViT) works in 10 minutes: an image is worth 16x16 words](https://theaisummer.com/vision-transformer/)
- [Understanding einsum for Deep learning: implement a transformer with multi-head self-attention from scratch](https://theaisummer.com/einsum-attention/)
- [How Positional Embeddings work in Self-Attention](https://theaisummer.com/positional-embeddings/)
- [Why multi-head self attention works: math, intuitions and 10+1 hidden insights](https://theaisummer.com/self-attention/)
## Code Examples
#### Multi-head attention
```python
import torch
from self_attention_cv import MultiHeadSelfAttention
model = MultiHeadSelfAttention(dim=64)
x = torch.rand(16, 10, 64) # [batch, tokens, dim]
mask = torch.zeros(10, 10) # tokens X tokens
mask[5:8, 5:8] = 1
y = model(x, mask)
```
#### Axial attention
```python
import torch
from self_attention_cv import AxialAttentionBlock
model = AxialAttentionBlock(in_channels=256, dim=64, heads=8)
x = torch.rand(1, 256, 64, 64) # [batch, tokens, dim, dim]
y = model(x)
```
#### Vanilla Transformer Encoder
```python
import torch
from self_attention_cv import TransformerEncoder
model = TransformerEncoder(dim=64,blocks=6,heads=8)
x = torch.rand(16, 10, 64) # [batch, tokens, dim]
mask = torch.zeros(10, 10) # tokens X tokens
mask[5:8, 5:8] = 1
y = model(x,mask)
```
#### Vision Transformer with/without ResNet50 backbone for image classification
```python
import torch
from self_attention_cv import ViT, ResNet50ViT
model1 = ResNet50ViT(img_dim=128, pretrained_resnet=False,
blocks=6, num_classes=10,
dim_linear_block=256, dim=256)
# or
model2 = ViT(img_dim=256, in_channels=3, patch_dim=16, num_classes=10,dim=512)
x = torch.rand(2, 3, 256, 256)
y = model2(x) # [2,10]
```
#### A re-implementation of Unet with the Vision Transformer encoder
```python
import torch
from self_attention_cv.transunet import TransUnet
a = torch.rand(2, 3, 128, 128)
model = TransUnet(in_channels=3, img_dim=128, vit_blocks=8,
vit_dim_linear_mhsa_block=512, classes=5)
y = model(a) # [2, 5, 128, 128]
```
#### Bottleneck Attention block
```python
import torch
from self_attention_cv.bottleneck_transformer import BottleneckBlock
inp = torch.rand(1, 512, 32, 32)
bottleneck_block = BottleneckBlock(in_channels=512, fmap_size=(32, 32), heads=4, out_channels=1024, pooling=True)
y = bottleneck_block(inp)
```
### Position embeddings are also available
#### 1D Positional Embeddings
```python
import torch
from self_attention_cv.pos_embeddings import AbsPosEmb1D,RelPosEmb1D
model = AbsPosEmb1D(tokens=20, dim_head=64)
# batch heads tokens dim_head
q = torch.rand(2, 3, 20, 64)
y1 = model(q)
model = RelPosEmb1D(tokens=20, dim_head=64, heads=3)
q = torch.rand(2, 3, 20, 64)
y2 = model(q)
```
#### 2D Positional Embeddings
```python
import torch
from self_attention_cv.pos_embeddings import RelPosEmb2D
dim = 32 # spatial dim of the feat map
model = RelPosEmb2D(
feat_map_size=(dim, dim),
dim_head=128)
q = torch.rand(2, 4, dim*dim, 128)
y = model(q)
```
## Acknowledgments
Thanks to Alex Rogozhnikov [@arogozhnikov](https://github.com/arogozhnikov) for the awesome einops package.
For my re-implementations I have sExcerpt of 6,355 characters
Read on GitHub
Adaloglou Nikolas · @The-AI-Summer
70
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:481875ebfdee0892, topic:deep-learning
matched fp:481875ebfdee0892, topic:transformer