Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
SwissArmyTransformer is a flexible and powerful library to develop your own Transformer variants.
| Date | Stars |
|---|---|
| 2026-07-24 | 1119 |
| 2026-07-25 | 1119 |
| 2026-07-28 | 1119 |
| 2026-07-30 | 1119 |
| 2026-08-06 | 1119 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Introduction
`sat`(`SwissArmyTransformer`) is a flexible and powerful library to develop your own Transformer variants.
`sat` is named after "swiss army knife", meaning that all the models (e.g. BERT, GPT, T5, GLM, CogView, ViT...) **share the same backbone code** and cater for versatile usages with some extra light-weight mixins.
`sat` is powered by `deepspeed-ZeRO` and model parallelism, aiming to provide the best practice for pretraining and finetuning large models (100M\~20B parameters).
## Install
```
pip install SwissArmyTransformer
```
# Features
* **Add model-agnostic components**, e.g. prefix-tuning, in just *ONE* line!
- [Prefix-tuning](https://arxiv.org/pdf/2101.00190) (or [P-tuning](https://arxiv.org/abs/2103.10385)) improves finetuning via adding trainable parameters in each attention layer. To apply it to a [GLM](https://arxiv.org/pdf/2103.10360.pdf) classification (or any other) model is easy with our library.
```python
class ClassificationModel(GLMModel): # can also be BertModel, RobertaModel, etc.
def __init__(self, args, transformer=None, **kwargs):
super().__init__(args, transformer=transformer, **kwargs)
self.add_mixin('classification_head', MLPHeadMixin(args.hidden_size, 2048, 1))
# Arm an arbitrary model with Prefix-tuning with this line!
self.add_mixin('prefix-tuning', PrefixTuningMixin(args.num_layers, args.hidden_size // args.num_attention_heads, args.num_attention_heads, args.prefix_len))
```
- GPT and other auto-regressive models act differently during training and inference. During inference, text is generated token-by-token and we need to cache previous states for efficiency. With our lib, you only need to consider the behavior during training (teacher-forcing) and transform it to a cached auto-regressive model via adding a mixin:
```python
model, args = AutoModel.from_pretrained('glm-10b-chinese', args)
model.add_mixin('auto-regressive', CachedAutoregressiveMixin())
# Generate a sequence with beam search
from sat.generation.autoregressive_sampling import filling_sequence
from sat.generation.sampling_strategies import BeamSearchStrategy
output, *mems = filling_sequence(model, input_seq,
batch_size=args.batch_size,
strategy=BeamSearchStrategy(args.batch_size))
```
* **Build your Transformer-based model with minimal codes**. We mentioned [GLM](https://arxiv.org/pdf/2103.10360.pdf), which only differs from standard transformer (called BaseModel) on position embedding (and training losses). We only need to focus on the related part when coding.
<details><summary>Extend the whole definition: </summary><p>
```python
class BlockPositionEmbeddingMixin(BaseMixin):
# Here define parameters for the mixin
def __init__(self, max_sequence_length, hidden_size, init_method_std=0.02):
super(BlockPositionEmbeddingMixin, self).__init__()
self.max_sequence_length = max_sequence_length
self.hidden_size = hidden_size
self.block_position_embeddings = torch.nn.Embedding(max_sequence_length, hidden_size)
torch.nn.init.normal_(self.block_position_embeddings.weight, mean=0.0, std=init_method_std)
# Here define the method for the mixin
def position_embedding_forward(self, position_ids, **kwargs):
position_ids, block_position_ids = position_ids[:, 0], position_ids[:, 1]
position_embeddings = self.transformer.position_embeddings(position_ids)
block_position_embeddings = self.block_position_embeddings(block_position_ids)
return position_embeddings + block_position_embeddings
class GLMModel(BaseModel):
def __init__(self, args, transformer=None):
super().__init__(args, transformer=transformer)
self.add_mixin('Excerpt of 8,956 characters
Read on GitHub
Qingsong Lv
239
169
Sleepy_chord
106
Zhengxiao Du · Tsinghua University
52
24
21
12
Jiazheng Xu · Tsinghua University, KEG Group · China
5
Aohan Zeng · Z.ai & Tsinghua University · China
5
Wenyi Hong · Tsinghua University
4
Fan Zhang · Tianjin University
2
lykeven · Tsinghua University · China
2
Jintao · Alibaba
2
Pierre Fernandez · Meta · France
1
1
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:49242d3f9999571f, topic:pytorch, readme:pretraining
matched fp:49242d3f9999571f, topic:transformer