Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR 2025.
| Date | Stars |
|---|---|
| 2026-07-31 | 281 |
| 2026-08-03 | 282 |
| 2026-08-06 | 282 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Mixture-of-Transformers (MoT)
[](https://github.com/facebookresearch/Mixture-of-Transformers/blob/main/LICENSE)
[](https://arxiv.org/abs/2411.04996)
[](https://x.com/liang_weixin/status/1854982365317415222)
💬 This playbook presents a step-by-step tutorial on how to implement the **Mixture-of-Transformer (MoT)** architecture on top of your own transformer model to enable native multimodal generation. It also contains a basic implementation.
> **Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models**. TMLR (March 2025).\
> Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Xi Victoria Lin\
> Paper: https://arxiv.org/abs/2411.04996
## Key Results
**Mixture-of-Transformers (MoT)** adopts **modality-aware sparsity** in every non-embedding transformer layer (e.g., feed-forward networks, attention matrices, and layer normalization).
- ✅ **Chameleon setting (text + image generation):**
MoT (7B) matches dense baseline quality using just **55.8% of the FLOPs**.
- ✅ **Chameleon setting (text + image + speech generation):**
MoT (443M) achieves dense-level speech quality with only **37.2% of the FLOPs**.
- ✅ **Transfusion setting (text autoregressive + image diffusion generation):**
MoT (7B) matches dense model quality using **one-third of the FLOPs**.
- ✅ **System profiling:**
MoT (Chameleon setting, 443M) achieves:
- Dense-level image quality in **47%** of the wall-clock time.
- Dense-level text quality in **75.6%** of the wall-clock time.
**(Measured on AWS `p4de.24xlarge` instances with NVIDIA A100 GPUs.)**
💬 Modality-aware sparsity in MoT offers an effective path to native generation of a growing number of modalities, from text and images to speech and beyond.
---
## Tutorial: Step-by-step implementation of MoT
💬 Let's dive in!
---
### **Prerequisite: Your Own Transformer Model**
To implement MoT, start with a basic transformer model implement. Here’s a simplified example:
```python
class FeedForward(torch.nn.Module):
...
class Attention(torch.nn.Module):
...
class TransformerBlock(torch.nn.Module):
def __init__(self, args):
super().__init__()
self.attention = Attention(args)
self.feed_forward = FeedForward(args)
def forward(self, x, *args):
x = x + self.attention(x, *args)
return x + self.feed_forward(x, *args)
class Transformer(torch.nn.Module):
def __init__(self, args):
super().__init__()
self.layers = nn.ModuleList([TransformerBlock(args) for _ in range(args.n_layers)])
def forward(self, x, *args):
for layer in self.layers:
x = layer(x, *args)
return x
```
---
### **Step 1: Modality-Specific Feed-Forward Networks**
The `ModalityUntiedFeedForward` class enhances the feed-forward network by creating separate experts for each modality, enabling specialized processing for tasks involving multiple modalities, such as text, image, or speech.
#### **Key Features**
1. **Decoupled Parameters**: Each modality has its own feed-forward expert, instantiated using the existing `FeedForward` class for modularity and simplicity.
2. **Modality-Specific Routing**: Tokens are (deterministically) routed to modality-specific expert based on the `modality_masks`.
3. **Modality-Specific normalization**: Each modality expert applies its own normalization layer to ensure tailored processing.
💬 Congrats! you have done more than half (67%) of the job -- because `Feedforward` typically takes 67% of the non-embedding parameters of Transformer!
#### **Code**
```python
class ModalityUntiedFeedForward(torch.nn.Module):
def __init_Excerpt of 14,762 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:10059df8b4cb3c33, llm:Repository description: 'Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR 2025.' (multi-modal, foundation models, architecture).
matched fp:10059df8b4cb3c33, llm:Repository description: 'Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR 2025.' (multi-modal, foundation models, architecture).
matched fp:10059df8b4cb3c33, llm:Repository description: 'Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR 2025.' (multi-modal, foundation models, architecture).