Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538
| Date | Stars |
|---|---|
| 2026-07-24 | 1246 |
| 2026-07-25 | 1247 |
| 2026-07-28 | 1247 |
| 2026-07-30 | 1247 |
| 2026-08-06 | 1247 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# The Sparsely Gated Mixture of Experts Layer for PyTorch
This repository contains the PyTorch re-implementation of the sparsely-gated MoE layer described in the paper [Outrageously Large Neural Networks](https://arxiv.org/abs/1701.06538) for PyTorch.
```python
from moe import MoE
import torch
# instantiate the MoE layer
model = MoE(input_size=1000, output_size=20, num_experts=10,hidden_size=66, k= 4, noisy_gating=True)
X = torch.rand(32, 1000)
#train
model.train()
# forward
y_hat, aux_loss = model(X)
# evaluation
model.eval()
y_hat, aux_loss = model(X)
```
# Requirements
To install the requirements run:
```pip install -r requirements.py```
# Example
The file ```example.py``` contains a minimal working example illustrating how to train and evaluate the MoE layer with dummy inputs and targets. To run the example:
```python example.py```
# CIFAR 10 example
The file ```cifar10_example.py``` contains a minimal working example of the CIFAR 10 dataset. It achieves an accuracy of 39% with arbitrary hyper-parameters and not fully converged. To run the example:
```python cifar10_example.py```
# Used by
[FastMoE: A Fast Mixture-of-Expert Training System](https://arxiv.org/pdf/2103.13262.pdf) This implementation was used as a reference PyTorch implementation for single-GPU training.
# Acknowledgements
The code is based on the TensorFlow implementation that can be found [here](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/expert_utils.py).
# Citing
```
@misc{rau2019moe,
title={Sparsely-gated Mixture-of-Experts PyTorch implementation},
author={Rau, David},
journal={https://github.com/davidmrau/mixture-of-experts},
year={2019}
}
```
Excerpt of 1,923 characters
Read on GitHub
25
2
inisis
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c1052855721b6a8e, topic:moe, name:mixture of experts, desc:mixture of experts
matched fp:c1052855721b6a8e, topic:pytorch