Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
| Date | Stars |
|---|---|
| 2026-07-31 | 1954 |
| 2026-08-06 | 1957 |
| 2026-08-14 | 1962 |
| 2026-08-18 | 1965 |
| 2026-08-19 | 1967 |
| 2026-08-20 | 1966 |
| 2026-08-22 | 1968 |
| 2026-08-24 | 1969 |
| 2026-08-26 | 1969 |
| 2026-08-28 | 1970 |
| 2026-08-30 | 1971 |
| 2026-09-01 | 1971 |
| 2026-09-02 | 1972 |
| 2026-09-07 | 1974 |
| 2026-09-13 | 1975 |
| 2026-09-15 | 1977 |
| 2026-09-18 | 1976 |
| 2026-09-21 | 1975 |
| 2026-09-22 | 1976 |
| 2026-09-26 | 1977 |
| 2026-09-28 | 1978 |
| 2026-09-30 | 1979 |
| 2026-10-02 | 1980 |
| 2026-10-04 | 1981 |
| 2026-10-06 | 1982 |
| 2026-10-07 | 1981 |
| 2026-10-12 | 1982 |
| 2026-10-13 | 1984 |
| 2026-10-20 | 1987 |
| 2026-10-21 | 1988 |
| 2026-10-22 | 1990 |
| 2026-10-23 | 1992 |
| 2026-10-25 | 1992 |
| 2026-10-27 | 1994 |
| 2026-10-28 | 1995 |
| 2026-10-29 | 1998 |
| 2026-10-30 | 2000 |
| 2026-11-02 | 2000 |
| 2026-11-03 | 2001 |
| 2026-11-04 | 2001 |
Today
— stars today
This week
+6 stars this week
This month
+20 stars this month
Momentum
0.0
growth rate 0.30%/day
<!-- markdownlint-disable first-line-h1 -->
<!-- markdownlint-disable html -->
<!-- markdownlint-disable no-duplicate-header -->
<div align="center">
<img src="images/logo.svg" width="60%" alt="DeepSeek LLM" />
</div>
<hr>
<div align="center">
<a href="https://www.deepseek.com/" target="_blank">
<img alt="Homepage" src="images/badge.svg" />
</a>
<a href="https://chat.deepseek.com/" target="_blank">
<img alt="Chat" src="https://img.shields.io/badge/🤖%20Chat-DeepSeek%20LLM-536af5?color=536af5&logoColor=white" />
</a>
<a href="https://huggingface.co/deepseek-ai" target="_blank">
<img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107&logoColor=white" />
</a>
</div>
<div align="center">
<a href="https://discord.gg/Tc7c45Zzu5" target="_blank">
<img alt="Discord" src="https://img.shields.io/badge/Discord-DeepSeek%20AI-7289da?logo=discord&logoColor=white&color=7289da" />
</a>
<a href="images/qr.jpeg" target="_blank">
<img alt="Wechat" src="https://img.shields.io/badge/WeChat-DeepSeek%20AI-brightgreen?logo=wechat&logoColor=white" />
</a>
<a href="https://twitter.com/deepseek_ai" target="_blank">
<img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-deepseek_ai-white?logo=x&logoColor=white" />
</a>
</div>
<div align="center">
<a href="LICENSE-CODE">
<img alt="Code License" src="https://img.shields.io/badge/Code_License-MIT-f5de53?&color=f5de53">
</a>
<a href="LICENSE-MODEL">
<img alt="Model License" src="https://img.shields.io/badge/Model_License-Model_Agreement-f5de53?&color=f5de53">
</a>
</div>
<p align="center">
<a href="#3-model-downloads">Model Download</a> |
<a href="#2-evaluation-results">Evaluation Results</a> |
<a href="#4-quick-start">Quick Start</a> |
<a href="#5-license">License</a> |
<a href="#6-citation">Citation</a>
</p>
<p align="center">
<a href="https://arxiv.org/pdf/2401.06066.pdf"><b>Paper Link</b>👁️</a>
</p>
## 1. Introduction
DeepSeekMoE 16B is a Mixture-of-Experts (MoE) language model with 16.4B parameters.
It employs an innovative MoE architecture, which involves two principal strategies: fine-grained expert segmentation and shared experts isolation.
It is trained from scratch on 2T English and Chinese tokens, and exhibits comparable performance with DeekSeek 7B and LLaMA2 7B, with only about 40% of computations.
For research purposes, we release the model checkpoints of DeepSeekMoE 16B Base and DeepSeekMoE 16B Chat to the public, which can be deployed on a single GPU with 40GB of memory without the need for quantization.
The model code file can be found [here](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/main/modeling_deepseek.py).
## 2. Evaluation Results
### DeepSeekMoE 16B Base
We evaluate DeepSeekMoE 16B on various benchmarks and compare it with a series of models, as shown in the following.
- Comparison with open source models on the Open LLM Leaderboard. DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.
<p align="center">
<img src="images/evaluation_deepseekmoe16b_base_openllm.jpg" alt="table" width="50%">
</p>
- Comparison with DeepSeek 7B on our internal benchmarks. DeepSeek 7B is a dense model trained on the same corpus as DeepSeekMoE 16B. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.
<p align="center">
<img src="images/evaluation_deepseekmoe16b_base_1.jpg" alt="table" width="50%">
</p>
- Comparison with LLaMA2 7B on our internal benchmarks. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.
<p align="center">
<img src="images/evaluation_deExcerpt of 12,241 characters
Read on GitHub
8
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:35eb08da6f1711bd, desc:mixture of experts