Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
The official repository of the dots.llm1 base and instruct models proposed by rednote-hilab.
| Date | Stars |
|---|---|
| 2026-07-31 | 492 |
| 2026-08-06 | 493 |
Today
+1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# dots1
<p align="center">
<img src="figures/new_logo2.png" width="300"/>
<p>
<p align="center">
  🤗 <a href="https://huggingface.co/rednote-hilab">Hugging Face</a>   |    📑 <a href="https://www.arxiv.org/abs/2506.05767">Paper</a>   
<br>
🖥️ <a href="https://huggingface.co/spaces/rednote-hilab/dots-demo">Demo</a>   |   💬 <a href="figures/wechat.jpg">WeChat (微信)</a>   |   📕 <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c">rednote</a>   |    <a href="https://x.com/rednotehilab"> 𝕏 X </a>  
</p>
Visit our Hugging Face (click links above), search checkpoints with names starting with `dots.llm1` or visit the [dots1 collection](https://huggingface.co/collections/rednote-hilab/dotsllm1-68246aaaaba3363374a8aa7c), and you will find all you need! Enjoy!
## News
- 2025.06.06: We released the `dots.llm1` series. Check our [report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf) for more details!
## 1. Introduction
The `dots.llm1` model is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models.
Leveraging our meticulously crafted and efficient data processing pipeline, `dots.llm1` achieves performance comparable to Qwen2.5-72B after pretrained on high-quality corpus without synthetic data. To foster further research, we open-source intermediate training checkpoints spanning the entire training process, providing valuable insights into the learning dynamics of large language models.
<p align="center">
<img width="90%" src="./figures/performance.png">
</p>
## 2. Model Summary
**This repo contains the base and instruction-tuned `dots.llm1` model**. which has the following features:
- Type: A MoE model with 14B activated and 142B total parameters trained on high-quality corpus.
- Training Stages: Pretraining and SFT.
- Architecture: Multi-head Attention with QK-Norm in attention Layer, fine-grained MoE utilizing top-6 out of 128 routed experts, plus 2 shared experts.
- Number of Layers: 62
- Number of Attention Heads: 32
- Supported Languages: English, Chinese
- Context Length: 32,768 tokens
- License: MIT
The highlights from `dots.llm1` include:
- **Enhanced Data Processing**: We propose a scalable and fine-grained *three-stage* data processing framework designed to generate large-scale, high-quality and diverse data for pretraining.
- **No Synthetic Data during Pretraining**: High-quality non-synthetic tokens was used in base model pretraining.
- **Performance and Cost Efficiency**: `dots.llm1` is an open-source model that activates only *14B* parameters at inference, delivering both comprehensive capabilities and high computational efficiency.
- **Infrastructure**: We introduce an innovative MoE all-to-all communication and computation overlapping recipe based on interleaved 1F1B pipeline scheduling and an efficient grouped GEMM implementation to boost computational efficiency.
- **Open Accessibility to Model Dynamics**: Intermediate model checkpoints are released spanning the entire training process, facilitating future research into the learning dynamics of large language models.
## 3. Example Usage
### Model Downloads
<div align="center">
| **Model** | **#Total Params** | **#Activated Params** | **Context Length** | **Download Link** |
| :------------: | :------------: | :------------: | :------------: | :------------: |
| dots.llm1.base | 142B | 14B | 32K | [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.base) |
| dots.llm1.inst | 142B | 14B | 32K | [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.inst) |
</div>
### Docker (recommended)
The docker images are available on [Docker Hub](https://hub.docker.com/repository/docker/rednotehilab/dots1/tags), based on the official images.
You can start a server via vllm.
```shExcerpt of 8,106 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:556841d806e9a98b, topic:large-language-models, topic:llm
matched fp:556841d806e9a98b, topic:natural-language-processing