Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
| Date | Stars |
|---|---|
| 2026-07-31 | 1521 |
| 2026-08-04 | 1524 |
| 2026-08-06 | 1524 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center"> <a href="Moonlight.pdf"><img width="80%" src="figures/banner.png"></a> </div> <!-- # Muon is Scalable For LLM Training --> <div align="center"> <a href="Moonlight.pdf"><img src="figures/logo.png" height="16" width="16" style="vertical-align:middle"><b> Tech Report</b></a> | <a href="https://huggingface.co/moonshotai/Moonlight-16B-A3B"><img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" height="16" width="16" style="vertical-align:middle"><b> HuggingFace</b></a> | <a href="https://github.com/NVIDIA/Megatron-LM/pull/1428"><img src="figures/megatron.png" height="16" width="16" style="vertical-align:middle"><b>Megatron-LM</b></a> </div> ## Abstract Recently, the [Muon optimizer](https://github.com/KellerJordan/Muon) based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves ∼ 2× computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce **Moonlight**, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research. Our code is available at [MoonshotAI/Moonlight](https://github.com/MoonshotAI/Moonlight). ## Key Ingredients Our work builds upon Muon while systematically identifying and resolving its limitations in large-scale training scenarios. Our technical contributions include: - **Analysis for Effective Scaling of Muon**: Through extensive analysis, we identify that weight decay plays a crucial roles in Muon's scalability. Besides, we proposed to keep a consistent update root mean square (RMS) across different matrix and non-matrix parameters through parameter-wise update scale adjustments. Such adjustments significantly enhanced training stability. - **Efficient Distributed Implementation**: We develop a distributed version of Muon with ZeRO-1 style optimization, achieving optimal memory efficiency and reduced communication overhead while preserving the mathematical properties of the algorithm. - **Scaling Law Validation**: We performed scaling law research that compares Muon with strong AdamW baselines, and showed the superior performance of Muon (see Figure 1). Based on the scaling law results, Muon achieves comparable performance to AdamW trained counterparts while requiring only approximately 52% of the training FLOPs. <div align="center"> <img width="90%" src="figures/scaling.png"> <p><em>Scaling up with Muon. <b>(a)</b> Scaling law experiments comparing Muon and Adam. Muon is 2 times more sample efficient than Adam. <b>(b)</b> The MMLU performance of our Moonlight model optimized with Muon and other comparable models. Moonlight advances the Pareto frontier of performance vs training FLOPs.</em></p> </div> ## Performance We named our lightweight model trained with Muon "Moonlight". We compared Moonlight with SOTA public models at similar scale: - **LLAMA3-3B** is a 3B-parameter dense model trained with 9T tokens - **Qwen2.5-3B** is a 3B-parameter dense model trained with 18T tokens - **Deepseek-v2-Lite** is a 2.4B/16B-parameter MOE model trained with 5.7T tokens | | **Benchmark (Metric)** | **Llama3.2-3B** | **Qwen2.5-3B** | **DSV2-Lite** | **Moonlight** | |---|---|---|---|---|---| | | Activated Param† | 2.81B | 2.77
Excerpt of 8,502 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c0b921e9c6891be5, llm:description: 'Muon is Scalable for LLM Training' (repo description)
matched fp:c0b921e9c6891be5, llm:description: 'Muon is Scalable for LLM Training' (repo description)
matched fp:c0b921e9c6891be5, llm:description: 'Muon is Scalable for LLM Training' (repo description)
