Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
GPU-optimized framework for training diffusion language models at any scale. The backend of Quokka, Super Data Learners, and OpenMoE 2 training.
| Date | Stars |
|---|---|
| 2026-07-31 | 343 |
| 2026-08-02 | 343 |
| 2026-08-06 | 343 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center"> MegaDLMs =========================== <h4>GPU-optimized framework for training diffusion language models at any scale.</h4> [](https://deepwiki.com/JinjieNi/MegaDLMs) [](./CHANGELOG.md) [](./LICENSE) <div align="left"> <p align="center" width="100%"> <img src="images/vs_other_backend.png" width="80%" height="100%"> </p> <p align="center" width="100%"> <img src="images/weak_scaling.png" width="80%" height="100%"> </p> <br> # Highlights - **Comprehensive Training Pipelines**: Full support for Diffusion Language Models (DLMs) and Autoregressive LMs, from pre-training and SFT to RL, on both dense and MoE architectures. - **Ultra Speed and Scalability**: MegaDLMs offers up to **47% Model FLOP Utilization (MFU)** and **3× faster training speed** compared with other frameworks (see [here](#performance-benchmarking) for benchmarking). - **Cutting-edge Backend**: Leverage flexible parallelism from [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and GPU-optimized Transformer layers with fused kernels and full-precision (FP8, FP16, BF16) support from [Transformer Engine](https://github.com/NVIDIA/TransformerEngine). - **HuggingFace Integration**: Seamlessly work with HuggingFace checkpoints. <br> # Latest News - [2025-11-2] We release MegaDLMs, the training backend for [Quokka](https://github.com/JinjieNi/Quokka), [Super Data Learners](https://github.com/JinjieNi/dlms-are-super-data-learners), and [OpenMoE 2](https://github.com/JinjieNi/OpenMoE2), an ultra fast and scalable framework for any-scale DLM training. We will merge the MoE part once OpenMoE 2 training is done. <br> # Quick Start ## 1. Installation We strongly recommend using the [PyTorch NGC Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) for optimal compatibility. The `24.11-py3` version (`nvcr.io/nvidia/pytorch:24.11-py3`) is recommended: ``` docker pull nvcr.io/nvidia/pytorch:24.11-py3 ``` Or building an image with the docker file starting with: ``` FROM nvcr.io/nvidia/pytorch:24.11-py3 # the remaining Dockerfile content ``` > If external images are not supported in your cluster, follow the [Complete Installation Guide](https://github.com/NVIDIA/Megatron-LM?tab=readme-ov-file#installation) to install - Docker, pip variants (dev,lts,etc.), source installation, and system requirements. The specific requirements are detailed in `requirements.txt`. ## 2. Setup Envs Setup the environment variables as instructed in `envs/.env`. <br> # Project Structure We also built an [interactive doc](https://deepwiki.com/JinjieNi/MegaDLMs) with DeepWiki to help you better understand this repo. ``` mega-dlms/ ├── megatron/ │ ├── core/ # Megatron Core (kernels, parallelism, building blocks) │ │ ├── models/ # Transformer models │ │ ├── transformer/ # Transformer building blocks │ │ ├── tensor_parallel/ # Tensor parallelism │ │ ├── pipeline_parallel/ # Pipeline parallelism │ │ ├── distributed/ # Distributed training (FSDP, DDP) │ │ ├── optimizer/ # Optimizers │ │ ├── datasets/ # Dataset loaders │ │ ├── inference/ # Inference engines │ │ └── export/ # Model export (e.g. TensorRT-LLM) │ ├── training/ # Training scripts │ ├── inference/ # Inference server │ ├── legacy/ # Legacy components │ └── post_training/ # Post-training (RLHF, etc.) ├── examples/ # Ready-to-use training examples ├── tools/ # Utility tools ├── tests/ # Comprehensive test suite └── docs/ # Documentation ``` <br> # System Requirements ##
Excerpt of 16,033 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:539e92d03888ee9b, topic:large-language-models