Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICLR2025] DiffuGPT and DiffuLLaMA: Scaling Diffusion Language Models via Adaptation from Autoregressive Models
| Date | Stars |
|---|---|
| 2026-07-31 | 401 |
| 2026-08-01 | 401 |
| 2026-08-06 | 401 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<p align="center" width="100%"> <img src="assets/logo.png" alt="logo" style="width: 75%; min-width: 300px; display: block; margin: auto;"> </p> # DiffuLLaMA & DiffuGPT: Scaling Diffusion Language Models [](https://huggingface.co/diffusionfamily) [](https://arxiv.org/abs/2410.17891) ## Overview Current Diffusion Language Models (DLMs) have been studied at a smaller scale compared to their autoregressive (AR) counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. We propose adapting existing AR models to build text diffusion models. We demonstrate connections between AR and diffusion modeling objectives and introduce a simple continual pre-training approach for training diffusion models. Through systematic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models **DiffuGPT** and **DiffuLLaMA**, using less than 200B tokens for training. Here we open-source adaptation code, efficient fine-tuning scripts, and evaluation toolkits. <p align="center" width="100%"> <img src="assets/overview.png" alt="overview" style="width: 100%; min-width: 300px; display: block; margin: auto;"> </p> <p align = "center"> The overview of adaptation process. </p> ### ⌚️ Update - May 2025. Minor updates for paper and code (bug fixed). Checkout the latest dLLM [Dream-7B](https://hkunlp.github.io/blog/2025/dream/) from our lab. 🎉 - 19 Feb 2025. DiffuLLaMA has been accepted at ICLR2025! 🎉 Checkout our camera ready copy at [openreview](https://openreview.net/pdf?id=j1tSLYKwg8). We uploaded [LoRA adapater of DiffuLLaMA](https://huggingface.co/diffusionfamily/diffullama-gsm) finetuned on GSM8K-symbolic data. - 22 Nov 2024. Update evaluation scripts. ### 📖 Setup Basic environments: ```bash python 3.11 transformers 4.44.2 torch 2.1.1+cu121 ``` For quick start: ```bash pip install -r LLaMA-Factory/requirements.txt pip install flash-attn==2.6.3 --no-build-isolation ``` Flash-attention is not required, but we suggest to use for fast training/inference. For training and finetuning, we provide our customed [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) based on `0.8.4.dev0`. ```bash cd LLaMA-Factory pip install -e ".[torch,metrics]" ``` > 📌 Note: We suggest you to set the HuggingFace cache_dir: `export HF_HOME=/path-to-huggingface/cache/`. For efficient inference, please install [flash-attention-2](https://github.com/Dao-AILab/flash-attention). Explanation for folders ```bash ├── DiffuLLaMA-training/ # our code to adapt LLaMA2, implemented using transformers, considering that LLaMA-Factory is less efficient for pre-training ├── evaluation/ # evaluation scripts ├── example_output/ # some generation examples ├── LLaMA-Factory/ # our code to adapt GPT, and LoRA finetuning │ ├── src/llamafactory # integrate diffusion training in LLaMA-Factory │ │ ├── train/ddm # main folder for diffusion training │ │ ├── model/loader # loading models │ │ ├── data/loader # loading data │ │ ├── attention_patch.py # we modify the way to pass the attention_mask variable │ ├── examples # config for diffusion training ├── attention_patch.py # we modify the way to pass the attention_mask variable ├── model.py # defined model for quick inference ├── inf_diffugpt.py # quick start inference ├── inf_diffullama.py # quick start inference ``` ### 🚀 Quick Start We provide the example code for inference and you can directly do some unconditional and conditional generation. ```bash ### DiffuGPT-small inference example python inf_diffugpt.py --model_name diffusionfamily/diffugpt-s --base_model_name gpt2 ### DiffuGPT-medium inference example python inf_diffugpt.py --model_name diffusionfam
Excerpt of 9,708 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9f70d0ee3fbf312e, llm:Repository title and description: 'DiffuLLaMA' and '[ICLR2025] DiffuGPT and DiffuLLaMA: Scaling Diffusion Language Models via Adaptation from Autoregressive Models' — diffusion language models (LMs) adapted from autoregressive models.
matched fp:9f70d0ee3fbf312e, llm:Repository title and description: 'DiffuLLaMA' and '[ICLR2025] DiffuGPT and DiffuLLaMA: Scaling Diffusion Language Models via Adaptation from Autoregressive Models' — diffusion language models (LMs) adapted from autoregressive models.
matched fp:9f70d0ee3fbf312e, llm:Repository title and description: 'DiffuLLaMA' and '[ICLR2025] DiffuGPT and DiffuLLaMA: Scaling Diffusion Language Models via Adaptation from Autoregressive Models' — diffusion language models (LMs) adapted from autoregressive models.