Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICLR 2025 Oral] Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
| Date | Stars |
|---|---|
| 2026-07-31 | 1027 |
| 2026-08-05 | 1027 |
| 2026-08-06 | 1027 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://arxiv.org/abs/2503.09573) (ICLR 2025 Oral)
By [Marianne Arriola](https://m-arriola.com/), [Aaron Gokaslan](https://skylion007.github.io), [Justin T Chiu](https://justinchiu.netlify.app), [Zhihan Yang](https://zhihanyang2022.github.io/), [Zhixuan Qi](https://zhixuanqi.com/), [Jiaqi Han](https://hanjq17.github.io/), [Subham Sekhar Sahoo](https://s-sahoo.github.io), [Volodymyr Kuleshov](https://www.cs.cornell.edu/~kuleshov/)
<!-- [](https://colab.research.google.com/drive/18nC6q7dWq154fI1BXPLwmtnS7Zvbrv6p?usp=sharing/) -->
[](https://openreview.net/forum?id=tyEyYT267x)
[](https://m-arriola.com/bd3lms/)
[](https://huggingface.co/collections/kuleshov-group/bd3-lms-67be95f81b96b15fec50d53f)
We introduce ***BD3-LMs***, a family of **B**lock **D**iscrete **D**enoising **D**iffusion **L**anguage **M**odels that achieve SOTA likelihoods among diffusion models and enable generation of arbitrary-length sequences. BD3-LMs combine the strengths of autoregressive and diffusion language models by decomposing a token sequence into blocks and performing discrete diffusion within each block. By tuning the block size, we interpolate between autoregressive and diffusion models which introduces a trade-off between quality and sample efficiency. We propose a recipe for building effective BD3-LMs that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance.
<!-- We provide a demo in this [](https://colab.research.google.com/drive/18nC6q7dWq154fI1BXPLwmtnS7Zvbrv6p?usp=sharing/) notebook. -->
In this repo, we provide:
* **The BD3-LM framework**
1. Block-autoregressive likelihood parameterization
2. Data-driven noise schedules to reduce training variance
3. Arbitrary-length discrete diffusion samplers
* **Baseline implementations**
1. Autoregressive model [[AR](https://arxiv.org/abs/2406.07524)]
2. Score Entropy Based Discrete Diffusion [[SEDD](https://arxiv.org/abs/2310.16834)]
3. Masked Diffusion Language Model [[MDLM](https://arxiv.org/abs/2406.07524)]
4. Semi-autoregressive Simplex-based Diffusion Language Model [[SSD-LM](https://arxiv.org/pdf/2210.17432)] *(supports sample generation only)*
<a name="code-organization"></a>
## Code Organization
1. ```main.py```: Routines for training and evaluation
2. ```noise_schedule.py```: Noise schedules
3. ```diffusion.py```: Forward/reverse diffusion
4. ```dataloader.py```: Dataloaders
5. ```utils.py```: LR scheduler, logging, `fsspec` handling
6. ```models/```: Network architectures. Supports [DiT](https://arxiv.org/abs/2212.09748) and AR transformer
7. ```configs/```: Config files for datasets/models/noise schedules/LR schedules
8. ```scripts/```: Shell scripts for training/evaluation
- ``train/``: Training scripts (LM1B, OWT)
- ``ppl/``: Likelihood evaluation on the pretraining set (LM1B, OWT)
- ``zs_ppl/``: Zero-shot likelihood evaluation on GPT2 benchmark datasets
- ``gen_ppl/``: Sample quality (generative perplexity under GPT2)
- ``var_len/``: Arbitrary-length sequence generation
9. ```ssd-lm/```: SSD-LM codebase
- ```run_generate_text_batch.sh```: Generates SSD-LM samples
- ```report_genppl.py```: Reports generative perplexity of SSD-LM samples
<a name="getting_started"></a>
## Getting Started
To get started, create a conda environment containing the required dependencies.
```bash
conda create --name bd3lm python=3.9
conda activate bd3lm
pip install -r requirements.txt
```
While BD3-LMs don't require FlashExcerpt of 10,062 characters
Read on GitHub
11
2
Nick Doiron · Yes · Israel
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:2fde8cd90481ec65, llm:Repository title and description: 'Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models' (ICLR 2025 Oral) — Python implementation of novel language model architecture combining autoregressive and diffusion approaches.
matched fp:2fde8cd90481ec65, llm:Repository title and description: 'Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models' (ICLR 2025 Oral) — Python implementation of novel language model architecture combining autoregressive and diffusion approaches.
matched fp:2fde8cd90481ec65, llm:Repository title and description: 'Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models' (ICLR 2025 Oral) — Python implementation of novel language model architecture combining autoregressive and diffusion approaches.