Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8 training).
| Date | Stars |
|---|---|
| 2026-07-24 | 399 |
| 2026-07-25 | 399 |
| 2026-07-28 | 399 |
| 2026-07-30 | 399 |
| 2026-07-31 | 399 |
| 2026-08-06 | 399 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# diffusers-torchao
**Optimize image and video generation with [`diffusers`](https://github.com/huggingface/diffusers), [`torchao`](https://github.com/pytorch/ao), combining `torch.compile()` 🔥**
We provide end-to-end inference and experimental training recipes to use `torchao` with `diffusers` in this repo. We demonstrate **53.88%** speedup on [Flux.1-Dev](https://huggingface.co/black-forest-labs/FLUX.1-dev)<sup>*</sup> and **27.33%** speedup on [CogVideoX-5b](https://huggingface.co/THUDM/CogVideoX-5b) when comparing *compiled* quantized models against their standard bf16 counterparts<sup>**</sup>.
<sub><sup>*</sup>The experiments were run on a single H100, 80 GB GPU.</sub>
<sub><sup>**</sup>The experiments were run on a single A100, 80 GB GPU. For a single H100, the speedup is **33.04%**</sub>
### Updates
* `torchao` is now officially supported as a quantization backend in `diffusers`. Check out the [docs](https://huggingface.co/docs/diffusers/main/en/quantization/torchao) for more details.
* `torchao` is being integrated into `diffusers` as an official quantization backend. Be on the lookout for [this PR](https://github.com/huggingface/diffusers/pull/10009) to get merged.
* `torchao` will soon be added as a quantization backend in `diffusers`, making it even easier to use with `diffusers`.
* Check out our new AoT compilation and serialization [guide](inference/aot_serialization.md) to reduce framework overheads.
No-frills code:
```diff
from diffusers import FluxPipeline
+ from torchao.quantization import autoquant
import torch
pipeline = FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16
).to("cuda")
+ pipeline.transformer = autoquant(pipeline.transformer, error_on_unseen=False)
image = pipeline(
"a dog surfing on moon", guidance_scale=3.5, num_inference_steps=50
).images[0]
```
Throw in `torch.compile()` to make it go brrr:
```diff
# If you are using "autoquant" then you should compile first and then
# apply autoquant.
+ pipeline.transformer.to(memory_format=torch.channels_last)
+ pipeline.transformer = torch.compile(
+ pipeline.transformer, mode="max-autotune", fullgraph=True
+)
```
This, alone, is sufficient to cut down inference time for Flux.1-Dev from 6.431 seconds to 3.483 seconds on an H100. Check out the [`inference`](./inference/) directory for the code.
> [!NOTE]
> Quantizing to a supported datatype and using base precision as fp16 can lead to overflows. The recommended base precision for CogVideoX-2b is fp16 while that of CogVideoX-5b is bf16. If comparisons were to be made in fp16, the speedup gains would be **~23%** and **~32%** respectively.
<h4>Table of contents</h4>
* [Environment](#environment)
* [Benchmarking results](#benchmarking-results)
* [Reducing quantization time and peak memory](#reducing-quantization-time-and-peak-memory)
* [Training with FP8](#training-with-fp8)
* [Serialization and loading quantized models](#serialization-and-loading-quantized-models)
* [Things to keep in mind when benchmarking](#things-to-keep-in-mind-when-benchmarking)
* [Benefitting from `torch.compile()`](#benefitting-from-torchcompile)
## Environment
We conducted all our experiments on a single A100 (80GB) and H100 GPUs. Since we wanted to benefit from `torch.compile()`, we used relatively modern cards here. For older cards, same memory savings (demonstrated more below) can be obtained.
We always default to using the PyTorch nightly, updated `diffusers` and `torchao` codebases. We used CUDA 12.2.
## Benchmarking results
We benchmark two models ([Flux.1-Dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) and [CogVideoX](https://huggingface.co/THUDM/CogVideoX-5b)) using different supported quantization datatypes in `torchao`. The results are as follows:
## Flux.1 Dev Benchmarks
<details>
<summary>Additional Results</summary>
| ckpt_id Excerpt of 47,988 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:2fa018632618fcaa, topic:diffusion-models, topic:text-to-image, topic:flux