Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[WACV'25 Oral] Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
| Date | Stars |
|---|---|
| 2026-07-31 | 520 |
| 2026-08-06 | 520 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Fine-Tuning Image-Conditional Diffusion Models
[[`Paper`](https://arxiv.org/abs/2409.11355)] [[`Project Page`](https://gonzalomartingarcia.github.io/diffusion-e2e-ft/)] [[`HF demo depth`](https://huggingface.co/spaces/GonzaloMartinGarcia/marigold-e2e-ft-depth)] [[`HF demo normals`](https://huggingface.co/spaces/GonzaloMartinGarcia/marigold-e2e-ft-normals)] [[`BibTeX`](#-Citation)]
<img src="assets/teaser_images.png" width="600" alt="Teaser Images">
## 📢 News
- 2024-10-28: Accepted to WACV 2025.
- 2024-10-17: Accepted to NeurIPS 2024 AFM Workshop.
- 2024-10-23: Training code release.
- 2024-09-24: Evaluation code release.
- 2024-09-18: Inference code release.
## ⏩ Quickstart
```bash
pip install torch diffusers transformers accelerate
```
```python
from diffusers import DiffusionPipeline
import diffusers
image = diffusers.utils.load_image(
"https://gonzalomartingarcia.github.io/diffusion-e2e-ft/static/lego.jpg"
)
# Depth
pipe = DiffusionPipeline.from_pretrained(
"GonzaloMartinGarcia/marigold-e2e-ft-depth",
custom_pipeline="GonzaloMartinGarcia/marigold-e2e-ft-depth",
).to("cuda")
depth = pipe(image)
pipe.image_processor.visualize_depth(depth.prediction)[0].save("depth.png")
pipe.image_processor.export_depth_to_16bit_png(depth.prediction)[0].save("depth_16bit.png")
# Normals
pipe = DiffusionPipeline.from_pretrained(
"GonzaloMartinGarcia/stable-diffusion-e2e-ft-normals",
custom_pipeline="GonzaloMartinGarcia/marigold-e2e-ft-normals",
).to("cuda")
normals = pipe(image)
pipe.image_processor.visualize_normals(normals.prediction)[0].save("normals.png")
```
## 🔧 Development Setup
Tested with Python 3.10.
1. Clone repository:
```bash
git clone https://github.com/VisualComputingInstitute/diffusion-e2e-ft.git
cd diffusion-e2e-ft
```
2. Install dependencies:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```
## 🤖 Models
The following checkpoints are available for inference. Note that the Marigold (Depth) and GeoWizard (Depth & Normals) diffusion estimators are the official checkpoints provided by their respective authors and were not trained by us. Following the Marigold training regimen, we have trained a Marigold diffusion estimator for normals.
"E2E FT" denotes models we have fine-tuned end-to-end on task-specific losses, either starting from the pretrained diffusion estimator or directly from Stable Diffusion.
Since the fine-tuned models are single-step deterministic models, the noise should always be `zeros` and the ensemble size and number of inference steps should always be `1`.
| Models | Diffusion Estimator | Stable Diffusion + E2E FT | Diffusion Estimator + E2E FT |
|-------------------------------|-------------------------------|---------------------------------------------|-------------------------------------|
| **Marigold (Depth)** | `prs-eth/marigold-depth-v1-0` | `GonzaloMartinGarcia/stable-diffusion-e2e-ft-depth` | `GonzaloMartinGarcia/marigold-e2e-ft-depth` |
| **Marigold (Normals)** | `GonzaloMartinGarcia/marigold-normals` | `GonzaloMartinGarcia/stable-diffusion-e2e-ft-normals` | `GonzaloMartinGarcia/marigold-e2e-ft-normals` |
| **GeoWizard (Depth&Normals)** | `lemonaddie/geowizard` | N/A | `GonzaloMartinGarcia/geowizard-e2e-ft` |
## 🏃 Inference
1. Marigold checkpoints:
```bash
python Marigold/run.py \
--checkpoint="GonzaloMartinGarcia/marigold-e2e-ft-depth" \
--modality depth \
--input_rgb_dir="input" \
--output_dir="output/marigold_ft"
```
```bash
python Marigold/run.py \
--checkpoint="GonzaloMartinGarcia/marigold-e2e-ft-normals" \
--modality normals \
--input_rgb_dir="input" \
--output_dir="output/marigold_ft"
```
| Argument | Description |
|-------------------------|-------------|
| `--checkpoint` | Hugging Face model path. |
| `--modality` Excerpt of 16,519 characters
Read on GitHub
13
6
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:91128725a5c60eff, desc:fine-tuning, desc:fine tuning