Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[SIGGRAPH 2025] Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control
| Date | Stars |
|---|---|
| 2026-07-24 | 823 |
| 2026-07-25 | 823 |
| 2026-07-28 | 824 |
| 2026-07-30 | 824 |
| 2026-08-06 | 824 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control
<a href='https://arxiv.org/abs/2501.03847'><img src='https://img.shields.io/badge/arXiv-2501.03847-b31b1b.svg'></a>
<a href='https://igl-hkust.github.io/das/'><img src='https://img.shields.io/badge/Project-Page-Green'></a>
[](https://huggingface.co/EXCAI/Diffusion-As-Shader)
[](https://huggingface.co/spaces/EXCAI/Diffusion-As-Shader)
## NEWS:
- Jun 5, 2025: We released our [script](assets/dataset.md) and Blender project for creating **synthetic datasets**.
- Jun 2, 2025: We added inference code based on `Wan2.1Fun 1.3B` fine-tuning to the `Wanfun` branch.
- Apr 2, 2025: Added functionality for complex and precise camera control for videos, based on `VGGT`. The `--override_extrinsics` hyperparameter can be adjusted to append or override camera motion in videos.
- Apr 1, 2025: Added support for `cotracker`.
- Feb 17, 2025: We uploaded a validation dataset to [Google Drive](https://drive.google.com/file/d/1pVB_2AEoz1v4vXWe6-pdDAEQdmlGEIci/view?usp=sharing), containing 4 tasks.
## Quickstart
### Create environment
1. Clone the repository and create conda environment:
```
git clone https://github.com/IGL-HKUST/DiffusionAsShader.git
conda create -n das python=3.10
conda activate das
```
2. Install pytorch, we recommend `Pytorch 2.5.1` with `CUDA 11.8`:
```
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu118
```
<!-- 3. Install `MoGe`:
```
pip install git+https://github.com/asomoza/image_gen_aux.git
``` -->
3. Make sure the submodule and requirements are installed:
```
mkdir -p submodules
git submodule update --init --recursive
pip install -r requirements.txt
```
If the submodules are not installed, you need to manually download them and move them to `submodules/`. Run the following commands to install the submodules:
```
# MoGe
git clone https://github.com/microsoft/MoGe.git submodules/MoGe
# VGGT
git clone https://github.com/facebookresearch/vggt.git submodules/vggt
```
4. Manually download these checkpoints to `checkpoints/`:
- SpatialTracker checkpoint: [Google Drive](https://drive.google.com/drive/folders/1UtzUJLPhJdUg2XvemXXz1oe6KUQKVjsZ).
- Our *Diffusion as Shader* checkpoint: https://huggingface.co/EXCAI/Diffusion-As-Shader
<!-- 5. Manually download the ZoeDepth checkpoints (dpt_beit_large_384.pt, ZoeD_M12_K.pt, ZoeD_M12_NK.pt) to `models/monoD/zoeDepth/ckpts/`. For more information, refer to [this issue](https://github.com/henry123-boy/SpaTracker/issues/20). -->
<!-- Then download a dataset:
```bash
# install `huggingface_hub`
huggingface-cli download \
--repo-type dataset Wild-Heart/Disney-VideoGeneration-Dataset \
--local-dir video-dataset-disney
``` -->
### Inference
The inference code was tested on
- Ubuntu 20.04
- Python 3.10
- PyTorch 2.5.1
- 1 NVIDIA H800 with CUDA version 11.8. (32GB GPU memory is sufficient for generating videos with our code.)
We provide a inference script for our tasks. You can run the `demo.py` script directly as follows.
**We also provide a validation dataset in [Google Drive](https://drive.google.com/file/d/1pVB_2AEoz1v4vXWe6-pdDAEQdmlGEIci/view?usp=sharing) for our 4 tasks. You can run the `scripts/evaluate_DaS.sh` to evaluate the performance of our model.**
We release the gradio interface for our tasks. You can run the `webui.py` script directly as follows.
```
python webui.py --gpu <gpu_id>
```
Or you can run these tasks one by one as follows.
#### 1. Motion Transfer
<table border="1">
<tr>
<th>Original</th>
<th>Object Replacement</th>
<th>Style Transfer</th>
</tr>
<tr>
<td><Excerpt of 17,532 characters
Read on GitHub
34
2
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:095f444f79d50620, topic:video-generation, desc:video generation, readme:video generation