Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICLR 2025] Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion
| Date | Stars |
|---|---|
| 2026-07-24 | 295 |
| 2026-07-25 | 295 |
| 2026-07-28 | 295 |
| 2026-07-30 | 295 |
| 2026-08-06 | 295 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# [ICLR 2025] Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion
## NEWS:
- [March. 2025] Inference code released! :rocket: :rocket: :rocket:
- [Feb. 2025] Phidias has been accepted to ICLR 2025! :fire: :fire: :fire:
https://github.com/user-attachments/assets/189ffcb2-d777-4d22-829c-c80c3a27a7fe
### [Project page](https://rag-3d.github.io/) | [Paper](https://arxiv.org/abs/2409.11406) | [Video](https://www.bilibili.com/video/BV11xtLeEERS/)
<!-- <br> -->
[Zhenwei Wang](https://zhenwwang.github.io/), [Tengfei Wang](https://tengfei-wang.github.io/), [Zexin He](https://github.com/ZexinHe), [Gerhard Hancke](https://rfidblog.org.uk/), [Ziwei Liu](https://liuziwei7.github.io/) and [Rynson W.H. Lau](https://www.cs.cityu.edu.hk/~rynson/).
<!-- <br> -->
## Abstract
>In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.
## Overview
<div class="half">
<img src="figures/overview.png" width="1080">
</div>
## Installation
- Environment Setup:
```bash
conda create -n phidias python==3.10
conda activate phidias
# install PyTorch and xFormers
# xformers is required! please refer to https://github.com/facebookresearch/xformers for details.
# for example, we use torch 2.1.0 + cuda 11.8
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 xformers --index-url https://download.pytorch.org/whl/cu118
# a modified gaussian splatting (+ depth, alpha rendering)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization
# for mesh extraction
pip install git+https://github.com/NVlabs/nvdiffrast
# other dependencies
pip install -r requirements.txt
```
- If you want to render reference maps by yourself, please install [Blender3.2.2](https://download.blender.org/release/Blender3.2/)
```
wget https://download.blender.org/release/Blender3.2/blender-3.2.2-linux-x64.tar.xz && \
tar -xf blender-3.2.2-linux-x64.tar.xz && \
rm blender-3.2.2-linux-x64.tar.xz
```
- Download our pretrained models and precomputed point-cloud features from [huggingface](https://huggingface.co/ZhenweiWang/Phidias-Diffusion/tree/main), and place them under `model/`
```bash
mkdir model
huggingface-cli download ZhenweiWang/Phidias-Diffusion --local-dir model/
```
## Inference
```bash
# Exaples 1:
# without 3D reference (vanilla zero123++ with white background)
python infer.py big --workspace results/no_ref --mv_controlnet_path None --rembg --test_path data_test/image_to_3d/chair_watermelon.png
# Exaples 2:
# user-specified 3D reference (3D-to-3D), using pre-rendered reference maps
python infer.py big --workspace results/3d_to_3d --rembg --test_path data_test/3d_to_3d --no-use_retrieval
# Exaples 3:
# retrieve 3D reference from objaverse subset and perform online rendering
python infer.py big --workspace results/image_to_3d --rembg --test_path data_test/image_to_3d --use_retrieval --top_k_retrieval 1 2 3 --online_rendering --blender_path blender-3.2.2-linuxExcerpt of 6,258 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:057d05f87bdf4b4a, topic:diffusion-models, readme:controlnet
matched fp:057d05f87bdf4b4a, topic:text-to-3d, readme:3d generation
matched fp:057d05f87bdf4b4a, topic:retrieval-augmented-generation