Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (CVPR 2023)
| Date | Stars |
|---|---|
| 2026-07-31 | 523 |
| 2026-08-03 | 523 |
| 2026-08-06 | 523 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation [Haochen Wang*](https://whc.is/), [Xiaodan Du*](https://xiaodan.io), [Jiahao Li*](https://www.linkedin.com/in/jiahaoli95/), [Raymond A. Yeh†](https://raymond-yeh.com), [Greg Shakhnarovich](https://home.ttic.edu/~gregory/) (* indicates equal contribution) TTI-Chicago, †Purdue University Abstract: *A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.* <a href="https://arxiv.org/abs/2212.00774"><img src="https://img.shields.io/badge/arXiv-2212.00774-b31b1b.svg" height=22.5></a> <a href="https://pals.ttic.edu/p/score-jacobian-chaining"><img src="https://img.shields.io/website?down_color=lightgrey&down_message=offline&label=Project%20Page&up_color=lightgreen&up_message=online&url=https%3A%2F%2Fpals.ttic.edu%2Fp%2Fscore-jacobian-chaining" height=22.5></a> <a href="https://colab.research.google.com/drive/1zixo66UYGl70VOPy053o7IV_YkQt5lCZ?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" height=22.5></a> <a href="https://huggingface.co/spaces/MirageML/sjc"><img src="https://img.shields.io/badge/🤗-Huggingface%20Space-cyan.svg" height=22.5></a> <!-- [ [arxiv](https://arxiv.org/abs/2212.00774) | [project page](https://pals.ttic.edu/p/score-jacobian-chaining) | [colab](https://colab.research.google.com/drive/1zixo66UYGl70VOPy053o7IV_YkQt5lCZ?usp=sharing ) ] --> Many thanks to [dvschultz](https://github.com/dvschultz) for the colab, and [AmanKishore](https://github.com/AmanKishore) for the hugging face demo. SJC is now integrated in [threestudio](https://github.com/threestudio-project/threestudio) as well. <img src="https://user-images.githubusercontent.com/17956191/206826892-d056ad70-b586-4b39-a4d3-6f91c814aa10.gif" width="500px"/> ## Updates - We have added subpixel rendering script for final high quality vis. The jittery videos you might have seen should be significantly better now. Please run `python /path/to/sjc/highres_final_vis.py` in the exp folder after the training is complete. There are a few toggles in the script you can play with, but the default is ok. It takes about 5 minutes / 11GB on an A5000, and the extra time is mainly due to SD Decoder. - If you are running SJC with a DreamBooth fine-tuned model: the model's output distribution is already significantly narrowed. It might help to use a lower guidance scale `--sd.scale 50.0` for example. Intense mode-seeking is one cause for multi-face problem. We have internally tried DreamBooth with view-dependent prompt fine-tuning. But by and large DreamBooth integration is not ready. ## TODOs - [ ] make seeds configurable. So far all seeds are hardcoded to 0. - [ ] add script to reproduce 2D experiments in Fig 4. The Fig might need change once it's tied to seeds. Note that for a simple aligned domain like faces, simple scheduling like using a single σ=1.5 could already generate some nice images. But not so for bedrooms; it's too diverse and annealing seems still needed. - [ ] main paper figures did not use subpix rendering; appendix figures did. Replace the main paper figures to make them consistent. ## License Since we use Stable Diffusion, we are releasing under their OpenRAIL license. Otherwise we do not identify any components or upstream code that carry restrictive
Excerpt of 13,020 characters
Read on GitHub
24
8
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:2b2286ec27e1b3ee, topic:diffusion-models