Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
🍳 [CVPR'24 Highlight] Pytorch implementation of "Taming Stable Diffusion for Text to 360° Panorama Image Generation"
| Date | Stars |
|---|---|
| 2026-07-31 | 269 |
| 2026-08-04 | 269 |
| 2026-08-06 | 269 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# PanFusion ### Taming Stable Diffusion for Text to 360° Panorama Image Generation Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, Jianfei Cai ### [Project Page](https://chengzhag.github.io/publication/panfusion) | [Paper](https://arxiv.org/abs/2404.07949)  ## Introduction This repo contains data preprocessing, training, testing, evaluation code of our CVPR 2024 paper. ## Installation We use Anaconda to manage the environment. You can create the environment by running the following command: ```bash git clone https://github.com/chengzhag/PanFusion cd PanFusion conda env create -f environment.yaml conda activate panfusion ``` If you are having issue with conda solving environment, or any other issues that might be caused by the version of the packages, you can try to create the environment with specific version of the packages: ```bash conda env create -f environment_strict.yaml ``` We use [wandb](https://www.wandb.com/) to log and visualize the training process. You can create an account then login to wandb by running the following command: ```bash wandb login ``` We provide the wandb [report](https://wandb.ai/pidan1231239/pano_diffusion/reports/PanFusion--Vmlldzo3NzM1OTYy?accessToken=mmneovtrelnqd21gw5sk2cp8j0av65meohuf0ua850398sivq7duvkcvu934qlbt) for identifying issues when reproducing the results. ## Demo You can download the pretrained checkpoints [last.ckpt](https://monashuni-my.sharepoint.com/:u:/g/personal/cheng_zhang_monash_edu/EeTrujeSOgdHh7vWsjXuMPAB8JtTaXS1uR8sp0y1kwQ4NQ?e=cI5jec) and put it in the `logs/4142dlo4/checkpoints` folder. Then run the following command to test the model: ```bash WANDB_MODE=offline WANDB_RUN_ID=4142dlo4 python main.py predict --data=Matterport3D --model=PanFusion --ckpt_path=last ``` The generated images are saved in the `logs/4142dlo4/predict` folder. We also provide out-of-domain prompts for testing: ```bash WANDB_MODE=offline WANDB_RUN_ID=4142dlo4 python main.py predict --data=Demo --model=PanFusion --ckpt_path=last ``` ## Data Preparation ### Download Data We follow [MVDiffusion](https://github.com/Tangshitao/MVDiffusion) to download the [Matterport3D](https://niessner.github.io/Matterport/) skybox dataset. Specifically, please fill the sign the form to request a download script `download_mp.py` and put it in the `data/Matterport3D` folder. Then run the following command to download and unzip the data: ```bash cd data/Matterport3D python download_mp.py -o ./Matterport3D --type matterport_skybox_images python unzip_skybox.py ``` We also use the [splits](https://www.dropbox.com/scl/fi/recc3utsvmkbgc2vjqxur/mp3d_skybox.tar?rlkey=ywlz7zvyu25ovccacmc3iifwe&dl=0) provided by [MVDiffusion](https://github.com/Tangshitao/MVDiffusion). Please download it to `data/Matterport3D` and unzip it with the following command: ```bash cd data/Matterport3D tar -xvf mp3d_skybox.tar ``` ### Stitch Matterport3D Skybox The Matterport3D skybox images are stitched into equirectangular projection images for training. Please run the following command to stitch the images: ```bash python -m scripts.stitch_mp3d ``` The stitched images are saved in the `data/Matterport3D/mp3d_skybox/*/matterport_stitched_images` folder. ### Caption Images We use the perspective image captions generated by [MVDiffusion](https://github.com/Tangshitao/MVDiffusion) for evaluation. Please download the captions [mp3d_skybox.tar](https://www.dropbox.com/scl/fi/recc3utsvmkbgc2vjqxur/mp3d_skybox.tar?rlkey=ywlz7zvyu25ovccacmc3iifwe&dl=0) and put it in the `data/Matterport3D` folder. Then run the following command to unzip the captions: ```bash cd data/Matterport3D tar -xvf mp3d_skybox.tar ``` We use blip to caption the equirectangular images for training. You can download the generated captions [mp3d_stitched_caption.tar](https://monashuni-my.sharepoint.com/:u:/g/personal/cheng_zhang_monash_edu/Ec1A8tOmt_5ItvT2aktSUioBHzC_LRYjqaHPqipJuUhPHw
Excerpt of 10,410 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a6eebd25b9a60168, desc:image generation, desc:stable diffusion