Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A PyTorch implementation of the paper "All are Worth Words: A ViT Backbone for Diffusion Models".
| Date | Stars |
|---|---|
| 2026-07-31 | 1107 |
| 2026-08-02 | 1107 |
| 2026-08-06 | 1107 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
## U-ViT<br> <sub><small>Official PyTorch implementation of [All are Worth Words: A ViT Backbone for Diffusion Models](https://arxiv.org/abs/2209.12152) (CVPR 2023)</small></sub> 💡Projects with U-ViT: * [UniDiffuser](https://github.com/thu-ml/unidiffuser), a multi-modal large-scale diffusion model based on a 1B U-ViT, is open-sourced * [DPT](https://arxiv.org/abs/2302.10586), [code](https://github.com/ML-GSAI/DPT), [demo](https://ml-gsai.github.io/DPT-demo) a conditional diffusion model trained with 1 label/class with SOTA SSL generation and classification results on ImageNet <img src="uvit.png" alt="drawing" width="400"/> Vision transformers (ViT) have shown promise in various vision tasks while the U-Net based on a convolutional neural network (CNN) remains dominant in diffusion models. We design a simple and general ViT-based architecture (named U-ViT) for image generation with diffusion models. U-ViT is characterized by treating all inputs including the time, condition and noisy image patches as tokens and employing long skip connections between shallow and deep layers. We evaluate U-ViT in unconditional and class-conditional image generation, as well as text-to-image generation tasks, where U-ViT is comparable if not superior to a CNN-based U-Net of a similar size. In particular, latent diffusion models with U-ViT achieve record-breaking FID scores of 2.29 in class-conditional image generation on ImageNet 256x256, and 5.48 in text-to-image generation on MS-COCO, among methods without accessing large external datasets during the training of generative models. Our results suggest that, for diffusion-based image modeling, the long skip connection is crucial while the down-sampling and up-sampling operators in CNN-based U-Net are not always necessary. We believe that U-ViT can provide insights for future research on backbones in diffusion models and benefit generative modeling on large scale cross-modality datasets. -------------------- This codebase implements the transformer-based backbone 📌*U-ViT*📌 for diffusion models, as introduced in the [paper](https://arxiv.org/abs/2209.12152). U-ViT treats all inputs as tokens and employs long skip connections. *The long skip connections grealy promote the performance and the convergence speed*. <img src="skip_im.png" alt="drawing" width="400"/> 💡This codebase contains: * An implementation of [U-ViT](libs/uvit.py) with optimized attention computation * Pretrained U-ViT models on common image generation benchmarks (CIFAR10, CelebA 64x64, ImageNet 64x64, ImageNet 256x256, ImageNet 512x512) * Efficient training scripts for [pixel-space diffusion models](train.py), [latent space diffusion models](train_ldm_discrete.py) and [text-to-image diffusion models](train_t2i_discrete.py) * Efficient evaluation scripts for [pixel-space diffusion models](eval.py) and [latent space diffusion models](eval_ldm_discrete.py) and [text-to-image diffusion models](eval_t2i_discrete.py) * A Colab notebook demo for sampling from U-ViT on ImageNet (FID=2.29) [](https://colab.research.google.com/github/baofff/U-ViT/blob/main/UViT_ImageNet_demo.ipynb) <img src="sample.png" alt="drawing" width="800"/> 💡This codebase supports useful techniques for efficient training and sampling of diffusion models: * Mixed precision training with the [huggingface accelerate](https://github.com/huggingface/accelerate) library (🥰automatically turned on) * Efficient attention computation with the [facebook xformers](https://github.com/facebookresearch/xformers) library (needs additional installation) * Gradient checkpointing trick, which reduces ~65% memory (🥰automatically turned on) * With these techniques, we are able to train our largest U-ViT-H on ImageNet at high resolutions such as 256x256 and 512x512 using a large batch size of 1024 with *only 2 A100*❗ Training speed and memory of U-ViT-H/2 on ImageNet 256x256 using a
Excerpt of 15,874 characters
Read on GitHub
22
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6b5e0831c5c3ba61, llm:Repository description: 'A PyTorch implementation of the paper "All are Worth Words: A ViT Backbone for Diffusion Models".' Language: Jupyter Notebook.
matched fp:6b5e0831c5c3ba61, llm:Repository description: 'A PyTorch implementation of the paper "All are Worth Words: A ViT Backbone for Diffusion Models".' Language: Jupyter Notebook.
matched fp:6b5e0831c5c3ba61, llm:Repository description: 'A PyTorch implementation of the paper "All are Worth Words: A ViT Backbone for Diffusion Models".' Language: Jupyter Notebook.