Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch
| Date | Stars |
|---|---|
| 2026-07-24 | 548 |
| 2026-07-25 | 548 |
| 2026-07-28 | 548 |
| 2026-07-30 | 548 |
| 2026-07-31 | 548 |
| 2026-08-06 | 548 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img src="./nuwa.png" width="400px"></img>
## NÜWA - Pytorch
<a href="https://discord.gg/xBPBXfcFHd"><img alt="Join us on Discord" src="https://img.shields.io/discord/823813159592001537?color=5865F2&logo=discord&logoColor=white"></a></br>
Implementation of <a href="https://arxiv.org/abs/2111.12417">NÜWA</a>, state of the art attention network for text to video synthesis, in Pytorch. It also contain an extension into video and audio generation, using a dual decoder approach.
<a href="https://www.youtube.com/watch?v=InhMx1h0N40">Yannic Kilcher</a>
<a href="https://www.youtube.com/watch?v=C9CTnZJ9ZE0">DeepReader</a>
## Status
- March 2022 - seeing signs of life with a difficult version of <a href="https://github.com/lucidrains/nuwa-pytorch/discussions/2#discussioncomment-2375973">moving mnist</a>
- April 2022 - It seems as though a <a href="https://github.com/lucidrains/video-diffusion-pytorch">diffusion based method</a> has taken the new throne for SOTA. However, I will continue on with NUWA, extending it to use multi-headed codes + <a href="https://github.com/lucidrains/RQ-Transformer/blob/main/rq_transformer/hierarchical_causal_transformer.py">hierarchical causal transformer</a>. I think that direction is untapped for improving on this line of work.
## Install
```bash
$ pip install nuwa-pytorch
```
## Usage
First train the VAE
```python
import torch
from nuwa_pytorch import VQGanVAE
vae = VQGanVAE(
dim = 512,
channels = 3, # default is 3, but can be changed to any value for the training of the segmentation masks (sketches)
image_size = 256, # image size
num_layers = 4, # number of downsampling layers
num_resnet_blocks = 2, # number of resnet blocks
vq_codebook_size = 8192, # codebook size
vq_decay = 0.8 # codebook exponential decay
)
imgs = torch.randn(10, 3, 256, 256)
# alternate learning for autoencoder ...
loss = vae(imgs, return_loss = True)
loss.backward()
# and the discriminator ...
discr_loss = vae(imgs, return_discr_loss = True)
discr_loss.backward()
# do above for many steps
# return reconstructed images and make sure they look ok
recon_imgs = vae(imgs)
```
Then, with your learned VAE
```python
import torch
from nuwa_pytorch import NUWA, VQGanVAE
# autoencoder
vae = VQGanVAE(
dim = 64,
num_layers = 4,
image_size = 256,
num_conv_blocks = 2,
vq_codebook_size = 8192
)
# NUWA transformer
nuwa = NUWA(
vae = vae,
dim = 512,
text_num_tokens = 20000, # number of text tokens
text_enc_depth = 12, # text encoder depth
text_enc_heads = 8, # number of attention heads for encoder
text_max_seq_len = 256, # max sequence length of text conditioning tokens (keep at 256 as in paper, or shorter, if your text is not that long)
max_video_frames = 10, # number of video frames
image_size = 256, # size of each frame of video
dec_depth = 64, # video decoder depth
dec_heads = 8, # number of attention heads in decoder
dec_reversible = True, # reversible networks - from reformer, decoupling memory usage from depth
enc_reversible = True, # reversible encoders, if you need it
attn_dropout = 0.05, # dropout for attention
ff_dropout = 0.05, # dropout for feedforward
sparse_3dna_kernel_size = (5, 3, 3), # kernel size of the sparse 3dna attention. can be a single value for frame, height, width, or different values (to simulate axial attention, etc)
sparse_3dna_dilation = (1, 2, 4), # cycle dilation of 3d conv attention in decoder, for more range
shift_video_tokens = True # cheap relative positions for sparse 3dna transformer, by shifting along spatial dimensions by one
).cuda()
# data
text = torch.ranExcerpt of 19,908 characters
Read on GitHub
Phil Wang · United States
195
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:d29c6479b8ec27f7, topic:text-to-video, desc:text-to-video, readme:text-to-video
matched fp:d29c6479b8ec27f7, topic:deep-learning