Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch
| Date | Stars |
|---|---|
| 2026-07-24 | 790 |
| 2026-07-25 | 790 |
| 2026-07-28 | 790 |
| 2026-07-30 | 790 |
| 2026-08-06 | 790 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img src="./phenaki.png" width="450px"></img>
## <a href="https://en.wikipedia.org/wiki/Phenakistiscope">Phenaki</a> - Pytorch
Implementation of <a href="https://phenaki.video/">Phenaki Video</a>, which uses <a href="https://arxiv.org/abs/2202.04200">Mask GIT</a> to produce text guided videos of up to 2 minutes in length, in Pytorch. It will also combine another technique involving a <a href="https://arxiv.org/abs/2209.04439">token critic</a> for potentially even better generations
Please join <a href="https://discord.gg/xBPBXfcFHd"><img alt="Join us on Discord" src="https://img.shields.io/discord/823813159592001537?color=5865F2&logo=discord&logoColor=white"></a> if you are interested in replicating this work in the open
<a href="https://www.youtube.com/watch?v=RYLomvaPWa4">AI Coffeebreak explanation</a>
## Appreciation
- <a href="https://stability.ai/">Stability.ai</a> for the generous sponsorship to work on cutting edge artificial intelligence research
- <a href="https://huggingface.co/">🤗 Huggingface</a> for their amazing transformers and accelerate library
- <a href="https://github.com/gmegh">Guillem</a> for his ongoing contributions
- You? If you are a great machine learning engineer and / or researcher, feel free to contribute to the frontier of open source generative AI
## Install
```bash
$ pip install phenaki-pytorch
```
## Usage
C-ViViT
```python
import torch
from phenaki_pytorch import CViViT, CViViTTrainer
cvivit = CViViT(
dim = 512,
codebook_size = 65536,
image_size = 256,
patch_size = 32,
temporal_patch_size = 2,
spatial_depth = 4,
temporal_depth = 4,
dim_head = 64,
heads = 8
).cuda()
trainer = CViViTTrainer(
cvivit,
folder = '/path/to/images/or/videos',
batch_size = 4,
grad_accum_every = 4,
train_on_images = False, # you can train on images first, before fine tuning on video, for sample efficiency
use_ema = False, # recommended to be turned on (keeps exponential moving averaged cvivit) unless if you don't have enough resources
num_train_steps = 10000
)
trainer.train() # reconstructions and checkpoints will be saved periodically to ./results
```
Phenaki
```python
import torch
from phenaki_pytorch import CViViT, MaskGit, Phenaki
cvivit = CViViT(
dim = 512,
codebook_size = 65536,
image_size = (256, 128), # video with rectangular screen allowed
patch_size = 32,
temporal_patch_size = 2,
spatial_depth = 4,
temporal_depth = 4,
dim_head = 64,
heads = 8
)
cvivit.load('/path/to/trained/cvivit.pt')
maskgit = MaskGit(
num_tokens = 5000,
max_seq_len = 1024,
dim = 512,
dim_context = 768,
depth = 6,
)
phenaki = Phenaki(
cvivit = cvivit,
maskgit = maskgit
).cuda()
videos = torch.randn(3, 3, 17, 256, 128).cuda() # (batch, channels, frames, height, width)
mask = torch.ones((3, 17)).bool().cuda() # [optional] (batch, frames) - allows for co-training videos of different lengths as well as video and images in the same batch
texts = [
'a whale breaching from afar',
'young girl blowing out candles on her birthday cake',
'fireworks with blue and green sparkles'
]
loss = phenaki(videos, texts = texts, video_frame_mask = mask)
loss.backward()
# do the above for many steps, then ...
video = phenaki.sample(texts = 'a squirrel examines an acorn', num_frames = 17, cond_scale = 5.) # (1, 3, 17, 256, 128)
# so in the paper, they do not really achieve 2 minutes of coherent video
# at each new scene with new text conditioning, they condition on the previous K frames
# you can easily achieve this with this framework as so
video_prime = video[:, :, -3:] # (1, 3, 3, 256, 128) # say K = 3
video_next = phenaki.sample(texts = 'a cat watches the squirrel from afar', prime_frames = video_prime, num_frames = 14) # (1, 3, 14, 256, 128)
# the total video
entire_video = torch.cat((video, video_next), dim = 2) # (1, 3, 17 + 14, 256, 128)
# and so on...
```
Or juExcerpt of 15,696 characters
Read on GitHub
Phil Wang · United States
142
5
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a1e63f14f756ef21, topic:deep-learning
matched fp:a1e63f14f756ef21, topic:text-to-video