Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.
| Date | Stars |
|---|---|
| 2026-07-24 | 325 |
| 2026-07-25 | 325 |
| 2026-07-28 | 325 |
| 2026-07-30 | 325 |
| 2026-08-06 | 325 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation
Official pytorch implementation of the following paper:
<p align="left">
<a href="https://arxiv.org/abs/2406.09399">OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation</a>.
<br>
<br>
<a href="https://www.wangjunke.info/">Junke Wang</a><sup>1,2</sup>, <a href="https://enjoyyi.github.io/">Yi Jiang</a><sup>3</sup>, <a href="https://shallowyuan.github.io/">Zehuan Yuan</a><sup>3</sup>, <a href="./">Binyue Peng</a><sup>3</sup>, <a href="https://zxwu.azurewebsites.net/">Zuxuan Wu</a><sup>1,2</sup>, <a href="https://fvl.fudan.edu.cn/">Yu-Gang Jiang</a><sup>1,2</sup>
<br>
<sup>1</sup>Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University <br>
<sup>2</sup>Shanghai Collaborative Innovation Center of Intelligent Visual Computing, <sup>3</sup>Bytedance Inc.
</p>
<p align="left">
<img src=assets/network.png width="852" height="284" />
</p>
We introduce OmniTokenizer, a joint image-video tokenizer which features the following properties:
- 🚀 **One model** and **one weight** for joint image and video tokenization;
- 🥇 **State-of-the-art reconstruction performance** on both image and video datasets;
- ⚡ High adaptability to **high resolution** and **long** video inputs;
- 🔥 Equipped with it, both **language model** and **diffusion model** could achieve competitive visual generation results.
Please refer to our [project page](https://www.wangjunke.info/OmniTokenizer/) for the reconstruction and generation results by OmniTokenizer.
## Setup
Please setup the environment using the following commands:
```
pip3 install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu118
pip3 install -r requirements.txt
```
Then download the datasets from the official websites. You can download the [annotation.zip](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/annotations.zip) processed by us and put them under ```./annotations```.
## Model Zoo for VQVAE and VAE
We release both VQVAE and VAE version of OmniTokenizer, that are pretrained on a wide range of image and video datasets:
| Type | Training Data | FID | FVD | ckpt |
| ---------- | ---------- | ---------- | ----------- | ----------- |
| VQVAE | ImageNet | 1.28[^1] | - | [imagenet_only.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/imagenet_only.ckpt) |
| VQVAE | CelebAHQ | 1.85 | - | [celebahq.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/celebahq.ckpt) |
| VQVAE | FFHQ |2.58 | - | [ffhq.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/ffhq.ckpt) |
| VQVAE | ImageNet + UCF | 1.11 | 42.35 | [imagenet_ucf.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/imagenet_ucf.ckpt) |
| VQVAE | ImageNet + K600 | 1.23 | 25.97 | [imagenet_k600.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/imagenet_k600.ckpt) |
| VQVAE | ImageNet + MiT | 1.26 | 19.87 | [imagenet_mit.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/imagenet_mit.ckpt) |
| VQVAE | ImageNet + Sthv2 | 1.21 | 20.30 | [imagenet_sthv2.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/imagenet_sthv2.ckpt) |
| VQVAE | CelebAHQ + UCF | 1.93 | 45.59 | [celebahq_ucf.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/celebahq_ucf.ckpt) |
| VQVAE | CelebAHQ + K600 | 1.82 | 89.13 | [celebahq_k600.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/celebahq_k600.ckpt) |
| VQVAE | FFHQ + UCF | 1.91 | 57.93 | [ffhq_ucf.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/ffhq_ucf.ckpt) |
| VQVAE | FFHQ + K600 | 2.69 | 87.58 | [ffhq_k600.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/ffhq_k600.ckpt) |
| VAE | ImageNet + UCF | 0.69 | 23.44 | [imagenet_ucf_vae.ckpt](https://huggingface.co/Daniel0724/OmniTokenizer/resolve/main/imagenet_ucf_vae.ckpt) |
| VAE | ImageNet + K60Excerpt of 7,343 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ab7bce47b5868a09, topic:image-generation, readme:diffusion model
matched fp:ab7bce47b5868a09, topic:video-generation