Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[CVPR 2025] MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
| Date | Stars |
|---|---|
| 2026-07-30 | 2246 |
| 2026-08-10 | 2257 |
| 2026-08-18 | 2258 |
| 2026-08-19 | 2259 |
| 2026-08-20 | 2260 |
| 2026-08-21 | 2262 |
| 2026-08-23 | 2263 |
| 2026-08-24 | 2263 |
| 2026-08-27 | 2264 |
| 2026-08-28 | 2264 |
| 2026-08-29 | 2264 |
| 2026-08-30 | 2265 |
| 2026-08-31 | 2264 |
| 2026-09-01 | 2263 |
| 2026-09-02 | 2264 |
| 2026-09-04 | 2265 |
| 2026-09-07 | 2266 |
| 2026-09-08 | 2267 |
| 2026-09-09 | 2268 |
| 2026-09-13 | 2267 |
| 2026-09-16 | 2268 |
| 2026-09-17 | 2269 |
| 2026-09-18 | 2268 |
| 2026-09-19 | 2269 |
| 2026-09-24 | 2266 |
| 2026-09-25 | 2267 |
| 2026-09-28 | 2268 |
| 2026-09-29 | 2269 |
| 2026-10-01 | 2270 |
| 2026-10-06 | 2269 |
| 2026-10-07 | 2272 |
| 2026-10-08 | 2273 |
| 2026-10-09 | 2276 |
| 2026-10-11 | 2277 |
| 2026-10-12 | 2280 |
| 2026-10-14 | 2282 |
| 2026-10-15 | 2286 |
| 2026-10-16 | 2287 |
| 2026-10-18 | 2288 |
| 2026-10-19 | 2289 |
| 2026-10-20 | 2290 |
| 2026-10-21 | 2289 |
| 2026-10-22 | 2291 |
| 2026-10-23 | 2291 |
| 2026-10-24 | 2292 |
| 2026-10-25 | 2294 |
| 2026-10-26 | 2296 |
| 2026-10-27 | 2299 |
| 2026-10-28 | 2299 |
| 2026-10-29 | 2299 |
| 2026-10-30 | 2301 |
| 2026-10-31 | 2302 |
| 2026-11-01 | 2303 |
| 2026-11-02 | 2303 |
| 2026-11-04 | 2303 |
Today
— stars today
This week
+4 stars this week
This month
+33 stars this month
Momentum
0.0
growth rate 0.17%/day
<div align="center"> <p align="center"> <h2>MMAudio</h2> <a href="https://arxiv.org/abs/2412.15322">Paper</a> | <a href="https://hkchengrex.github.io/MMAudio">Webpage</a> | <a href="https://huggingface.co/hkchengrex/MMAudio/tree/main">Models</a> | <a href="https://huggingface.co/spaces/hkchengrex/MMAudio"> Huggingface Demo</a> | <a href="https://colab.research.google.com/drive/1TAaXCY2-kPk4xE4PwKB3EqFbSnkUuzZ8?usp=sharing">Colab Demo</a> | <a href="https://replicate.com/zsxkib/mmaudio">Replicate Demo</a> </p> </div> ## [Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://hkchengrex.github.io/MMAudio) [Ho Kei Cheng](https://hkchengrex.github.io/), [Masato Ishii](https://scholar.google.co.jp/citations?user=RRIO1CcAAAAJ), [Akio Hayakawa](https://scholar.google.com/citations?user=sXAjHFIAAAAJ), [Takashi Shibuya](https://scholar.google.com/citations?user=XCRO260AAAAJ), [Alexander Schwing](https://www.alexander-schwing.de/), [Yuki Mitsufuji](https://www.yukimitsufuji.com/) University of Illinois Urbana-Champaign, Sony AI, and Sony Group Corporation CVPR 2025 ## Highlight MMAudio generates synchronized audio given video and/or text inputs. Our key innovation is multimodal joint training which allows training on a wide range of audio-visual and audio-text datasets. Moreover, a synchronization module aligns the generated audio with the video frames. Check out this fun video: [](https://youtu.be/SLz3NWLyHxg) [[Does Your Voice Match Your Face? https://youtu.be/SLz3NWLyHxg]](https://youtu.be/SLz3NWLyHxg) ## Results (All audio from our algorithm MMAudio) Videos from Sora: https://github.com/user-attachments/assets/82afd192-0cee-48a1-86ca-bd39b8c8f330 Videos from Veo 2: https://github.com/user-attachments/assets/8a11419e-fee2-46e0-9e67-dfb03c48d00e Videos from MovieGen/Hunyuan Video/VGGSound: https://github.com/user-attachments/assets/29230d4e-21c1-4cf8-a221-c28f2af6d0ca For more results, visit https://hkchengrex.com/MMAudio/video_main.html. ## Installation We have only tested this on Ubuntu. ### Prerequisites We recommend using a [miniforge](https://github.com/conda-forge/miniforge) environment. - Python 3.9+ - PyTorch **2.5.1+** and corresponding torchvision/torchaudio (pick your CUDA version https://pytorch.org/, pip install recommended) <!-- - ffmpeg<7 ([this is required by torchaudio](https://pytorch.org/audio/master/installation.html#optional-dependencies), you can install it in a miniforge environment with `conda install -c conda-forge 'ffmpeg<7'`) --> **1. Install prerequisite if not yet met:** ```bash pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --upgrade ``` (Or any other CUDA versions that your GPUs/driver support) <!-- ``` conda install -c conda-forge 'ffmpeg<7 ``` (Optional, if you use miniforge and don't already have the appropriate ffmpeg) --> **2. Clone our repository:** ```bash git clone https://github.com/hkchengrex/MMAudio.git ``` **3. Install with pip (install pytorch first before attempting this!):** ```bash cd MMAudio pip install -e . ``` (If you encounter the File "setup.py" not found error, upgrade your pip with pip install --upgrade pip) **Pretrained models:** The models will be downloaded automatically when you run the demo script. MD5 checksums are provided in `mmaudio/utils/download_utils.py`. The models are also available at https://huggingface.co/hkchengrex/MMAudio/tree/main See [MODELS.md](docs/MODELS.md) for more details. ## Demo By default, these scripts use the `large_44k_v2` model. In our experiments, inference only takes around 6GB of GPU memory (in 16-bit mode) which should fit in most modern GPUs. ### Command-line interface With `demo.py` ```bash python demo.py --duration=8 --video=<path to video> --prompt "your prompt" ``` The output (audio in `.flac` format, and video in `.mp4` format) will be sav
Excerpt of 9,692 characters
Read on GitHub
Rex Cheng · Israel
124
2
cocktailpeanut
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:df0e55aeaf3ff832, topic:deep-learning
matched fp:df0e55aeaf3ff832, topic:computer-vision