Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Official implementations for paper: DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models
| Date | Stars |
|---|---|
| 2026-07-24 | 1789 |
| 2026-07-25 | 1789 |
| 2026-07-28 | 1789 |
| 2026-07-30 | 1789 |
| 2026-08-06 | 1789 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h2 align="center">DreamTalk: When Expressive Talking Head Generation <br> Meets Diffusion Probabilistic Models</h2> <p align="center"> <a href='https://dreamtalk-project.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> <a href='https://arxiv.org/abs/2312.09767'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://youtu.be/VF4vlE6ZqWQ'><img src='https://badges.aleen42.com/src/youtube.svg'></a> </p>  DreamTalk is a diffusion-based audio-driven expressive talking head generation framework that can produce high-quality talking head videos across diverse speaking styles. DreamTalk exhibits robust performance with a diverse array of inputs, including songs, speech in multiple languages, noisy audio, and out-of-domain portraits. ## News - __[2023.12]__ Release inference code and pretrained checkpoint. ## Installation ``` conda create -n dreamtalk python=3.7.0 conda activate dreamtalk pip install -r requirements.txt conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge conda update ffmpeg pip install urllib3==1.26.6 pip install transformers==4.28.1 pip install dlib ``` ## Download Checkpoints In light of the social impact, we have ceased public download access to checkpoints. If you want to obtain the checkpoints, please request it by emailing [email protected] . It is important to note that sending this email implies your consent to use the provided method **solely for academic research purposes**. Put the downloaded checkpoints into `checkpoints` folder. ## Inference Run the script: ``` python inference_for_demo_video.py \ --wav_path data/audio/acknowledgement_english.m4a \ --style_clip_path data/style_clip/3DMM/M030_front_neutral_level1_001.mat \ --pose_path data/pose/RichardShelby_front_neutral_level1_001.mat \ --image_path data/src_img/uncropped/male_face.png \ --cfg_scale 1.0 \ --max_gen_len 30 \ --output_name acknowledgement_english@M030_front_neutral_level1_001@male_face ``` `wav_path` specifies the input audio. The input audio file extensions such as wav, mp3, m4a, and mp4 (video with sound) should all be compatible. `style_clip_path` specifies the reference speaking style and `pose_path` specifies head pose. They are 3DMM parameter sequences extracted from reference videos. You can follow [PIRenderer](https://github.com/RenYurui/PIRender) to extract 3DMM parameters from your own videos. Note that the video frame rate should be 25 FPS. Besides, videos used for head pose reference should be first cropped to $256\times256$ using scripts in [FOMM video preprocessing](https://github.com/AliaksandrSiarohin/video-preprocessing). `image_path` specifies the input portrait. Its resolution should be larger than $256\times256$. Frontal portraits, with the face directly facing forward and not tilted to one side, usually achieve satisfactory results. The input portrait will be cropped to $256\times256$. If your portrait is already cropped to $256\times256$ and you want to disable cropping, use option `--disable_img_crop` like this: ``` python inference_for_demo_video.py \ --wav_path data/audio/acknowledgement_chinese.m4a \ --style_clip_path data/style_clip/3DMM/M030_front_surprised_level3_001.mat \ --pose_path data/pose/RichardShelby_front_neutral_level1_001.mat \ --image_path data/src_img/cropped/zp1.png \ --disable_img_crop \ --cfg_scale 1.0 \ --max_gen_len 30 \ --output_name acknowledgement_chinese@M030_front_surprised_level3_001@zp1 ``` `cfg_scale` controls the scale of classifer-free guidance. It can adjust the intensity of speaking styles. `max_gen_len` is the maximum video generation duration, measured in seconds. If the input audio exceeds this length, it will be truncated. The generated video will be named `$(output_name).mp4` and put in the output_video folder. Intermediate results, including the cropped portrait, will be in the `tmp/$(output_name)` folder.
Excerpt of 6,612 characters
Read on GitHub
7
4
Ikko Eltociear Ashimine · Japan
1
Alibaba OSS · Alibaba Group Holding Limited · China
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:5435a4d4d3838dfd, topic:video-generation, topic:talking-head, readme:video generation