Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Official implementation for "Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model" (NeurIPS 2024)
| Date | Stars |
|---|---|
| 2026-08-03 | 257 |
| 2026-09-28 | 258 |
| 2026-10-18 | 258 |
| 2026-11-04 | 258 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
## ___***Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model***___
<!-- {: width="50%"} -->
<!--  -->
<div align="center">
<a href='https://arxiv.org/abs/2406.15735'><img src='https://img.shields.io/badge/arXiv-2406.15735-b31b1b.svg'></a>
<a href='https://cond-image-leak.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a>
<a href='https://zhuanlan.zhihu.com/p/705300028?utm_campaign=shareopn&utm_medium=social&utm_psn=1788953432507416577&utm_source=wechat_session'><img src='https://img.shields.io/badge/Zhihu-Page-Blue'></a>
<a href='https://huggingface.co/spaces/Xiang-cd/DynamiCrafter-CIL'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20DynamiCrafter%20CIL%20Hugging%20Face%20I2V-Demo-blue'></a>
_**[Min Zhao](https://gracezhao1997.github.io/), [Hongzhou Zhu](https://zhuhz22.github.io/), [Chendong Xiang](https://xiang-cd.github.io/), [Kaiwen Zheng](https://scholar.google.com/citations?user=0d80xSIAAAAJ&hl=en), [Chongxuan Li](https://zhenxuan00.github.io/), [Jun Zhu](https://ml.cs.tsinghua.edu.cn/~jun/software.shtml)**_
</div>
This is the official implementation for [Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model](https://neurips.cc/virtual/2024/poster/93640) (Accepted by **NeurIPS 2024**).
## 🔆 Overview
Diffusion models have obtained substantial progress in image-to-video (I2V) generation. However, such models are not fully understood. In this paper, we report a significant but previously overlooked issue in I2V diffusion models (I2V-DMs), namely, conditional image leakage. I2V-DMs tend to over-rely on the conditional image at large time steps, neglecting the crucial task of predicting the clean video from noisy inputs, which results in videos lacking dynamic and vivid motion. We further address this challenge from both inference and training aspects by presenting plug-and-play strategies accordingly. These strategies are validated on various I2V-DMs including [DynamiCrafter](https://doubiiu.github.io/projects/DynamiCrafter),
[SVD](https://github.com/Stability-AI/generative-models) and [VideoCrafter1](https://github.com/AILab-CVC/VideoCrafter).
## ✅ To do list:
- [x] __[2024.06.25]__: Release code, models and project page.
- [x] __[2024.08.11]__: Add inference code for Animate-Anything and PIA
## ⚙️ Setup Environment
Our plug-and-play strategies can be applied to various I2V-DMs, allowing the direct use of their original environments.
For example, to set up DynamiCrafter:
```
cd examples/DynamiCrafter
conda create -n dynamicrafter python=3.8.5
conda activate dynamicrafter
pip install -r requirements.txt
```
To set up VideoCrafter1:
```
cd examples/VideoCrafter
conda create -n videocrafter python=3.8.5
conda activate videocrafter
pip install -r requirements.txt
```
To set up SVD:
```
cd examples/SVD
conda create -n svd python=3.9.18
conda activate svd
pip install -r requirements.txt
```
To set up Animate-Anything:
```
cd examples/animate-anything
conda create -n animation python=3.10
conda activate animation
pip install -r requirements.txt
```
To set up PIA:
```
cd examples/PIA
conda env create -f pia.yml
conda activate pia
```
## ☀️ Dataset
Download the WebVid dataset from [here](https://github.com/m-bain/webvid), where we use Webvid-2M subset. Put ```.csv``` file in ```examples/dataset/results_2M_train.csv``` and video data in ```examples/dataset/```. **We use the raw data without any filters.**
## 🧊 Inference Strategy
We clone the repo of [DynamiCrafter](https://doubiiu.github.io/projects/DynamiCrafter) and [VideoCrafter1](https://github.com/AILab-CVC/VideoCrafter) and implement [SVD](https://github.com/Stability-AI/generative-models) by ourselves. We apply our plug and plug-and-play strategies on them.
### 🎒 Initial Noise Distribution
| Model Excerpt of 18,700 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:2ef92a2b2bf7a1b9, desc:diffusion model