Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Official PyTorch Implementation of Unified Video Action Model (RSS 2025)
| Date | Stars |
|---|---|
| 2026-07-24 | 400 |
| 2026-07-25 | 400 |
| 2026-07-28 | 400 |
| 2026-07-30 | 400 |
| 2026-08-06 | 400 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Unified Video Action Model
[[Project page]](https://unified-video-action-model.github.io/)
[[Paper]](https://arxiv.org/abs/2503.00200)
[[Colab (PushT)]](https://colab.research.google.com/drive/1WQfiGat3pTr5DZqWa760zI3JrdH_M86O?usp=sharing)
[Shuang Li](https://shuangli59.github.io/),
[Yihuai Gao](https://yihuai-gao.github.io/),
[Dorsa Sadigh](https://dorsa.fyi/),
[Shuran Song](https://shurans.github.io/)
Stanford University
<img src="media/overview.png" alt="drawing" width="100%"/>
## 🛝 Try UVA on Colab
We provide a colab notebook for UVA on [PushT](https://colab.research.google.com/drive/1WQfiGat3pTr5DZqWa760zI3JrdH_M86O?usp=sharing) using the pretrained checkpoint.
## 🛠️ Installation
Install the conda environment:
```console
$ conda install mamba -c conda-forge
```
```console
$ mamba env create -f conda_environment.yml
```
## Simulation Experiments
### Testing
Download the pretrained checkpoints from the following links and put them in the `checkpoints/` folder.
* Checkpoint trained on [PushT](https://drive.google.com/file/d/1OduHcxfc2hqUYSccMQNf9g-vAt-q2UhF/view?usp=sharing)
* Checkpoint trained on [PushT-M](https://drive.google.com/file/d/1ZppZJyQdEdjhu8TIt4ddyaWy_mSdjoAZ/view?usp=sharing)
* Checkpoint trained on [Libero10](https://drive.google.com/file/d/11c2VrmaRp48yw__5A5xpcu8EPzkexHSi/view?usp=sharing)
```
pip install gdown
gdown 1OduHcxfc2hqUYSccMQNf9g-vAt-q2UhF -O checkpoints/pusht.ckpt
gdown 1ZppZJyQdEdjhu8TIt4ddyaWy_mSdjoAZ -O checkpoints/pusht_multitask.ckpt
gdown 11c2VrmaRp48yw__5A5xpcu8EPzkexHSi -O checkpoints/libero10.ckpt
```
```
CUDA_VISIBLE_DEVICES=0 python eval_sim.py --checkpoint checkpoints/pusht.ckpt --output_dir checkpoints/pusht
```
```
CUDA_VISIBLE_DEVICES=0 python eval_sim.py --checkpoint checkpoints/pusht_multitask.ckpt --output_dir checkpoints/pusht_multitask
```
```
CUDA_VISIBLE_DEVICES=0 python eval_sim.py --checkpoint checkpoints/libero10.ckpt --output_dir checkpoints/libero10
```
### Training
#### Download Pretrained Models
We start from a pretrained VAE model and a pretrained image generation model [MAR](https://github.com/LTH14/mar). Run the following command to download the pretrained models.
```
python unified_video_action/utils/download.py
```
#### Train Video Generation Model
We found that two-stage training works better than training on both video and action tasks directly. In the first stage, the model is trained on `video generation` task, and in the second stage, it is fine-tuned on both video and action tasks.
To train the UVA model for the video generation task, we set `predict_action=False` and `selected_training_mode=video_model`. We did not incorporate additional video data during training. We believe that pretraining the model on large-scale web video datasets could substantially improve its generalization capabilities, and we plan to explore this approach in future work.
UVA's performance may currently be constrained by the model size. To evaluate it on larger or more complex real-world tasks, please consider using a larger UVA model.
Training video and action model takes longer time than training policy model only. We recommend using at least 4 GPUs for training.
To train the UVA model on the PushT dataset, run the following command:
```
accelerate launch --num_processes=8 train.py \
--config-dir=. \
--config-name=uva_pusht.yaml \
model.policy.action_model_params.predict_action=False \
model.policy.selected_training_mode=video_model \
model.policy.optimizer.learning_rate=1e-4 \
logging.project=uva \
hydra.run.dir="checkpoints/uva_pusht_video_model"
```
#### Train Joint Video and Action Model
To train the UVA model on the joint video and action tasks, we set `predict_action=True` and remove `selected_training_mode=video_model`.
To train the UVA model on the UMI multi-task dataset, run the following command:
```
accelerate launch --num_processes=8 train.py \
--config-dir=. \
--config-name=uva_pusht.yaml \
model.pExcerpt of 15,032 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9eec9829a1d61a89, topic:video-generation, readme:video generation
matched fp:9eec9829a1d61a89, topic:robotics