Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICCV2023 Oral] Unmasked Teacher: Towards Training-Efficient Video Foundation Models
| Date | Stars |
|---|---|
| 2026-07-31 | 348 |
| 2026-08-06 | 348 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Unmasked Teacher
This repo is the official implementation of ["Unmasked Teacher: Towards Training-Efficient Video Foundation Models"](https://arxiv.org/abs/2303.16058).
By [Kunchang Li](https://scholar.google.com/citations?user=D4tLSbsAAAAJ), [Yali Wang](https://scholar.google.com/citations?user=hD948dkAAAAJ), [Yizhuo Li](https://scholar.google.com/citations?user=pyBSGjgAAAAJ), [Yi Wang](https://scholar.google.com.hk/citations?hl=zh-CN&user=Xm2M8UwAAAAJ), [Yinan He](https://dblp.org/pid/93/7763.html), [Limin Wang](https://scholar.google.com/citations?user=HEuN8PcAAAAJ) and [Yu Qiao](https://scholar.google.com/citations?user=gFtI-8QAAAAJ&hl).
## Update
- :star_struck: **2023/11/06**: Glad to see that **UMTScore’s ranking of video-text alignment is most consistent with humans**. Check [FETV](https://github.com/llyx97/FETV).
- :1st_place_medal: **2023/10/01**: **Get Champion with unmasked teacher on [Perception Test Challenge](https://ptchallenge-workshop.github.io/), ICCV 2023.** Check our [solutions](https://github.com/OpenGVLab/perception_test_iccv2023).
- :warning: **2023/09/25**: **Bug for MSVD retrieval.** Check https://github.com/OpenGVLab/unmasked_teacher/issues/12. Results have been updated.
- :rocket: **2023/09/20**: **Fix bug in UMT pretraining.** Add autocast for teacher forward, which can halve the pretraining time.
- :fire: **2023/07/19**: **All the code and models are released.**
- [single_modality](./single_modality/): Single-modality pretraining and finetuning.
- Action Classification: [Kinetics](https://www.deepmind.com/open-source/kinetics), [Moments in Time](http://moments.csail.mit.edu/), [Something-Something](https://developer.qualcomm.com/software/ai-datasets/something-something).
- Action Detection: [AVA](http://research.google.com/ava/).
- **The models and scripts are in [MODEL_ZOO](./single_modality/MODEL_ZOO.md). Have a try!**
- [multi_modality](./multi_modality/): Multi-modality pretraining and finetuning.
- Video-Text Retrieval: [MSRVTT](https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/), [DiDeMo](https://github.com/LisaAnne/TemporalLanguageRelease), [ActivityNet](http://activity-net.org/), [LSMDC](https://sites.google.com/site/describingmovies/), [MSVD](https://www.cs.utexas.edu/users/ml/clamp/videoDescription/), [Something-Something](https://github.com/jayleicn/singularity).
- Video Question Answering: [ActivityNet-QA](https://github.com/MILVLG/activitynet-qa), [MSRVTT-QA](https://github.com/xudejing/video-question-answering), [MSRVTT-MC](https://github.com/yj-yu/lsmdc), [MSVD-QA](https://github.com/xudejing/video-question-answering).
- **The models and scripts are in [MODEL_ZOO](./multi_modality/MODEL_ZOO.md). Have a try!**
- :bowing_man: We are hiring researchers, engineers and interns in **General Vision Group, Shanghai AI Lab**. If you are interested in working with us, please contact [Yi Wang](https://shepnerd.github.io/) (`[email protected]`).
- **2023/07/14**: Unmasked Teacher is accpeted by ICCV2023! 🎉🎉
- **2023/03/17**: We gave a blog in Chinese [Zhihu](https://zhuanlan.zhihu.com/p/618221217).
## Introduction
Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our methodExcerpt of 10,968 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:72b10b58cc1d7727, llm:Repository title and description: 'Unmasked Teacher: Towards Training-Efficient Video Foundation Models' (ICCV2023 Oral). Python implementation for video foundation model training efficiency.
matched fp:72b10b58cc1d7727, llm:Repository title and description: 'Unmasked Teacher: Towards Training-Efficient Video Foundation Models' (ICCV2023 Oral). Python implementation for video foundation model training efficiency.
matched fp:72b10b58cc1d7727, llm:Repository title and description: 'Unmasked Teacher: Towards Training-Efficient Video Foundation Models' (ICCV2023 Oral). Python implementation for video foundation model training efficiency.