Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code and Pretrained Models for ICLR 2023 Paper "Contrastive Audio-Visual Masked Autoencoder".
| Date | Stars |
|---|---|
| 2026-07-24 | 292 |
| 2026-07-25 | 292 |
| 2026-07-28 | 292 |
| 2026-07-30 | 292 |
| 2026-08-06 | 292 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Contrastive Audio-Visual Masked Autoencoder
- [Introduction](#introduction)
- [Citing](#citing)
- [What's in this repo?](#whats-in-this-repo)
- [The CAV-MAE Model](#the-cav-mae-model)
- [Data Preparation](#data-preparation)
- [Step 1. Extract audio track and image frames from the video](#step-1-extract-audio-track-and-image-frames-from-the-video)
- [Step 2. Build a label set and json file for your dataset.](#step-2-build-a-label-set-and-json-file-for-your-dataset)
- [CAV-MAE Pretraining](#cav-mae-pretraining)
- [Adapt vision-MAE checkpoint](#adapt-vision-mae-checkpoint)
- [Build a virtual environment and install packages](#build-a-virtual-environment-and-install-packages)
- [Run CAV-MAE pretraining](#run-cav-mae-pretraining)
- [Do additional pretraining based on AS-2M pretrained CAV-MAE](#do-additional-pretraining-based-on-as-2m-pretrained-cav-mae)
- [Audio-Visual Event Classification](#audio-visual-event-classification)
- [AudioSet](#audioset)
- [VGGSound](#vggsound)
- [Retrieval](#retrieval)
- [Inpainting](#inpainting)
- [Pretrained Models](#pretrained-models)
- [CAV-MAE Pretrained Models (Main)](#cav-mae-pretrained-models-main)
- [CAV-MAE Pretrained Models (Ablation Study)](#cav-mae-pretrained-models-ablation-study)
- [CAV-MAE Pretrained+Finetuned Models](#cav-mae-pretrainedfinetuned-models)
- [AudioSet and VGGSound Data Lists](#audioset-and-vggsound-data-lists)
- [Contact](#contact)
## Introduction
<p align="center"><img src="https://github.com/YuanGongND/cav-mae/blob/master/CAV-MAE_Poster.png?raw=true?raw=true" alt="Illustration of CAV-MAE." width="900"/></p>
**[[Paper]](https://openreview.net/pdf?id=QPtMRyk5rb)** **[[Review]](https://openreview.net/forum?id=QPtMRyk5rb)** **[[5-Minute Video]](https://recorder-v3.slideslive.com/?share=80147&s=29e0bafa-9193-4971-ac6f-e5a4cf7d69a0)** **[[Slides]](https://docs.google.com/presentation/d/1l-cofX9liikkVG6TkH2XwboYLVJoUDhpcyyq-fqtfcM/edit?usp=sharing)**
**[[MIT News](https://news.mit.edu/2023/scaling-audio-visual-learning-without-labels-0605)]**
This repository contains the official implementation (in PyTorch) of the **Contrastive Audio-Visual Masked Autoencoder (CAV-MAE)** proposed in the ICLR 2023 paper [Contrastive Audio-Visual Masked Autoencoder](https://openreview.net/forum?id=QPtMRyk5rb) (Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James Glass).
CAV-MAE **combines** two major self-supervised learning frameworks: **contrastive learning** and **masked data modeling**, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation.
CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task.
**Reviews:** The reviews of this paper and our responses are on [OpenReview](https://openreview.net/forum?id=QPtMRyk5rb), we thank the chair and anonymous reviewers' invaluable comments.
[](https://paperswithcode.com/sota/multi-modal-classification-on-audioset?p=contrastive-audio-visual-masked-autoencoder)
[](https://paperswithcode.com/sota/multi-modal-classification-on-vgg-sound?p=contrastive-audio-visual-masked-autoencoder)
## Citing
Please cite our paper if you find this repository useful.
```
@inproceedings{gong2023contrastive,
title={Contrastive Audio-Visual Masked Autoencoder},
author={Yuan Gong and AExcerpt of 27,597 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6534263dfcfa3b52, topic:computer-vision
matched fp:6534263dfcfa3b52, topic:audio-processing
matched fp:6534263dfcfa3b52, topic:multimodal