Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code and Pretrained Models for Interspeech 2023 Paper "Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong Audio Event Taggers"
| Date | Stars |
|---|---|
| 2026-07-24 | 421 |
| 2026-07-25 | 421 |
| 2026-07-28 | 421 |
| 2026-07-30 | 421 |
| 2026-08-06 | 421 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong Audio Event Taggers
- [Introduction](#introduction)
- [Citation](#citation)
- [For Applications](#for-applications)
- [For Research](#for-research)
- [Available Models and Audio Tagging Performance](#available-models-and-audio-tagging-performance)
- [License](#license)
- [Contact](#contact)
## Introduction
[**[Paper]**](https://www.isca-archive.org/interspeech_2023/gong23d_interspeech.pdf)
[**[HuggingFace Space]**](https://huggingface.co/spaces/yuangongfdu/whisper-at) (Try Whisper-AT without Coding!)
[**[Colab Demo]**](https://colab.research.google.com/drive/1BbOGWCMjkOlOY5PbEMGk5RomRSqMcy_Q?usp=sharing)
[**[Local Notebook Demo]**(for user without Colab access)](https://github.com/YuanGongND/whisper-at/blob/main/sample/whisper_at_demo.ipynb)
[**[Python Package]**](https://pypi.org/project/whisper-at/) [](https://pepy.tech/project/whisper-at)
<p align="center"><img src="https://github.com/YuanGongND/whisper-at/blob/main/poster.png?raw=true" alt="Illustration of Whisper-AT." width="800"/></p>
<div align="center">
(Please turn on audio to listen to the sounds)
<video src="https://github.com/YuanGongND/whisper-at/assets/17163494/b479320a-b7f7-4bfc-acba-087b447623bd" width="400" />
</div>
Whisper-AT is a joint audio tagging and speech recognition model. It inherits strong speech recognition ability from [OpenAI Whisper](https://github.com/openai/whisper), and its ASR performance is exactly the same as the original Whisper.
The API interface and usage are also identical to the original OpenAI Whisper, so users can seamlessly switch from the original Whisper to Whisper-AT.
The advantage of Whisper-AT is that with minimal (less than 1%**) additional computational cost, Whisper-AT outputs general audio event labels ([527-class AudioSet labels](https://github.com/YuanGongND/whisper-at/blob/main/audioset_label.csv)) in desired temporal resolution in addition to the ASR transcripts. This makes audio tagging much easier and faster than using a standalone audio tagging model.
Internally, Whisper-AT freezes all original Whisper parameters, and trains a Time- and Layer-wise Transformer (TL-TR) on top of the Whisper encoder representations for the audio tagging task.
To help better understand the pros and cons of this work, we have attached the anonymous reviews and our responses [[here]](https://github.com/YuanGongND/whisper-at/tree/main/review). We thank the anonymous reviewers' invaluable comments.
** *Not for all models, see the paper for details.*
<hr style="border: 0; height: 1px; background-color: #e0e0e0;">
***Quick Start (Run in 8 lines of code)***
In shell,
```
pip install whisper-at
```
For Mac/Windows users, there is a known bug, please use the following workaround:
```bash
# install all dependencies except triton
pip install numba numpy torch tqdm more-itertools tiktoken==0.3.3
# install whisper-at without any dependency
pip install --no-deps whisper-at
```
Then, in Python,
```python3
import whisper_at as whisper
audio_tagging_time_resolution = 10
model = whisper.load_model("large-v1")
result = model.transcribe("audio.mp3", at_time_res=audio_tagging_time_resolution)
# ASR Results
print(result["text"])
# Audio Tagging Results
audio_tag_result = whisper.parse_at_label(result, language='follow_asr', top_k=5, p_threshold=-1, include_class_list=list(range(527)))
print(audio_tag_result)
```
## Citation
Please cite our Interspeech 2023 paper if you find this repository useful.
```
@inproceedings{gong_whisperat,
author={Gong, Yuan and Khurana, Sameer and Karlinsky, Leonid and Glass, James},
title={Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong Audio Event Taggers},
year=2023,
booktitle={Proc. Interspeech 2023}
}
```
## For Applications
**The best way to learn how to use Whisper-AT is this [[**Colab Tutorial**]](https://colab.research.google.com/drive/Excerpt of 19,772 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:80f3bf6dec92dc9a, topic:speech-recognition, topic:audio-processing, readme:speech recognition