Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Efficient Training of Audio Transformers with Patchout
| Date | Stars |
|---|---|
| 2026-07-24 | 386 |
| 2026-07-25 | 386 |
| 2026-07-28 | 386 |
| 2026-07-30 | 386 |
| 2026-08-06 | 386 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# PaSST: Efficient Training of Audio Transformers with Patchout
This is the implementation for [Efficient Training of Audio Transformers with Patchout](https://arxiv.org/abs/2110.05069)
Patchout significantly reduces the training time and GPU memory requirements to train transformers on audio spectrograms, while improving their performance.
<p align="center"><img src="https://github.com/kkoutini/PaSST/blob/main/.github/speed_mem_map.png?raw=true" width="600"/></p>
Patchout works by dropping out some of the input patches during training.
In either an unstructured way (randomly, similar to dropout),
or entire time-frames or frequency bins of the extracted patches (similar to SpecAugment),
which corresponds to rows/columns in step 3 of the figure below.
<p align="center"><img src="https://github.com/kkoutini/PaSST/raw/main/.github/passt_diag.png?raw=true" width="600"/></p>
## Table of contents
- [Pre-trained models for Inference and embeddings extractions](#pre-trained-models-for-inference-and-embeddings-extractions)
- [Getting the logits from the pretrained models](#getting-the-logits-from-the-pretrained-models)
- [Getting a pre-trained model for fine-tuning](#getting-a-pre-trained-model-for-fine-tuning)
- [Development environment](#development-environment)
- [Setting up the development experiments environment](#setting-up-the-development-experiments-environment)
- [Setting up using the exported conda environment](#setting-up-using-the-exported-conda-environment)
- [Checking the environment](#checking-the-environment)
- [Getting started](#getting-started)
- [General information](#general-information)
- [Configuring the experiment](#configuring-the-experiment)
- [Training on Audioset](#training-on-audioset)
- [Examples with Pre-trained models](#examples-with-pre-trained-models)
- [Examples fine-tuning on downstream datasets](#examples-of-fine-tuning-on-downstream-datasets)
- [Citation](#citation)
- [Contact](#contact)
## Pre-trained models for Inference and embeddings extractions
If you only want to use the embeddings generated by the pretrained models, use
your own fine-tuning framework, or you need it only for inference, you can find a stripped down version of this repo [here](https://github.com/kkoutini/passt_hear21).
The package follows [HEAR 2021 NeurIPS Challenge](https://neuralaudio.ai/hear2021-results.html) API, and can be installed:
```shell
pip install hear21passt
```
This repo is a complete framework for training the models and fine-tuning pre-trained models on Audioset on downstream tasks.
### Getting the logits from the pretrained models
```python
from hear21passt.base import get_basic_model,get_model_passt
import torch
# get the PaSST model wrapper, includes Melspectrogram and the default pre-trained transformer
model = get_basic_model(mode="logits")
print(model.mel) # Extracts mel spectrogram from raw waveforms.
print(model.net) # the transformer network.
# example inference
model.eval()
model = model.cuda()
with torch.no_grad():
# audio_wave has the shape of [batch, seconds*32000] sampling rate is 32k
# example audio_wave of batch=3 and 10 seconds
audio = torch.ones((3, 32000 * 10))*0.5
audio_wave = audio.cuda()
logits=model(audio_wave)
```
### Getting a pre-trained model for fine tuning
```python
from hear21passt.base import get_basic_model,get_model_passt
import torch
# get the PaSST model wrapper, includes Melspectrogram and the default pre-trained transformer
model = get_basic_model(mode="logits")
print(model.mel) # Extracts mel spectrogram from raw waveforms.
# optional replace the transformer with one that has the required number of classes i.e. 50
model.net = get_model_passt(arch="passt_s_swa_p16_128_ap476", n_classes=50)
print(model.net) # the transformer network.
# now model contains mel + the transformer pre-trained model ready to be fine tuned.
# It's still expecting input of the shape [batch, seconds*32000] sampling rate is 32k
model.train()
mExcerpt of 14,253 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:298609aefb5033f6, topic:pytorch
matched fp:298609aefb5033f6, topic:transformer