Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Fine-tune and evaluate Whisper models for Automatic Speech Recognition (ASR) on custom datasets or datasets from huggingface.
| Date | Stars |
|---|---|
| 2026-07-24 | 365 |
| 2026-07-25 | 365 |
| 2026-07-28 | 365 |
| 2026-07-30 | 365 |
| 2026-08-04 | 365 |
| 2026-08-06 | 365 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Fine-tuning and evaluating Whisper models for Automatic Speech Recognition This repository contains the relevant scripts to fine-tune and evaluate Whisper models of various configurations available over huggingface 🤗. Scripts in this repository support fine-tuning these models using custom datasets which haven't been made available over huggingface. Some of the models trained and evaluated using these scripts can be found [here on huggingface](https://huggingface.co/vasista22). Code snippets have been made available to extract relevant embeddings from different layers of whisper models with varied configurations. ## Contents - [Setup](#setup) - [Data Preparation for custom datasets](#data-preparation-for-custom-datasets) - [Hyperparameter tuning](#hyperparameter-tuning) - [Fine-tune on a dataset from huggingface](#fine-tune-on-a-dataset-from-huggingface) - [Fine-tune on a custom dataset](#fine-tune-on-a-custom-dataset) - [Evaluate on a dataset from huggingface](#evaluate-on-a-dataset-from-huggingface) - [Evaluate on a custom dataset](#evaluate-on-a-custom-dataset) - [Transcribe a single audio file](#transcribe-a-single-audio-file) - [Faster evaluation with whisper-jax](#faster-evaluation-with-whisper-jax) - [Extract embeddings from whisper models](#extract-embeddings-from-whisper-models) - [Interesting works around Whisper](#interesting-works-around-whisper) ## Setup These scripts have been tested with Python 3.8 and cuda 11.3. It is recommended that you setup a virtual environment for the installation purpose and work within the same. The following set of commands would setup a virtual environment and complete the installation: ```bash python3 -m venv env_whisper-finetune source env_whisper-finetune/bin/activate python3 -m pip install --upgrade pip pip install -r requirements.txt ``` In order to push your model to huggingface, you would need to login using the command line interface. Also, `git-lfs` would need to be installed to push large model files. Executing the following commands should help in this regard: ```bash sudo apt-get install git-lfs huggingface-cli login ``` ## Data Preparation for custom datasets **NOTE:** The contents of this section are relevant only if you are interested in using dataset(s) that aren't available over huggingface. You may proceed to the later sections of this README if this isn't applicable to your usecase. One could be interested in working with a dataset that isn't available over huggingface. To fine-tune whisper models or evaluate them on such datasets, a preliminary data preparation is needed to make them compatible with the huggingface's sequence-to-sequence training pipeline. The script which converts the dataset into the required format, expects two files named `text` and `audio_paths`. The `audio_paths` file is expected to contain the absolute paths to each of the audio files to be used in the fine-tuning or evaluation process. Also, each entry in the file has to be indexed by a unique utterance ID. The contents of the file should be organized in the following manner. ```bash <unique-id> <absolute path to the audio file-1> <unique-id> <absolute path to the audio file-2> ... <unique-id> <absolute path to the audio file-N> ``` The `text` file is expected to contain the transcriptions corresponding to each of the audio files mentioned in the `audio_paths` file. Also, each entry in the file has to be indexed by a unique utterance ID. The ordering of unique utterance IDs in both the `text` and `audio_paths` files should be consistent. The contents of the `text` file should be organized in the following manner. ```bash <unique-id> <Transcription (ground truth) corresponding to the audio file-1> <unique-id> <Transcription (ground truth) corresponding to the audio file-2> ... <unique-id> <Transcription (ground truth) corresponding to the audio file-N> ``` The `sample_data` folder of this repository provides a reference on how these two files are to be organized. Once
Excerpt of 24,971 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:76ae9ac503c160b9, topic:speech-recognition, topic:asr, topic:whisper
matched fp:76ae9ac503c160b9, topic:pytorch, topic:jax