This repository contains the code for the Open ASR Leaderboard. The leaderboard is a Gradio Space that allows users to compare the accuracy of ASR models on a variety of datasets. The leaderboard is hosted at hf-audio/open_asr_leaderboard.
The Open ASR Leaderboard evaluates models on a diverse set of publicly available ASR benchmarks hosted on the Hugging Face Hub. These datasets cover a wide range of domains, languages, and recording conditions to provide a fair and comprehensive comparison across models.
-
Main Test Sets (English, short-form): The main benchmark datasets used for evaluation (short-form English) are available here.
-
Chunked sets with session-level references: Earnings22-Cleaned-AA-chunked splits 6 earnings calls into 341 chunks, with one reference transcript per call in the parent Earnings22-Cleaned-AA set. Models transcribe each chunk as usual; the scorer concatenates a call's chunk predictions before computing WER against the call-level reference. Run with
--dataset_path ArtificialAnalysis/Earnings22-Cleaned-AA-chunked --dataset earnings22_cleaned_aa_chunked --split test. -
English, long-form: The ASR Longform benchmark dataset includes earnings21 and earnings22. We also evaluate on CORAAL, but it is stored as a separate dataset since it has multiple splits.
-
Multilingual Benchmark: The ASR Multilingual benchmark dataset includes FLEURS, MCV and MLS. For Hindi, data from Voice Arena is used. The Armenian FLEURS and MCV test sets are hosted under Metric-AI/open-asr-leaderboard-multilingual-datasets.
-
Private datasets: After submitting a model to the leaderboard, the maintainers will evaluate on private sets, as described here.
English short-form, English long-form, and multilingual evaluations use Hugging Face Jobs to guarantee reproducibility: every run executes a Docker image on the same hardware, to minimize environment and driver differences.
For API models see here.
Jobs are launched on the following hardware (flavor in HF Jobs terminology):
name pretty name cpu ram storage accelerator cost/min cost/hour
h200 Nvidia H200 23 vCPU 256 GB 3000 GB 1x H200 (141 GB) $0.0833 $5.00
Example costs for a full run over the main public datasets for English short-form:
- $2.92 for
nvidia/parakeet-tdt-0.6b-v3 - $4.75 for
openai/whisper-large-v3-turbo - $5.58 for
Qwen/Qwen3-ASR-1.7B
Each model family has its own Docker image with the necessary software requirements. The evaluation configurations are hosted as HF Spaces.
To launch an evaluation:
-
Hugging Face Hub setup
- Create an account at https://huggingface.co/ and add credits for HF Jobs: https://huggingface.co/settings/billing
- Create a WRITE token and copy it.
- Create a Storage Bucket to store results: https://huggingface.co/new-bucket
-
One-time local setup
A local setup is needed to launch the evaluation and score with the repo's normalizer.
# Clone the repository
git clone [email protected]:huggingface/open_asr_leaderboard.git
cd open_asr_leaderboard
# Create a minimal conda environment (no GPU required locally)
conda create -n leaderboard_jobs python=3.10 -y
conda activate leaderboard_jobs
pip install -r requirements/requirements_jobs.txt
huggingface-cli login # paste your WRITE token when prompted- Launch an evaluation 🚀
# Open the relevant submit_jobs script, uncomment the models/datasets you want, then run:
RESULTS_BUCKET="<your-bucket>" HF_TOKEN=hf_... bash qwen/submit_jobs.sh
# Jobs are submitted in parallel (one per dataset). The script waits for all
# jobs to finish, syncs results from the bucket, and prints a CSV summary.
# Billing to org
ORG_NAME="<org-name>" RESULTS_BUCKET="<your-bucket>" HF_TOKEN=hf_... bash qwen/submit_jobs.shFor long-form evaluations (Earnings21, Earnings22, and the eight CORAAL subsets),
use the submit_jobs_*_longform.sh script of the model's family, e.g.:
RESULTS_BUCKET="<your-bucket>" HF_TOKEN=hf_... bash transformers/submit_jobs_whisper_longform.shFor contributors who want to test locally, the requirements/ folder contains per-family dependency files. The Dockerfiles in the HF Spaces can also be used to build a local container.
Each model family has a run_eval.py entry point driven by a corresponding bash script (e.g. run_whisper.sh). The script outputs a JSONL file with predictions and prints WER and RTFx after completion. See the sub-folders of this repo for examples; the latest scripts are in the HF Spaces linked above.
Pareto plots of WER against RTFx and model size are available directly on the leaderboard.
The benchmark_fitting folder contains scripts to measure whether a model overfits reference transcripts in the test sets. See
benchmark_fitting/README.md for more information.
Please follow the pull request template; it contains a submission checklist and guidelines.
After a normalization change, existing JSONL manifests can be rescored without rerunning inference: manifests store the raw text and pred_text, and normalization is applied at scoring time.
# dry run over every row of every results file
python scripts/rescore_all.py
# some models / files only, keeping the updated CSVs locally for inspection
python scripts/rescore_all.py --models openai/whisper-large-v3 nvidia/canary-1b-v2 \
--targets english multilingual --language de --out_dir /tmp/rescored
# re-use results already synced locally, and open the PRs
python scripts/rescore_all.py --skip_sync --open_pr@misc{srivastav2025openasrleaderboardreproducible,
title={Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation},
author={Vaibhav Srivastav and Steven Zheng and Eric Bezzam and Eustache Le Bihan and Nithin Koluguri and Piotr Żelasko and Somshubra Majumdar and Adel Moumen and Sanchit Gandhi},
year={2025},
eprint={2510.06961},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2510.06961},
}