Star history, last 90 days
Date	Stars
2026-07-30	396
2026-08-07	399
2026-08-15	452
2026-08-18	466
2026-08-19	468
2026-08-20	469
2026-08-22	471
2026-08-24	474
2026-08-25	475
2026-08-26	477
2026-08-27	478
2026-08-28	479
2026-09-03	481
2026-09-05	483
2026-09-06	484
2026-09-08	485
2026-09-09	485
2026-09-10	486
2026-09-11	487
2026-09-12	487
2026-09-13	488
2026-09-15	488
2026-09-16	488
2026-09-18	490
2026-09-19	491
2026-09-20	492
2026-09-22	493
2026-09-23	495
2026-09-24	494
2026-09-26	496
2026-09-28	498
2026-09-30	499
2026-10-01	500
2026-10-02	502
2026-10-04	503
2026-10-05	504
2026-10-06	505
2026-10-08	506
2026-10-09	507
2026-10-11	508
2026-10-12	510
2026-10-13	509
2026-10-14	513
2026-10-15	515
2026-10-17	516
2026-10-18	517
2026-10-19	518
2026-10-21	519
2026-10-22	522
2026-10-24	523
2026-10-25	525
2026-10-26	526
2026-10-27	528
2026-10-29	530
2026-10-30	531
2026-10-31	534
2026-11-01	536
2026-11-03	537
2026-11-04	536

Awesome TTS & Voice Generation Models

A curated list of open-source Text-to-Speech (TTS) and voice cloning models. Models are sorted by release date (newest first).

Text-to-Speech (TTS) Models
Anything to Audio
Audio Restoration & Enhancement
Speech Recognition (ASR)
Audio Codecs & Tokenizers
Audio Transcription Models
Additional Resources

Text-to-Speech (TTS) Models

TTS Quick Comparison

Model	Voice Cloning	ASR	Languages	Streaming
Iroha-TTS-V1	✅	❌	Japanese	❌
OGG TTS	✅	❌	8	✅
Ming-3.0-tts-0.5B	✅	❌	7	—
Fish Audio Drama 3	✅	❌	83	✅
Resona	❌	❌	10	❌
AuK	✅	❌	—	—
AuK-Flash	✅	❌	—	—
rumik-oss 1	❌	❌	22 Indic languages + English	—
Irodori-TTS-v4.1-Anime	—	❌	Japanese	—
ICE-012 Audio	✅	❌	590	✅
Coqui TTS Anonymizer	✅	❌	16	✅
TontaubeV1	✅	❌	7	✅
Breeze TTS 2	✅	❌	2	✅
Sopro v2 Turbo	✅	❌	4	✅
CuteTTS	✅	❌	5	✅
Rynsan TTS	—	❌	Khasi, Garo, Pnar, English, Hindi	—
Audio8 TTS Preview 0.1B	✅	❌	8	—
Kiseki-TTS	❌	✅	Japanese	—
FireRedTTS3	✅	❌	24	—
Audio8-TTS-Preview-0.6b	✅	❌	Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, Spanish	❌
NeuTTS-2E	❌	❌	English	✅
Scylla's Band	❌	❌	en_us, en_gb, es, it	✅
sanoTTS	❌	❌	English, Nepali, Hindi, Vietnamese, Indonesian, Chinese	❌
WordVoice-base-0.5B	✅	❌	2	—
FreyaTTS	❌	❌	Turkish	❌
Inflect-Nano-v2	❌	❌	English	❌
Gepard	✅	❌	English, Spanish, Portuguese, Dutch	✅
Higgs Audio v3 TTS	✅	❌	102	✅
dots.tts	✅	❌	Multilingual	❌
Confucius4-TTS	✅	❌	14	❌
WavTTS	✅	❌	English, Chinese	❌
MOSS-TTS	✅	❌	31	✅
VoxFlash-TTS	✅	❌	Chinese, English	❌
Miso TTS	✅	❌	English	❌
Raon-OpenTTS-1B	✅	❌	English	—
OronTTS	✅	❌	Mongolian, Kazakh	❌
Supertonic 3	✅	❌	31	✅
Scenema Audio	✅	❌	English, German, French, Spanish, Italian, Portuguese, Japanese, Chinese, Korean, Russian, Arabic, Hindi, Swahili	❌
Dramabox	✅	❌	English	❌
Sarashina2.2-TTS	✅	❌	Japanese, English	❌
LongCat-AudioDiT	✅	❌	Chinese, English	❌
VoXtream2	✅	❌	English	✅
SILMA TTS	✅	❌	Arabic, English	—
Fish Audio S2 Pro	✅	❌	80+	✅
LongCat-Next	✅	✅	Chinese, English	✅
Voxtral-4B-TTS	✅	❌	9	✅
Blue (Light Blue) TTS	✅	❌	Hebrew, English, Spanish, Italian, German	❌
KittenTTS	✅	❌	English, Multiple	✅
Ming-omni-tts	✅	❌	Chinese, English	❌
SoulX-Singer	✅	❌	Mandarin, English, Cantonese	✅
SoproTTS	✅	❌	English	✅
Qwen3-TTS	✅	❌	10	✅
TADA	❌	❌	English	❌
Irodori-TTS-500M-v2	✅	❌	Japanese	❌
KugelAudio	✅	❌	23 European languages	✅
LEMAS-TTS	✅	❌	10	❌
MioTTS-2.6B	✅	❌	English, Japanese	✅
MOSS-TTS-Nano	✅	❌	20	✅
NeuTTS	✅	❌	English, Spanish, German, French	✅
OmniVoice	✅	❌	600+	❌
T5Gemma-TTS	✅	❌	English, Chinese, Japanese	❌
TinyTTS	❌	❌	English	✅
VoxCPM2	✅	❌	30	✅
Soprano	❌	❌	English	✅
GLM-TTS	✅	❌	Chinese, English	✅
Echo-TTS	✅	❌	English	❌
VibeVoice-Realtime	✅	❌	Multilingual	✅
Fun-CosyVoice 3.0	✅	❌	9 + 18+ Chinese dialects	✅
LFM2-Audio-1.5B	✅	✅	English	✅
Marvis-TTS	✅	❌	English, French, German	✅
IndexTTS2	✅	❌	Chinese, English	✅
Maya1	✅	❌	English	✅
Step-Audio-EditX	✅	❌	Mandarin, English, Sichuanese, Cantonese, Japanese, Korean	✅
KaniTTS	❌	❌	English, German, Chinese, Korean, Arabic, Spanish	✅
VibeVoice-Finetuning	❌	❌	—	❌
VoxCPM	✅	❌	Chinese, English	✅
FireRedTTS2	✅	❌	EN, ZH, JP, KO, FR, DE, RU	✅
Audio Flamingo 3 (AF3) / Audio Flamingo Next	❌	✅	Multi-lingual	✅
ZipVoice	—	—	Chinese, English	—
Fish Speech	✅	❌	8	✅
Chatterbox	✅	❌	23+	❌
Orpheus-TTS	✅	❌	Multilingual	✅
MegaTTS3	✅	❌	Chinese, English	✅
Spark-TTS	✅	❌	Chinese, English	✅
Step-Audio	✅	✅	Chinese, English, Japanese	✅
Kokoro-82M	✅	❌	8	✅
KokoClone	✅	❌	7	✅
LuxTTS	✅	❌	-	✅
MiMo-Audio	✅	✅	Multi-lingual	✅
SoulX-Podcast	✅	❌	Mandarin, English, Cantonese, Sichuanese, Henanese	✅
VieNeu-TTS	✅	❌	Vietnamese	✅
Dia	✅	❌	English	✅
MeloTTS	❌	❌	English, Spanish, French, Chinese, Japanese, Korean	❌
Kimi-Audio	✅	✅	Multi-lingual	✅
eSpeak-NG	❌	❌	100+	✅

Iroha-TTS-V1

Description: Iroha-TTS-V1 is an expressive Japanese text-to-speech model for character dialogue from Iroha Studios, adapted from Fish Audio S2-Pro by supervised fine-tuning and policy optimisation. A single set of weights exposes seven controls: whole-line emotion (joy, anger, sadness, surprise), experimental emotion intensity via activation steering, mid-line emotion changes with inline tags, zero-shot voice cloning from a short reference recording and its transcript, placed vocal events ([laugh], [sigh], [cry]), whispering, and explicit kana readings for names and heteronyms. The slow and fast S2-Pro transformers predict discrete audio codes decoded to 44.1 kHz WAV; adaptation uses LoRA on both transformers followed by policy optimisation on the slow model's MLP only. Released as model weights plus runnable inference code — no training data or training pipeline — it targets character dialogue, audio drama and voice agents, and is benchmarked specifically on Japanese emotion control rather than general TTS quality.

Release Date: October 31, 2026

Feature	Value
Parameters	not stated; shipped as LoRA, adapter and intensity safetensors on top of Fish Audio S2-Pro, whose base weights are about 10 GB in BF16 and about 11 GB are downloaded on first run
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Pronunciation	✅
Languages	Japanese
Streaming	❌
License
Base Model	fishaudio/s2-pro (revision 1de9996b6be38b745688de084d87a5633f714e4e)
Architecture	Fish S2-Pro slow and fast transformers over discrete audio codes, 9 residual codebooks of 1024, model context 4096, max 400 frames, 44.1 kHz output
Adaptation	SFT with LoRA rank 32 alpha 16 on attention and MLP in both transformers (2,700 plus 1,600 steps), then policy optimisation with rsLoRA rank 16 alpha 64 on the slow MLP only, 8 candidates per request, KL weight 0.02
Output	single-pass full-utterance WAV capped at 400 frames or about 18.6 s, with no automatic sentence splitting
Voice Cloning Detail	a short reference recording plus its exact transcript; speaker similarity 0.694 without emotion and 0.636 with emotion
Emotion Tags	whole-line joy, anger, sadness and surprise, plus inline [happy] [angry] [sad] [surprised] tags for mid-line transitions
Intensity	activation steering at layers 20 and 22 with coefficients -0.06, -0.03, 0 and 0.045; experimental, only 70.83% of 72 contexts followed the full four-level ordering
Vocal Events	[laugh], [sigh] and [cry] placed inline; sighs detected in 56 of 60 requests (93.3%) and laughter in 44 of 59 evaluable clips (74.6%)
Whisper	[whisper] tag or a whispered reference; inference guidance 1.5 raised detections from 4 to 32 of 60 while lowering speaker similarity and UTMOS
Pronunciation Detail	explicit kana in brackets such as 水神[みずかみ] rewrites the input text before generation; 94.7% of 264 outputs correct versus 39.4% without, though the base model with the same frontend reached 93.9%
Emotion Accuracy	85.6% on emo2vec large and 93.3% on EmoRuler, versus 17.9% and 6.0% for the original Fish S2-Pro
Paired Comparison	with engine, inputs, reference encodings and seeds fixed and only the weights changed, EmoRuler accuracy rises from 6.04% to 89.17% and emotion2vec+ from 17.92% to 83.54%, with CER essentially unchanged
Mos	Japanese listening test overall 3.85 and 31.7% win share, ahead of Fish Audio Drama 3 at 3.69 and 23.2%; pairwise win rates 69.3% vs S2-Pro, 58.0% vs Drama 3, 58.7% vs Irodori-TTS v4-Large and 82.7% vs IndexTTS-2.5
Japanese Benchmarks	4.80% CER on CV3-Eval versus 5.30% for the original model rerun in the same environment; 3.33% CER and 0.745 speaker similarity on the MiniMax Japanese set
Binaural	optional spatial post-processing via spatial/binaural.py with TH Köln HRIRs; a separate CPU step, not a model feature
Vram	NVIDIA GPU with at least 28 GB free VRAM, Linux and Python 3.12
Release	launch post published October 31, 2026; technical notes dated October 29, 2026 with an October 30 listening-study snapshot; HF repository created October 20, 2026
Access	as of launch, anonymous access to the model repository and online Space is restricted and downloads require a Hugging Face account with access
License Terms	Fish Audio Research License, Copyright 39 AI, INC.; research and non-commercial use is free but commercial use requires a separate license from Fish Audio; new inference code in emotts/ and infer.py is MIT; S2-Pro base weights are downloaded separately under the same research license
Limitations	Japanese only; intensity control is experimental and cannot combine with inline tags; no segment-level evaluation of mid-line transitions; training data and pipeline are not released; the development panel was reused for model selection and four of its six voices appeared in EmoRuler training

Features: Iroha-TTS-V1 treats expression as a set of explicit, composable inputs over one shared set of weights rather than a collection of separate specialist models. Seven controls ride on the same S2-Pro adaptation: whole-line emotion, an experimental intensity dial, mid-line tag transitions, reference-based voice cloning, placed vocal events, whispering and kana readings. The training is deliberately paired — a teacher model generates happy, angry, sad and surprised versions of the same line with the same reference audio, candidates are filtered for emotion, wording, voice and quality, and LoRA plus policy optimisation then learns the tag's acoustic effect instead of guessing emotion from word choice. The most interesting result is the strictly paired comparison: with the inference engine, inputs, reference encodings and random seeds held fixed and only the weights changed, emotion accuracy rises from 6.04% to 89.17% on EmoRuler while character error rate stays essentially unchanged, which isolates the adaptation from the surrounding pipeline. The release is also candid about what did not work — more unpaired data gave limited gains, intensity labels without acoustic differences taught nothing, emotion rewards induced unrequested laughter in 11.7% of happy samples, and explicit kana rewriting rather than a learned reading tag produced most of the pronunciation gain. That honesty matters because the headline numbers come from a development panel reused for model selection, with four of six voices present in the classifier's training data: this is strong evidence for controllable Japanese character dialogue, not a claim of general TTS superiority.

Links:

· · · · · · · · · · · · · ·

OGG TTS

Description: OGG TTS is a 0.8B-parameter zero-shot text-to-speech model from FashionFlora, trained on about 8,200 hours of professionally acted speech in English, Polish, Spanish, German, Russian, French, Portuguese and Italian. It generates discrete tokens of the Fish Audio 44.1 kHz codec with block diffusion: each 16-frame (0.74 s) block is produced in parallel by iterative confidence-ordered unmasking, while blocks follow one another left to right, so finished blocks never change and can be played immediately. The backbone is initialised from OmniVoice (Qwen3-0.6B, masked discrete diffusion) and a 164M-parameter chained multi-token-prediction decoder fills in the nine residual codebooks of each block. Expression is controlled with 24 optional inline tags covering emotion, delivery and vocal events, and multi-speaker dialogue is generated in a single pass. This is a preview release — the published checkpoint is an intermediate step of a 400k-step pre-training run that is still in progress, and formal evaluation is withheld for the final checkpoint.

Release Date: October 30, 2026

Feature	Value
Parameters	0.81B (815,092,736 F32, safetensors-measured; codec excluded)
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	8 (English, Polish, Spanish, German, Russian, French, Portuguese, Italian)
Streaming	✅
License
Architecture	OmniVoice / Qwen3-0.6B backbone, 28 layers, hidden 1024, bidirectional attention within a block and block-causal across blocks, plus a 9-layer chained MTP acoustic decoder of 164M parameters
Base Model	k2-fsa/OmniVoice (Qwen3-0.6B masked discrete diffusion)
Codec	Fish Audio Drama 3 / S2 codec, 44.1 kHz, 21.53 frames per second, 1 semantic codebook of 4096 plus 9 residual codebooks of 1024
Decoding	6 or 8 confidence-ordered unmasking iterations per block with classifier-free guidance, then 4 MTP rounds per residual codebook; KV cache; length decided by an end-of-speech token
Block Size	16 frames or 0.74 s; roughly 0.17 real-time factor on one GPU
Voice Cloning Detail	zero-shot from a few seconds of reference audio plus its transcript, or built-in speaker identities with no audio prompt
Control Tags	24 optional inline tags in three groups — emotion [angry] [frustrated] [excited] [tired] [playful] [sad] [scared] [shouting]; delivery [whispering] [softly] [voice breaking] [singing] [pause] [slowly]; vocal events [laughing] [sighing] [gasping] [panting] [coughing] [clears throat] [sniffling] [scoffing] [yawning] [humming]
Multi Speaker	whole dialogue generated in one pass with per-voice prompts and turn markers such as S1 and S2
Tag Annotation	audio-language-model pass over 502,668 sentences under a closed 24-tag list, tagging only what is audible; 57,016 sentences or 11% received at least one tag; tags are dropped from part of training to keep them optional
Training Data	about 8.2k hours from 584 audiobooks and games, 16,628 speakers, 6.1M sentences, 559M codec frames, 5.6B codec tokens, about 112M text tokens
Language Hours	English ~2,610 h, Polish ~1,300 h, Spanish ~820 h, German ~750 h, Russian ~645 h, French ~500 h, Portuguese ~300 h, Italian ~290 h
Training	400k steps planned at 16k tokens per step, AdamW with cosine decay in bf16 on one NVIDIA RTX PRO 6000 Blackwell 96 GB; roughly 190 hours of audio per training hour
Compute Cost	about 173 GPU hours and USD 1,040 projected total including about USD 455 of annotation; five discarded pilot runs consumed about 20 GPU hours
Evaluation	in progress — Seed-TTS (EN) and CV3 (PL, DE, FR, ES, IT, RU, PT) for WER/CER, speaker similarity, UTMOS, tag compliance and real-time factor
Release	GitHub repository and model card first published October 30, 2026 at the 20k checkpoint; the 40k checkpoint revision followed October 31, 2026
License Terms	Apache-2.0 for the code, Fish Audio Community License Agreement v1.0 for the Fish Audio codec and software; research and non-commercial use is free and limited commercial use is permitted while annual revenue is below USD 500,000; hosted services need a separate written license
License Note	the authors state that because the training data contains copyrighted recordings, the released weights are intended for research
Limitations	preview checkpoint with weak tag control and unfinished speaker similarity; Italian and Portuguese have the least data; rare tags such as [singing] are unreliable; very short multi-speaker turns can be dropped

Features: OGG TTS swaps the usual autoregressive token-by-token speech decoder for block diffusion over Fish Audio codec tokens: a Qwen3-0.6B masked-diffusion backbone unmasks the semantic codes of a 0.74 s block in a handful of confidence-ordered, classifier-free-guided iterations, then a chained multi-token-prediction transformer fills all nine residual codebooks of that block in parallel. Blocks are generated left to right and once complete are immutable, which makes low-latency streaming a structural property rather than a chunking hack — roughly 0.17 real-time factor on one GPU while producing 44.1 kHz audio. The second bet is on data: 8.2k hours of professionally acted, multi-cast audiobook and AAA-game dialogue, with 24 emotion, delivery and vocal-event tags produced by an audio-language model instructed to label only what is audible, and tags deliberately dropped from part of the training set so control stays optional at inference. The release is unusually transparent about cost and status — a single RTX PRO 6000, about USD 1,040 of projected compute and annotation, five discarded pilot runs, and an explicit statement that this is an intermediate checkpoint of a run still in progress. That honesty cuts both ways: the architecture and recipe are fully documented, but there are no published WER, speaker-similarity or UTMOS numbers yet, so the quality claims remain unverified.

Links:

· · · · · · · · · · · · · ·

Ming-3.0-tts-0.5B

Description: Ming-3.0-tts-0.5B is the dense half of inclusionAI's Ming-3.0-tts family: a 0.5B speech-generation model on a Qwen2.5-0.5B backbone, shipped as a single 1.48B-parameter checkpoint (safetensors-measured 1,479,454,868 in BF16) that bundles the backbone, a DiT patch head, an AudioVAE audio tokenizer inherited from the Ming-2.0 generation, a CAM++ speaker encoder and the text tokenizer. Output is 44.1 kHz. It speaks seven languages — Chinese, English, German, Spanish, French, Korean and Japanese — plus Sichuanese and Cantonese dialects, and exposes control as task modes rather than arbitrary combinations: zero-shot cloning from a reference clip, voice design from a natural-language description, free-form delivery instructions, 14 inline paralinguistic tags (breath, cough, sigh, laughter, hesitation, surprise, question and confirmation sounds, in Chinese or English), explicit emotion, speed/pitch/volume attributes, and built-in IP voices (100+ Chinese, five English; recommended US_James, US_Stella, 灵小柔, 灵楷). On the independent Artificial Analysis leaderboards (October 2026) it ranks 1st among open-weights models on Controlled Voice (Elo 1069) and 1st overall on Pronunciation Robustness (93.2%). It is MIT-licensed, with the technical report still pending; the honest weak spot is speaker similarity, which trails content accuracy across every language.

Release Date: October 19, 2026

Feature	Value
Parameters	1,479,454,868 (0.5B dense backbone + AudioVAE + DiT head + aggregator, safetensors-measured, BF16)
Voice Cloning	✅
Asr	❌
Languages	7 (Chinese, English, German, Spanish, French, Korean, Japanese) + Sichuanese and Cantonese dialects
License
Architecture	dense Qwen2.5-0.5B backbone + DiT patch head + AudioVAE audio tokenizer (Qwen2-style, 24 layers, hidden 896) + CAM++ speaker encoder (BailingMMNativeForConditionalGeneration)
Sample Rate	44.1 kHz
Controls	emotion (7 categories), speed/pitch/volume, dialect, free-form natural-language instruction, text-described voice design
Paralinguistic Tags	14 (breath, cough, sigh, laughter, hesitation, surprise, question/confirmation — Chinese or English)
Built In Voices	100+ Chinese, 5 English (recommended US_James, US_Stella, 灵小柔, 灵楷)
Tasks	TTS, FREEFORM, VOICE_DESIGN, MULTILINGUAL, BASIC, EMOTION, DIALECT, IP (task modes are mutually exclusive)
Text Normalization	FST grammars via pynini (--no_text_normalization to skip)
Sibling Model	Ming-3.0-tts-tiny (MoE on Ling V3, 7.9B total / 1.3B active) adds multi-speaker podcast synthesis
Seed Tts Zh	WER 0.800% / SIM 0.758 (Seed-TTS reference 1.110% / 0.796; 3rd on Chinese WER among compared open models)
Seed Tts En	WER 1.584% / SIM 0.691
Multilingual Minimax	German 0.365% WER, Spanish 1.099% WER, French 4.264% WER (SIM 0.675), Korean 1.728% CER, Japanese 3.765% CER
Nvbench Single	zh CER 2.70 / PCER 22.80 / OCER 3.20; en CER 7.60 / PCER 37.00 / OCER 8.90 (SOTA on all six single-tag metrics jointly with tiny)
Dialect Wsc	easy CER 2.27% (SOTA) / SIM 0.689 / ACC 80.68%; hard CER 3.35% / SIM 0.691 / ACC 87.64%
Emotion Cv3	72.33% average (3rd); emotion_cv3n: 37.33% average (2nd)
Voice Design Instructtts	72.29% average (3rd; APS 78.54, DSD 77.28, RP 61.06)
Attribute Control	speed 99% / volume 95% / pitch 86% (avg 93.30%, leads the compared set), WER 0.317%, SIM 0.712
Leaderboard Aa	Controlled Voice 1069 (1st open-weights, 13th overall), Pronunciation Robustness 93.2% (1st overall), Provider Voice 1188 (3rd open-weights, 17th overall) — October 2026
Runtime	transformers 4.46.3, torch 2.6 / CUDA 12.4 Docker image, vLLM-Omni accelerated inference
Sampling	cfg_scale 2.0, temperature 0, flow_steps 10, sigma 0.25, seed 1234
Hardware	tested on NVIDIA H800-80GB and H20-96G
Limitations	speaker similarity weaker than content accuracy (French SIM is the clearest gap); pitch less accurate than speed and volume; emotion control harder when the words carry no emotional signal; open-ended voice design still behind Qwen3-TTS; technical report pending

Features: The interesting part is not the model size but the control surface: a 0.5B dense checkpoint that exposes voice cloning, text-described voice design, free-form delivery instructions, 14 inline paralinguistic tags, dialect selection and explicit emotion/attribute controls as distinct task modes, with the modes deliberately non-composable so a request never mixes incompatible conditioning. That design is what produces SOTA single-tag paralinguistic accuracy (zh CER 2.70) and SOTA Cantonese base CER (6.15% with 98.00% accuracy) from a model small enough for a single H800, and the independent arena results back it up — first among open-weights models on Controlled Voice and first overall on Pronunciation Robustness, where text normalization (96.5% on Expanding Shorthand) and sequence fidelity (90.6%) are the deciding axes. The honest caveat is that content accuracy and speaker similarity move independently here: intelligibility is excellent, voice identity is the weaker half, and the older Ming-omni-tts still holds the overall records for attribute-control accuracy, pitch and similarity.

Links:

· · · · · · · · · · · · · ·

Fish Audio Drama 3

Description: Fish Audio Drama 3 is a 35-billion-parameter mixture-of-experts text-to-speech model with roughly 3B parameters active per token. Its explicit target is not narration but dramatic performance: multi-character scenes with up to 16 tagged speakers, free-form director's notes, performative tags, multi-round revision and infill editing. The backbone is a hybrid attention stack — 40 layers mixing full attention with linear attention — over a 256-expert / 8-active MoE whose experts are stored as FP8 E4M3 blocks while the remainder stays in BF16, giving a measured 35.9B parameters across 8 safetensors shards. The audio side uses a codebook-MTP decoder: a 4-layer shared multi-token-prediction tower over 10 codebooks decoded causally across the whole sequence, feeding the same ModifiedDAC codec as Fish Audio S2 (10 codebooks at ~21.5 Hz, 44.1 kHz output). It was initialised from Qwen3.5, and post-training used GRPO with a single learned human-preference reward model and no KL penalty.

Release Date: October 13, 2026

Feature	Value
Parameters	35.9B total with ~3B active per token (35,901,609,088 safetensors-measured across 8 shards; 62,245 tensors)
Weight Layout	32,212,254,720 FP8 E4M3 expert params, 3,687,388,288 BF16, 1,966,080 F32; index total 39,594,895,616 bytes
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	83 (language codes in the model card; the card describes roughly 80 languages and dialects)
Streaming	✅
License
Architecture	FishOmniModel, model_type fish_omni, bfloat16 dtype
Attention	hybrid — full attention plus linear attention over 40 layers, hidden 2048, 16 heads, 2 KV heads, head_dim 256, attn_output_gate enabled
Moe	256 experts with 8 active per token, moe_intermediate_size 512, shared-expert gate, bsgemm-moe quantization with 128x128 weight blocks and dynamic activation scheme
Audio Decoder	codebook_mtp_shared — 4 layers, 4 shared enrichment layers, dim 2048, audio_hidden_dim 4096, intermediate_size 6144, 10 codebooks, vocab 4096, rope_base 10000000
Context Length	262,144 max position embeddings, vocab 252,288
Base Model	initialised from Qwen3.5
Codec	ModifiedDAC, 10 codebooks at approximately 21.5 Hz, 44.1 kHz output
Speakers	up to 16 tagged characters via `<
Chunking	long text auto-chunked at roughly 300 UTF-8 bytes with up to 3 speakers per chunk, streamed per chunk
Speech Template	reference-bracket-v1
Training Data	several hundred billion tokens spanning roughly 80 languages and dialects
Post Training	GRPO reinforcement learning with a single learned human-preference reward model, no rule-based rewards and no KL penalty
Editing	editing-as-continuation via previous_takes, plus infill editing with lower WER than the Cartesia Sonic-3 infill API on Ming-Freeform-Audio-Edit in English and Chinese
Revision	multi-turn revision raises emotion accuracy from 55.5% to 58.2% and style accuracy from 76.5% to 81.8% over 3 rounds on Step-Audio-Edit
Throughput	271 frames per second versus 177 for Fish Audio S2 on one H200 despite more than 7x the parameters
Serving	on one H200, concurrency 1 gives 166.7 frames per second with 88.9 ms median TTFA and RTF 0.131; concurrency 16 gives 927.6 frames per second with 210.1 ms median TTFA; concurrency 64 gives 1,687.3 frames per second with 450.9 ms median TTFA
Vocoder Placement	moving the vocoder to a second GPU raises concurrency-256 throughput from 2,422 to 4,160 frames per second, about 72% more, and cuts median TTFA from 3,403 ms to 2,146 ms
Vram	NVIDIA GPU with at least 48 GB for the FP8 checkpoint
Deployment	vLLM-Omni, SGLang-Omni, mlx-audio, llama.cpp, audio.cpp, CTranslate2 and ComfyUI nodes
Release	v3.0.0-beta pre-release on GitHub on October 13, 2026, with the technical report on arXiv as 2610.18010
License Terms	research and non-commercial use free of charge; commercial use permitted while annual revenue is below the USD 500,000 revenue threshold; hosted services require a separate written license; attribution notice and a "Built with Fish Audio" display are required; voice impersonation without consent and removal of watermarks or provenance are prohibited; outputs may not be used to train other speech or audio models

Features: Drama 3 treats speech generation as a directing problem rather than a speaking problem, and the architecture is designed around that bet. A 16-way speaker-tag system, performative point tags such as [sighs] and [laughs], scoped tags like <whisper>...</whisper>, free-form style descriptions and unstructured director's notes all push the user toward staging a scene rather than writing a prompt and accepting one take. The second bet is editing as a first-class verb: previous_takes turns a revision into a continuation of the model's own output, multi-round revision measurably lifts emotion accuracy from 55.5% to 58.2% and style accuracy from 76.5% to 81.8%, and infill editing beats Cartesia Sonic-3's infill API on WER in both English and Chinese. The third bet is that sparsity plus FP8 is enough to make a 35B speech model serveable: 256 experts at 8 active keep per-token cost near 3B while FP8 E4M3 expert storage keeps the footprint down, and the reported 271 frames per second on one H200 against Fish Audio S2's 177 — despite more than 7x the parameters — is the number that makes the case. Two design details are worth flagging for anyone building on top: the codebook-MTP decoder decodes all 10 codebooks causally over the entire sequence rather than per-frame, which is what makes long multi-speaker scenes coherent; and the hybrid full-plus-linear attention stack is the price of a 262,144-token context at this scale. The post-training recipe is notable for what it omits — GRPO with a single learned human-preference reward model, no rule-based rewards and no KL penalty, a deliberately small number of moving parts for a model of this size. The community license is also unusually concrete for a 35B speech model: commercial use is allowed below USD 500,000 annual revenue, outputs may not be used to train other speech or audio models, and the attribution notice is mandatory.

Links:

Additional Tools:

Tool	Type	Link
darioooooo0o/drama-3-w4a16-32gb	NVIDIA CUDA W4A16 int4 experts with int8 backbone, 30.7 GiB peak for 32 GB GPUs	darioooooo0o/drama-3-w4a16-32gb
darioooooo0o/drama-3-w4a16	NVIDIA CUDA W4A16 int4 experts, 41.5 GB down to 25.9 GB, about 38% smaller	darioooooo0o/drama-3-w4a16
akhaliq/drama-3-int4	NVIDIA CUDA hybrid W4A16 GPTQ-Marlin int4, 22.1 GB, runs on Ampere A100 sm80, published WER and speaker-similarity evals	akhaliq/drama-3-int4
WaveCut/Drama-3-OrbitQuant-W4A4	NVIDIA CUDA OrbitQuant W4A4, RPBH rotation with Lloyd-Max codebooks, 25.44 GB bundle	WaveCut/Drama-3-OrbitQuant-W4A4
WaveCut/Drama-3-OrbitQuant-W2A2	NVIDIA CUDA OrbitQuant W2A2, 11.8 GB, smallest CUDA quant in this set	WaveCut/Drama-3-OrbitQuant-W2A2
Reza2kn/drama-3-nvfp4	NVIDIA Blackwell NVFP4 E2M1 with E4M3 scales per 16 input channels, CUTLASS kernels, 27.42 GB total	Reza2kn/drama-3-nvfp4
asfberlin/drama-3-128e-w4a16	NVIDIA CUDA W4A16 pruned to 128 of 256 experts by measured router usage, 12.7 GiB on disk, 15.4 GiB peak for 2 concurrent requests on RTX 5080	asfberlin/drama-3-128e-w4a16
skysay/drama3-emo-w4	NVIDIA CUDA mixed W4A8 with W6A8 on layer 0, emotion-calibrated over 630 seed-matched takes, for ComfyUI-Drama3	skysay/drama3-emo-w4
mlx-community/fish-audio-drama-3-bf16	Apple Silicon MLX BF16, 67 GB	mlx-community/fish-audio-drama-3-bf16
mlx-community/fish-audio-drama-3-8bit	Apple Silicon MLX 8-bit, 42 GB, needs a 64 GB Mac	mlx-community/fish-audio-drama-3-8bit
mlx-community/fish-audio-drama-3-4bit	Apple Silicon MLX 4-bit, 22 GB	mlx-community/fish-audio-drama-3-4bit
mlx-community/fish-audio-drama-3-3bit	Apple Silicon MLX 3-bit, 16.9 GB, fits a 32 GB Mac, WER 1.4% and UTMOS 4.04	mlx-community/fish-audio-drama-3-3bit
vanch007/Drama-3-MLX-BF16	Apple Silicon MLX BF16, 69.0 GB and RTF 5.238 on M3 Max 128 GB	vanch007/Drama-3-MLX-BF16
vanch007/Drama-3-MLX-8bit	Apple Silicon MLX 8-bit affine quantization with group size 64, 32.6 GB and RTF 4.827 on M3 Max 128 GB	vanch007/Drama-3-MLX-8bit
vanch007/Drama-3-MLX-4bit	Apple Silicon MLX 4-bit affine quantization with group size 64, 23.2 GB and RTF 3.453 on M3 Max 128 GB	vanch007/Drama-3-MLX-4bit
fcmeyer/drama-3-MLX-3bit-HQ	Apple Silicon MLX 3-bit experts except 15 sensitivity-swept layers at 4-bit, 19.7 GB plus 1.6 GB codec, 34 GB peak on M5 Max at 2.7x real time	fcmeyer/drama-3-MLX-3bit-HQ
fcmeyer/drama-3-MLX-4bit-HQ	Apple Silicon MLX 4-bit quality-weighted, 24.5 GB, 39 GB peak on M5 Max at 2.3x real time	fcmeyer/drama-3-MLX-4bit-HQ
fcmeyer/drama-3-MLX-5bit	Apple Silicon MLX 5-bit, 26.3 GB, 40 GB peak on M5 Max at 2.8x real time	fcmeyer/drama-3-MLX-5bit
pentacoxian-dev/drama3-small-GGUF	GGUF pruned to 45% of expert slots and distilled back to Japanese and English, IQ3_XXS at 9.74 GiB, 10.26 GiB measured peak on RTX 4090, requires an audio.cpp patch	pentacoxian-dev/drama3-small-GGUF
Jmica/Drama-3-GGUF	GGUF audio.cpp-native, Q2_K, Q3_K_S and Q4_K_S expert banks with explicit backbone precision overrides	Jmica/Drama-3-GGUF
EryriLabs/drama-3-llamacpp-GGUF	GGUF llama.cpp build, Q4_K experts with Q8 backbone at 20.8 GB plus a 1.7 GB mmproj, run via mara-drama3	EryriLabs/drama-3-llamacpp-GGUF
jordimas/drama-3-ct2	CTranslate2 conversion preserving FP8 E4M3 experts and block scales, needs a custom CTranslate2 build with Drama3 support	jordimas/drama-3-ct2
PedroMarinhoDev/Drama-3-ComfyUI	ComfyUI weights for ComfyUI-Drama3, FP8 and int8 expert variants plus W4A8 and W6A8, ComfyUI v0.40+ with 8 GB VRAM and 32 to 48 GB RAM	PedroMarinhoDev/Drama-3-ComfyUI
UltimateIntent/Fish-Audio-Drama-3-Q2-Q3	ComfyUI Q2_K and Q3_K expert quants with bundled workflow JSON, ASR and synthetic-voice auxiliaries	UltimateIntent/Fish-Audio-Drama-3-Q2-Q3

· · · · · · · · · · · · · ·

Resona

Description: Resona is an 82M-parameter neural TTS engine built for the opposite of the cloud: 100% air-gapped, no GPU, no telemetry, synthesized on a commodity CPU. It targets the gap the author describes between cloud APIs (which cost latency and privacy) and small open engines that sound robotic on Indian languages and Hinglish code-switching. The pipeline is a text normalizer feeding a language router; Hinglish text passes through a custom transducer with a protected technical-term whitelist before hitting a Devanagari/IPA phonemizer (built on espeak-ng, which must be installed separately), then a token-and-rhythm alignment stage and an AdaIN prosody predictor driven by a 256-D voice style tensor, an 82M neural vocoder, and finally a breathing-cadence injector that places 180 ms clause pauses and 350 ms sentence boundaries. Two checkpoints ship (resona-v1.pth and resona-indic-v1.pth, ~327 MB each) and 21 "Grade A" voice personas are selectable by name, each defined by a pre-rendered style tensor rather than a reference clip. Speed is the headline metric: RTF 0.18x and a 180 ms cold start measured on a single thread of an Intel Core i7-12700H, at ~420 MB RAM, outputting 24 kHz WAV.

Release Date: September 26, 2026

Feature	Value
Parameters	82M (two ~327 MB checkpoints — resona-v1.pth and resona-indic-v1.pth)
Voice Cloning	❌
Asr	❌
Languages	10 (Hinglish and Hindi are native; es/fr/it/pt/ja/zh and en-GB are routed through the espeak-ng IPA G2P front-end into the same vocoder)
Streaming	❌
License
Output Sample Rate	24 kHz
Voices	21 "Grade A" personas, each with a persona character, pitch and speed; 8 Hinglish/Indic flagships include `anjura`, `divya`, `meera`, `priya`, `arjun`, `kabir`, `aman`, `atul`
G2P	espeak-ng Devanagari/IPA for Hindi; American and British English IPA; Spanish, French, Italian, Portuguese IPA; Romaji/Kana for Japanese; Pinyin for Mandarin
Code Switching	custom Hinglish transducer with a protected technical-term whitelist so terms are not mis-transliterated
Prosody Control	AdaIN prosody predictor conditioned on a 256-D voice style tensor
Breathing Cadence	180 ms clause pauses and 350 ms sentence boundaries inserted automatically
Pipeline	text normalizer → language router → Hinglish transducer / phonemizer → token and rhythm alignment → AdaIN prosody predictor → 82M neural vocoder → breathing cadence → 24 kHz WAV
Benchmarks	cold start 180 ms, RTF 0.18x, 48 words/s, ~420 MB RAM — measured on one thread of an Intel Core i7-12700H
Quality Metrics	none reported (no MOS, speaker similarity, intelligibility or third-party evaluation)
Requirements	Python 3.9–3.12 and `espeak-ng` installed on the host (auto-discovery plus CLI fallback)
Interfaces	Python API (`from resona import ResonaPipeline`) and a `resona-tts` CLI with `--list-voices`, `--voice`, `--lang`, `--speed`
Repository	self-contained Python package (core, preprocessing, voices, vocab, cli), bundled lexicons, example scripts, an automated test suite, and SHA-256 release integrity hashes
Provenance	the repository ships a NOTICE file describing upstream architectural foundations

Features: What Resona is really selling is a budget: 82M parameters and a single-thread CPU RTF of 0.18x puts it in a class of its own in this list, where most TTS models assume a GPU. The engineering consequence is visible in the architecture — rather than a large autoregressive acoustic model, the quality budget is spent on deterministic post-processing (the breathing-cadence pause injector) and on front-end robustness (the Hinglish transducer plus a protected technical-term whitelist, which is a surprisingly concrete acknowledgment that code-switched text usually breaks on exactly the product names, resolutions and menu items a tutorial voice needs to say). The 21 personas as 256-D style tensors with per-voice pitch and speed is a lightweight alternative to voice cloning: no reference audio, no training, just a style vector per speaker. The "10 languages" figure deserves a careful read, though — Hinglish and Hindi are the native path, while the remaining eight come from espeak-ng's existing IPA front-end feeding the same 82M vocoder, so the breadth is inherited from the phonemizer rather than from acoustic training in those languages. This is also a very young release: first public commit 2026-09-26, no published quality metrics, no MOS or speaker-similarity numbers, and speed figures that are self-reported on one CPU. Treat the engineering and the licensing as the substance, and the "studio-grade" quality claims as unverified.

Links:

· · · · · · · · · · · · · ·

AuK

Description: AuK is a 1.5B foundation model from Tencent for speech generation and editing, trained on millions of hours of diverse audio data. Through a single natural-language instruction interface it unifies an unusually broad task set: zero-shot TTS (speak text in the reference voice) and instruct TTS (voice from a description alone, no reference), content editing (rewrite what is said; even lyric editing that preserves melody and voice), acoustic editing (pitch by semitones, speed, volume), paralinguistic editing (emotion, timbre, de-accent, nonverbal sounds, whisper conversion), and enhancement & separation (denoise/dereverberate, speech separation, music/vocal separation, target-speaker extraction). Architecture: a diffusion transformer with layer-fusion weights, conditioned by a Qwen2.5-Omni-3B MLLM encoder and a separate VAE (loaded at runtime). Day-0 SGLang-Omni serving support, Gradio and ComfyUI integrations, and a task Cookbook are provided. Released under MIT.

Release Date: September 9, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
License
Parameters	1.5B
Architecture	diffusion transformer + layer fusion, Qwen2.5-Omni-3B MLLM encoder, separate VAE
Variants	AuK (this, base) + AuK-Flash (distilled, 4-step inference)
Editing	content, lyric, pitch, speed, volume, emotion, timbre, de-accent, nonverbal, whisper conversion
Enhancement Separation	speech enhancement, speech separation, music separation, target speaker extraction
Deployment	SGLang-Omni (day-0), Gradio, ComfyUI

Features: Unifies generation and the full editing/enhancement/separation spectrum in one instruction-following model — most systems pick one lane (TTS, or editing, or separation); AuK does zero-shot + instruct TTS, lyric rewriting with melody preservation, emotion/timbre/de-accent/whisper paralinguistic edits, and source separation through the same natural-language interface. A diffusion transformer with layer fusion, distilled into a 4-step AuK-Flash variant for fast inference.

Links:

Additional Tools:

Tool	Type	Link
ComfyUI-AuK	ComfyUI node	ComfyUI-AuK

· · · · · · · · · · · · · ·

AuK-Flash

Description: AuK-Flash is the distilled variant of AuK, Tencent's 1.5B foundation model for speech generation and editing, optimized for fast 4-step inference. It exposes the same natural-language instruction interface as the base model: zero-shot TTS (reference voice) and instruct TTS (voice description, no reference), content and lyric editing, pitch/speed/volume acoustic edits, emotion/timbre/de-accent/nonverbal/whisper paralinguistic edits, plus speech enhancement and speech/music/target-speaker separation. Architecture matches the base: diffusion transformer with layer-fusion weights, Qwen2.5-Omni-3B MLLM encoder, and a separate runtime-loaded VAE. Released under MIT.

Release Date: September 9, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
License
Parameters	1.5B
Architecture	diffusion transformer + layer fusion (distilled to 4 inference steps), Qwen2.5-Omni-3B MLLM encoder, separate VAE
Base Model	tencent/AuK
Editing	content, lyric, pitch, speed, volume, emotion, timbre, de-accent, nonverbal, whisper conversion
Enhancement Separation	speech enhancement, speech separation, music separation, target speaker extraction
Deployment	SGLang-Omni, Gradio, ComfyUI

Features: Distills the AuK foundation model's diffusion transformer down to 4 inference steps, making the full generate-and-edit capability set (including enhancement and separation) practical for interactive use — traded against the base model's maximum quality, with the two variants loadable side-by-side from the same codebase.

Links:

· · · · · · · · · · · · · ·

rumik-oss 1

Description: rumik-oss 1 is a 3B multilingual text-to-speech model from rumik ai, trained on fewer than 70,000 hours of speech while performing competitively with existing TTS models. It covers 22 Indic languages in their native scripts and romanized forms plus English, supporting both single-language and code-switched synthesis. Delivery is conditioned via <description="..."> tags (tone, accent, pace), with inline vocalization control (<laugh>, <chuckle>, <sigh>). It extends CohereLabs/tiny-aya-fire with discrete speech tokens from the mimi codec: following the flattened codec-token formulation used in llama-mimi, text conditioning and audio generation share a single autoregressive sequence, predicting eight codebook tokens per frame before advancing, with the frozen mimi decoder reconstructing the 24 kHz waveform. The model ships with 4 fixed voices (Ira, Aisha, Siya, Zoya) that perform equally well across all 22 languages; there is no zero-shot voice cloning. Licensed under Cohere's CC-BY-NC-4.0 with acceptable-use addendum (research and non-commercial use only).

Release Date: September 6, 2026

Feature	Value
Voice Cloning	❌
Asr	❌
Languages	22 Indic languages + English (native scripts and romanized; code-switching supported)
License
Parameters	3B (3,381,533,697 BF16)
Architecture	CohereLabs/tiny-aya-fire backbone + flattened mimi codec tokens (8 codebooks/frame, llama-mimi formulation)
Audio Codec	kyutai/mimi (frozen decoder), 24 kHz output
Pronunciation	✅
Highlights	description-conditioned delivery (`<description>` tags), inline vocalizations (`<laugh>`/`<chuckle>`/`<sigh>`)
Variants	rumik-oss-1 (post-trained), rumik-oss-1-base (speaker-conditioned pre-post-training)

Features: Brings competitive multilingual TTS to 22 Indic languages with under 70k training hours, using a flattened mimi codec-token formulation (single autoregressive sequence for text conditioning + audio) on the tiny-aya-fire backbone. Code-switched synthesis, description-conditioned delivery, and inline vocalization tags are first-class capabilities, and its 4 voices perform equally well across all 22 languages — unusual, as most TTS voices are language-specific.

Links:

Additional Tools:

Tool	Type	Link
ComfyUI-Rumik-TTS	ComfyUI node	ComfyUI-Rumik-TTS

· · · · · · · · · · · · · ·

Irodori-TTS-v4.1-Anime

Description: Irodori-TTS-v4.1-Anime is a Japanese text-to-speech model fine-tuned from Aratako/Irodori-TTS-v4.1-Small using anime-style speech data. Because the base model's annotation pipeline is not publicly documented, the fine-tuning data was annotated independently — so caption conditioning and emoji controls may behave differently from the base model. The full-precision checkpoint (0.8B params, F32) ships at the repository root, with quantized variants (int8-weight-only, int8-dynamic, int4-weight-only, float8-weight-only, float8-dynamic) in subdirectories. It follows the base model's MIT License and ethical restrictions; inference uses the original Irodori-TTS repository.

Release Date: September 4, 2026

Feature	Value
Voice Cloning	—
Asr	❌
Languages	Japanese
License
Parameters	~0.8B (766,052,385 F32)
Architecture	Irodori-TTS (Aratako) fine-tune; caption-conditioned with emoji controls
Base Model	Aratako/Irodori-TTS-v4.1-Small
Variants	int8-weight-only, int8-dynamic, int4-weight-only, float8-weight-only, float8-dynamic

Features: A community fine-tune that ports the Irodori-TTS line into the anime-voice domain using an independently built annotation pipeline (since the base model's is undocumented), and ships the result with five ready-made quantization variants (int4/int8/fp8) for efficient inference.

Links:

· · · · · · · · · · · · · ·

ICE-012 Audio

Description: ICE-012 Audio is a multilingual text-to-speech model from DarkPs (a FanuonAI organization) with streaming output and reference-based voice cloning. Its defining trait is breadth of language coverage — 590 language names/variants are accepted (name or 2–3-letter ID, with a language-agnostic fallback), including 13 Arabic dialects ("Lahgtna" variants) alongside the full ISO list. It introduces an active acoustic adapter — conditioning codec embeddings before the backbone and refining hidden states after it. Voice is controllable along six axes: gender (male/female), age (child → elderly), pitch (5 levels), accent (10 English accents), style (e.g. whisper), and speed (0.5–2.0×), plus an --auto-voice mode where the model picks a voice automatically. The checkpoint is ~714M parameters (F16) and runs via transformers with trust_remote_code=True. Released under CC BY-NC 4.0.

Release Date: August 29, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	590 names/variants (incl. 13 Arabic Lahgtna dialects; language-agnostic fallback)
Streaming	✅
License
Parameters	~714M (714,409,993 F16)
Architecture	causal LM with active acoustic adapter (conditions codec embeddings pre-backbone, refines hidden states post-backbone)
Voice Controls	gender, age, pitch, accent, style, speed; auto-voice mode

Features: The active acoustic adapter wraps the backbone on both sides — conditioning codec embeddings before it and refining hidden states after — while a six-axis voice-control space (gender/age/pitch/accent/style/speed plus auto-voice) and 590-language coverage make it one of the broadest single-checkpoint TTS releases for dialect and minority-language synthesis.

Links:

· · · · · · · · · · · · · ·

Coqui TTS Anonymizer

Coqui TTS Anonymizer (rm00cr/coqui-tts)

Description: Coqui TTS Anonymizer is a fork of Coqui TTS whose headline capability is a speaker-anonymization pipeline: give it a recording plus a donor voice, and it returns the same words spoken in the donor's voice. The transcript survives verbatim; the original speaker's vocal identity does not. The paper behind it (Your Voice Cloning System is Secretly a Voice Anonymizer, arXiv 2608.27360) makes the mechanism explicit — XTTS v2's voice cloning preserves prosodic structure independently of speaker identity, so the cloning path can be repurposed as voice conversion by conditioning on a pseudo-speaker, with no retraining. The fork adds an anonymizer/ package (pipeline, config, voice pools, donor selection, scoring, resumable batch, CLI), anonymization entry points on the Xtts class, and a weight-free unit-test suite. Evaluation covers seven European languages across CommonVoice and Multilingual LibriSpeech, reporting near-optimal privacy (EER ≈ 0.49) with substantially better speech quality than dedicated anonymization baselines and no language-specific training.

Release Date: August 27, 2026

Feature	Value
Parameters	XTTS v2 checkpoints (~2 GB, downloaded once); no new weights trained
Voice Cloning	✅
Asr	❌
Languages	16 (XTTS v2)
Streaming	✅
License
Base Model	coqui-ai/TTS (XTTS v2)
Pipeline	Whisper transcript → tokenize → XTTS GPT with reference conditioning → HiFi-GAN decode at 24 kHz
Modes	single (one forward pass, seconds) / refine (segment + rescore + regenerate weak segments + crossfade stitch, minutes, GPU) / iterate (N compounding passes, keeps the best-scoring, minutes, GPU)
Donor Selection	explicit file, directory, or comma-separated list; or a voice pool (directory of per-speaker folders, or a CSV manifest) with `most_distant` selection
Selection Metric	ECAPA2 mean cosine similarity; the pool speaker furthest from the target wins, then conditioning on its `select_top_k` least-similar clips (default 10)
Selection Preview	`anonymize select` ranks the pool without loading XTTS
Scoring	WER, BLEU, target_similarity, reference_similarity, overall_quality (weighted 0.05 / 0.05 / 0.30 / 0.60)
Iterate Tradeoff	on the bundled sample, three iterations moved similarity-to-original 0.198 → 0.141 while WER rose 0.00 → 0.07
Cli	`anonymize run` / `anonymize batch` / `anonymize select` / `anonymize config`
Batch	resumable, file-locked checkpoint, `manifest.csv` recording the chosen donor per file, per-file failure isolation
Config Precedence	CLI flag > `--config` file > environment variable > built-in default
Env Vars	`XTTS_MODEL_DIR`, `ANONYMIZER_DEVICE`, `ANONYMIZER_MODE`, `ANONYMIZER_WHISPER_MODEL`, `ANONYMIZER_VOICE_POOL`
Install	`uv sync` then `uv run anonymize download-model`
Api	`from anonymizer import Anonymizer` then `anon.anonymize("interview.wav", reference="donor.wav")`
Hardware	CUDA GPU strongly recommended; CPU is workable for `single` only
Limitations	transcript content is not redacted; language must be set explicitly; quality depends on the donor; Whisper errors propagate; prosody and timing shift; not a formal privacy guarantee
Upstream Models	ⓍTTS v2, VITS, YourTTS, Tortoise, Bark, ~1100 Fairseq models, plus the MelGAN / HiFi-GAN / UnivNet vocoder family

Features: The paper's insight is a capability inversion: a voice cloning model is a voice anonymizer in disguise. XTTS v2 was trained on 27k hours to keep prosody and linguistic content while swapping speaker identity on demand — which is precisely the operation anonymization needs, only with the "speaker" axis pointed at a pseudo-speaker rather than a target identity. Reframing it that way means zero retraining and zero language-specific data, and it is why the released system is a ~300-line façade over an existing checkpoint rather than a new model. Two engineering decisions make it usable rather than merely clever. First, donor choice is automated and per-utterance: a fixed donor is a weak default (if the donor resembles the speaker being anonymized, almost no identity is removed), so the pool mode embeds the target and every candidate with ECAPA2 and picks the speaker furthest away, recording the choice in the batch manifest so a run stays auditable. Second, quality is a knob with a measured cost: refine resynthesizes only weak segments, while iterate compounds conversion — the repo publishes the resulting similarity/WER trade-off rather than claiming a free lunch. The honest framing of the limitation matters too: this hides who is speaking, not what was said, and the authors are explicit that empirical ECAPA2 similarity is not a formal privacy guarantee.

Links:

· · · · · · · · · · · · · ·

TontaubeV1

Description: TontaubeV1 is a multilingual text-to-speech model from TontaubeAI (craitech) designed for expressive voice cloning, long-form generation, and low-latency streaming. Its release contains four causal codebook predictors: CB0 generates semantic audio and duration from text, while progressively smaller CB1–CB3 add acoustic detail. CB0 uses a Qwen3-1.7B-derived transformer trunk and CB1–CB3 progressively shallower Qwen3-0.6B-derived trunks, each with a two-layer audio-token head. The four output streams are decoded with DualCodec, and the inference path uses VibeVoice's acoustic encoder/decoder for continuous reconstruction and streaming. It ships bundled synthetic voices plus zero-shot cloning from up to 60 s of reference audio, with public speaking styles audiobook, conversational, and agentic. Released under the Tontaube Community Model License 1.0, which is explicitly not open-source.

Release Date: August 26, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	7 (English, German primary; Spanish, French, Italian, Dutch, Portuguese secondary)
Streaming	✅
License
Parameters	~2.87B (2,873,962,498; CB0 1.83B + CB1 449M + CB2 327M + CB3 269M)
Architecture	4-stage Qwen3-derived codebook cascade (CB0–CB3) + DualCodec + VibeVoice decode
Styles	audiobook, conversational, agentic

Features: The four-stage codebook cascade (CB0 semantic+duration → CB1–CB3 progressive acoustic refinement) lets a single multilingual model deliver expressive, long-form, low-latency speech with strong zero-shot cloning. On the 1,088 English zero-shot Seed-TTS examples it posts 1.66% mean utterance-level WER (measured with Whisper large-v3 at semantic temperature 0.6), and the RTX-5090 streaming path reaches ~200 ms to first encoded audio.

Links:

· · · · · · · · · · · · · ·

Breeze TTS 2

Description: Breeze TTS 2 is an open-weight text-to-speech model from BreezeBlue / RESONIA built for real-time interaction. It ranks #1 among open-weight models on the Artificial Analysis TTS leaderboard while outperforming frontier proprietary systems. Its open-ended natural-language instruction-following supports reference-free voice design (create a voice from a text description) and reference-guided voice direction (clone a voice while steering tone, emotion, pace, delivery), alongside standard reference-audio voice cloning. Ultra-low-latency streaming reaches 0.32 RTF (≈3.1× real time with the warmed-up fast path) and under 40 ms time-to-first-audio on an NVIDIA H100, emitting 24 kHz PCM. Source code is Apache-2.0; model weights are governed by the BreezeBlue Research and Non-Commercial License (commercial use needs written authorization from RESONIA).

Release Date: August 25, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	2 (English, Chinese)
Streaming	✅
License
Parameters	3B (3,466,363,713)
Architecture	seq2seq backbone + depth decoder + codec with CUDA-graph fast path (no named backbone disclosed)
Highlights	#1 open-weight on Artificial Analysis TTS leaderboard; vocal events inline (laugh/cough)

Features: Pairs natural-language voice control with real-time streaming: a single model handles reference-free voice design (no reference audio needed) and voice direction (clone + steer prosody), and ships a CUDA-graph fast path that hits sub-40 ms TTFA at ~3.1× real time on H100 — open-weight quality that the authors claim exceeds frontier proprietary TTS.

Links:

Additional Tools:

Tool	Type	Link
ComfyUI-Breeze-TTS-2	ComfyUI node	ComfyUI-Breeze-TTS-2

· · · · · · · · · · · · · ·

Sopro v2 Turbo

Description: Sopro (Portuguese for "breath") is a lightweight voice-cloning text-to-speech family. This repo ships sopro-v2-turbo, a 120M-parameter open model that streams and runs comfortably on a laptop CPU or in the browser (ONNX runtime), reaching SOTA-level intelligibility against much larger systems. It supports zero-shot voice cloning from 5–20 s of reference audio, four languages (English, European Portuguese, French, German), and a streaming path with ~300 ms time-to-first-audio on a laptop CPU (0.24 RTF offline / 0.21 RTF streaming on an M3 CPU, 0.07 RTF on H100). Released under Apache-2.0.

Release Date: August 25, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	4 (English, European Portuguese, French, German)
Streaming	✅
License
Parameters	120M (121,574,193)
Deployment	in-browser ONNX runtime; int8 AR weights on CPU; causal vocoder
Architecture	autoregressive TTS + chunked-attention streaming path + causal vocoder (F5-TTS/CosyVoice/Vocos lineage acknowledged)

Features: Packs SOTA-level intelligibility into a 120M footprint that runs in the browser or on a laptop CPU, with a chunked-attention + causal-vocoder streaming path (~300 ms TTFA) — making zero-shot multilingual voice cloning practical for on-device and edge deployment rather than GPU-only serving.

Links:

· · · · · · · · · · · · · ·

CuteTTS

Description: CuteTTS is a lightweight (~230M-parameter) continuous autoregressive TTS model from OPPO that models continuous latents rather than discrete codec tokens, running efficiently on GPUs, CPUs, and Apple silicon. It delivers ultra-low latency — ~40 ms to the first audio chunk and ~9× real-time throughput on an RTX 4090 — with strong speech quality and zero-shot voice cloning (best-in-comparison 78.9 SIM on LibriSpeech test-clean). Multilingual support covers English, Chinese, French, German, and Spanish. A distilled variant (CuteTTS-distill) trades slight quality for further efficiency. Ships with a web demo, Python API, and CLI.

Release Date: August 24, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	5 (English, Chinese, French, German, Spanish)
Streaming	✅
License
Parameters	~230M
Architecture	continuous autoregressive modeling of latents + speaker encoder + audio VAE (discrete-codec-free design)
Variants	CuteTTS, CuteTTS-distill

Features: Autoregressively models continuous latent audio representations instead of discrete codec tokens, eliminating codebook-related artifacts and quantization loss at only ~230M parameters. Combined with a lightweight speaker encoder and audio VAE, this yields best-of-class speaker similarity among compared open models and ~40 ms first-chunk latency while remaining practical for CPU/Apple-silicon inference.

Links:

· · · · · · · · · · · · · ·

Rynsan TTS

Description: Rynsan TTS is a multilingual text-to-speech model that extends k2-fsa/OmniVoice to support Khasi, Garo, and Pnar — languages of Meghalaya, India that have historically had limited representation in modern speech technology. Rather than building a system from scratch, Rynsan retains the multilingual capabilities of the base model while adding speech data for these low-resource Khasic languages. Developed under the Tynrai AI initiative, its broader goal is accessible speech technology for the diverse languages and dialects of Meghalaya, supporting their preservation and use in voice-based applications. A live demo is available at ri.tynrai.in/demo. The repository is gated (manual access approval).

Release Date: August 22, 2026

Feature	Value
Asr	❌
Languages	5+ (English, Hindi + extension languages Khasi `kha`, Garo `grt`, Pnar `pbv`; base OmniVoice supports more)
License
Parameters	~0.61B
Architecture	OmniVoice (k2-fsa) multilingual TTS, extended fine-tune
Base Model	k2-fsa/OmniVoice
Developer	Toiar / Tynrai AI

Features: Extends a modern multilingual TTS foundation to three substantially under-resourced Khasic languages — a rare production-oriented entry for indigenous-language speech tech, aimed at accessibility, education, and language preservation rather than benchmark leadership.

Links:

· · · · · · · · · · · · · ·

Audio8 TTS Preview 0.1B

Description: Audio8 TTS Preview 0.1B is the smallest release in the Audio8 TTS family ("the smallest zero-shot TTS worth running"): a ~170M-parameter generative model plus a separate ~120M-parameter codec decoder, making the complete audio generation stack much smaller than most modern multilingual TTS systems. It supports speech generation and zero-shot voice cloning (reference audio + matching transcript). Primary languages are Chinese and English, with German, Spanish, French, Italian, Japanese, and Korean as experimental/multilingual-evaluation targets. Released under the custom Audio8 Community License v1.0: non-commercial use is free, and commercial use is free only for entities with annual revenue under US$2M.

Release Date: August 19, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	8 (Chinese + English primary; de/es/fr/it/ja/ko experimental)
License
Parameters	~0.17B main model (+ ~120M codec decoder)
Architecture	Audio8 Falcon H1 DualAR — slow AR (semantic tokens) + fast AR (codec codebooks), 10 codebooks × 4096 entries
Audio Codec	bundled 44.1 kHz neural codec (~21.5 frames/s)
Context	up to 2,048 packed text/audio positions
Variants	0.1B (this), 0.6B

Features: Packs practical zero-shot cloning into a ~170M-parameter model using an Falcon-H1-derived DualAR design (slow AR predicts semantic tokens per frame; fast AR predicts the frame's 10 codebooks conditioned on the slow hidden state). On Seed-TTS it posts EN WER 1.662 at only ~0.17B — within reach of 4B+ systems — and ships with its own 44.1 kHz codec so no external codec checkpoint is needed.

Links:

· · · · · · · · · · · · · ·

Kiseki-TTS

Description: Kiseki-TTS is a small, fast Japanese text-to-speech model from telecomadm1145 built on top of Qwen/Qwen3-TTS-Tokenizer-12Hz. It generates discrete neural audio codec tokens at 12.5 Hz (4–6× fewer autoregressive steps than 50–75 Hz codecs) and decodes them to waveform with the Qwen3 TTS codec. The acoustic decoder is a linear-time Mamba2 SSM rather than a self-attention stack, so generation cost is constant per frame — memory does not grow with utterance length and there is no KV cache to manage. Because TTS and ASR were trained jointly in a single multi-task run, the same checkpoint also performs ASR (Japanese speech → text, reading only codec layer 0). It is a single-domain voice (ASMR-style Japanese training data) with no speaker conditioning or voice cloning.

Release Date: August 15, 2026

Feature	Value
Voice Cloning	❌
Asr	✅
Languages	Japanese only (ja)
License
Parameters	~0.41B (0.33B backbone + 78M audio branch)
Architecture	Transformer encoder (12 layers, bidirectional self-attention) + cross-attention → Mamba2 SSM decoder (6 layers, no causal self-attention)
Audio Codec	Qwen3-TTS-Tokenizer-12Hz (12.5 Hz, 16 quantizer layers)
Base Model	Kiseki-1.1-0.3B (seq2seq translation model)
Training Data	telecomadm1145/asmr_archive_qwentts_encoded

Features: The decoder deliberately omits causal self-attention — temporal context is carried entirely by the Mamba2 recurrent state while text conditioning enters through cross-attention whose K/V are computed once during prefill. This yields O(1) state per frame (a fixed SSM tensor plus a 3-frame conv window) instead of an O(T) KV cache, so long-form synthesis degrades gracefully past the ~41 s training ceiling instead of hitting a memory cliff. Combined with the 12.5 Hz codec and a shared multi-token-prediction head that resolves all 16 codebook layers in one trunk pass, the model is both compute- and memory-bandwidth-bound rather than quadratic in length.

Links:

· · · · · · · · · · · · · ·

FireRedTTS3

Description: FireRedTTS3 is a unified speech generation and editing system from the FireRed Team built on semantically enriched continuous speech representations. It ships in two variants: FireRedTTS3-Base (zero-shot voice cloning across 24 languages and 21 Chinese dialects) and FireRedTTS3-Instruct (natural-language voice design and combined semantic + acoustic speech editing in one model). Beyond cloning, it supports instruction-based voice design (no reference audio needed) and editing operations such as insertion / deletion / substitution (semantic) and speed / pitch / volume changes (acoustic).

Release Date: August 5, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	24 (plus 21 Chinese dialects)
License
Architecture	Qwen3 backbone + patch-level diffusion autoregressive (DiTAR) + RedAE codec + CAM++ speaker encoder
Variants	Base (cloning), Instruct (cloning + voice design + editing)

Features: Represents speech with semantically enriched continuous (non-quantized) representations, enabling a single system to do zero-shot cloning, text-driven voice design, and fine-grained semantic + acoustic editing. On Seed-TTS-eval it reaches an average WER/CER of 3.04% with 78.8% speaker similarity; MiniMax-MLS-Test average SIM 84.8%.

Links:

Additional Tools:

Tool	Type	Link
FireRedTTS3-ComfyUI	ComfyUI node	FireRedTTS3-ComfyUI

· · · · · · · · · · · · · ·

Audio8-TTS-Preview-0.6b

Description: Audio8 TTS Preview 0.6B is a 0.6B-parameter multilingual text-to-speech model with zero-shot voice cloning. It uses a DualAR architecture inspired by Fish Audio S2 Pro: a slow AR transformer predicts one semantic token per audio frame, and a fast AR transformer predicts the frame's codec codebooks conditioned on the slow hidden state and preceding codebooks. The bundled 44.1 kHz neural audio codec handles both reference-audio encoding and waveform decoding — no additional codec checkpoint is required. The model supports 11 recommended languages (Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, Spanish) with zero-shot voice cloning from a reference audio clip + matching transcript.

Release Date: July 28, 2026

Feature	Value
Parameters	601,159,424 (0.6B, excluding the codec)
Voice Cloning	✅
Asr	❌
Languages	Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, Spanish (11)
Streaming	❌
License
Architecture	DualAR (slow AR + fast AR), inspired by Fish Audio S2 Pro
Slow Ar	24 layers, width 896, 14 attention heads, 2 KV heads
Fast Ar	4 layers, width 896, 14 attention heads, 2 KV heads
Acoustic Tokens	10 codebooks, 4,096 entries per codebook
Codec	44.1 kHz, 2,048 samples per model frame (~21.5 frames/s), bundled (no external codec needed)
Context Length	up to 2,048 packed text/audio positions
Sample Rate	44,100 Hz
Inference	transformers with trust_remote_code=True; CUDA-capable GPU recommended
Dependencies	torch>=2.5.0, torchaudio>=2.5.0, transformers>=4.57.0,<5, soundfile, safetensors
Preview Status	language coverage intentionally limited; broader multilingual + Chinese dialect support planned
Library Name	transformers (custom_code)
Pipeline Tag	text-to-speech
Createdat	2026-07-28T07:53:00Z

Features: The DualAR architecture is the technical centerpiece: rather than a single autoregressive decoder predicting all codebook levels sequentially (the standard codec-LLM TTS pattern), Audio8 splits the work into a slow AR that predicts one semantic token per audio frame and a fast AR that predicts the frame's remaining codec codebooks conditioned on the slow hidden state. This separation lets the semantic-level reasoning happen at the slow AR's 24-layer depth while the acoustic codebook prediction stays lightweight at 4 layers — reducing the total compute per frame without sacrificing semantic quality. The bundled 44.1 kHz codec (no external codec checkpoint needed) and the 10-codebook / 4,096-entry acoustic token design give the model self-contained high-fidelity output at a compact 0.6B scale, making it one of the smallest multilingual zero-shot-cloning TTS systems shipping at 44.1 kHz.

Links:

· · · · · · · · · · · · · ·

NeuTTS-2E

Description: NeuTTS-2E is a super-fast, highly realistic, on-device emotional text-to-speech model from Neuphonic. It is the next generation after NeuTTS Air / Nano (which continue to ship for multilingual + zero-shot-cloning contexts) — narrowed in scope to an English-only alpha focused on:

Release Date: July 21, 2026

Feature	Value
Parameters	0.2B (compact LM backbone + codec)
Voice Cloning	❌
Asr	❌
Languages	English (English-only alpha)
Streaming	✅
License
Backbone	compact LM backbone tuned for emotional TTS token generation
Codec	efficient codec (compact, paired with the LM)
Speakers	4 fixed (`emily`, `paul`, `sophie`, `steven`)
Emotions	6 + neutral (`angry`, `disgusted`, `fearful`, `happy`, `sad`, `surprised`, `neutral`)
Emotion Control Mode	single-argument selection (no composable multi-axis axes like Scylla's Band)
Input Format	text only — no phonemizer, no system dependencies
On Device	yes (laptop-class CPU real-time / better-than-real-time)
Distribution Formats	safetensors (torch), Q4 GGUF, Q8 GGUF
Formats In Collection	`neuphonic/neutts-2e` (safetensors), `neuphonic/neutts-2e-q4-gguf` (smallest footprint), `neuphonic/neutts-2e-q8-gguf` (mid-tier compression)
Gguf Features	imatrix, conversational, endpoints_compatible
Pipeline Tag	text-to-speech
Library Name	(HF tag does not declare transformers / safetensors stem beyond safetensors itself)
Downloads	194 / 241 / 216 (torch / q4 / q8)
Intended Use	embedded voice agents, on-device assistants, toys, privacy-sensitive applications
Comparison With Air Nano	Air/Nano continue to ship for zero-shot cloning + multilingual contexts; 2E is the next-gen focused English emotional variant
Safety Note	model is alpha; legitimate project landing is `neuphonic.com` (not `neutts.com`)

Features: The technical center of NeuTTS-2E is maximum speed per parameter at on-device budgets — the 0.2B LM + codec pair delivers real-time-or-better on laptop-class CPUs while exposing discrete categorical emotion control (angry / disgusted / fearful / happy / sad / surprised / neutral) plus a fixed four-speaker cast for consistency in agent / toy / accessibility voice personas. Two design choices distinguish it from the surrounding TTS field:

First, the categorical emotion surface is single-axis and discrete (one emotion per call), not the continuous multi-axis composable vector surface used by models like Scylla's Band ([neurotica base + 6-axis continuous strengths]). The project's positioning — production-grade agents + toys + accessibility — benefits from a one-argument API where emotion="happy" is the explicit operational state. The release locks emotional mode at generation time, which simplifies downstream filtering / guardrails.

Second, the distribution-shape design (one model, three deployment formats) is a deliberate on-device-first posture: the safetensors torch build for max-quality GPU/server; Q8 GGUF for mid-tier compression; Q4 GGUF for the small-footprint embedded target. All three are direct llama.cpp-compatible drops of the same model — no retraining-per-format — letting users pick size vs quality at deployment time without changing the production API. The combined CPU-first + GGUF-first design pattern is the opposite of the cloud-first TTS systems in this list — and is what makes 2E suitable for embedded voice agents, toys, and privacy-sensitive applications where audio + text must remain on-device.

Links:

· · · · · · · · · · · · · ·

Scylla's Band

Description: Scylla's Band is a multilingual, multi-voice, expressive TTS model from Spybyscript, designed specifically for local and self-hosted inference through ONNX Runtime (with an experimental LiteRT backend for explicit native / mobile use). The architecture is a continuous-latent TTS family:

Release Date: July 19, 2026

Feature	Value
Parameters	not stated (architecture: 4-layer duration predictor (192 hidden) + 12-layer rectified-flow acoustic generator (512 hidden, AdaLN, QK norm))
Voice Cloning	❌
Asr	❌
Languages	en_us, en_gb, es, it (4 public text-input languages)
Streaming	✅
License
Sample Rate	24,000 Hz
Managed Voices	10 (ariadne, felix, gwen, ink, max, orpheus, rex, scylla, stone, tuesday)
Voice Default Locale	en_us for most; ink / orpheus / tuesday default to en_gb
Voice Style Dim	128 (style features) + 32 (prosody features)
Affect Axes	6 (calm, joy, anger, sadness, sarcasm, questioning — all continuous in `[0, 1]`)
Affect Overlay Axes	sarcasm, questioning (mixable with any core delivery)
Affect Cfg Scope	duration + acoustic-flow prediction (preserves voice / reference)
Encoders Default	ONNX Runtime (Python CLI / Python API / Android sample / `libscyllasband` native)
Encoders Experimental	LiteRT (experimental / explicit-selection)
Cli Quality Default	8-step Heun sampling
Graph Budgets	512 G2P text tokens / 512 phone frames / 640 latent frames
Latent Target Buckets	256 / 384 / 512 / 640 (smallest-fit selection)
Vocoder	Scylla's Band acoustic adapter + frozen `charactr/vocos-mel-24khz`
Hop Lengths	256 (waveform) / 512 (latent)
Text Frontend	phrase-level multilingual G2P (74-phone vocabulary)
Span Context	3 segments over up to 768 phones with 512-dim context state
Prefix Context	up to 24 acoustic latent frames from preceding chunk
Long Form Features	boundary metadata + punctuation pause floors + prefix-latent carryover + span context
Group Speak Input	`[voice]`, `[voice:language]`, `[voice:language:axis=value,...]` annotations
Bundle Contract	1.0.0 / `scyllasband-duration-flow`
Intended Use	single-voice speech synthesis (10 voices); en/es/it; long-form narration; multi-voice dialogue from tagged text; continuous affect control; ONNX desktop/server; ONNX + LiteRT native/mobile
Not Intended	arbitrary-speaker cloning / impersonation / fraud / deceptive speech
Distributions	training data, trainer checkpoints, and export tooling not distributed
Cli Commands	download, validate-bundle, list-voices, normalize-text, speak, group-speak, stream, plan
Library Name	onnxruntime (tags include onnx, tflite, litert, duration-flow)

Features: Three design decisions distinguish Scylla's Band in the multilingual TTS class. First, decoupling duration and acoustic flow as separate rectified-flow stages — duration is a 192-hidden, 4-layer predictor operating on a 512-phone window, acoustic latents a 512-hidden, 12-layer AdaLN / QK-norm generator at 24-dim. This split lets affect-CFG act on both stages independently while retaining voice / reference conditioning, supporting the 6-axis continuous composability. Second, 6 affect axes (with sarcasm and questioning as overlays mixed with any core delivery) instead of mutually-exclusive discrete emotion classes — calm=0.5, joy=0.5 is a valid input, and axes stay in [0, 1] so multi-axis states are expressible without combinatorial blow-up. Third, the ONNX-first runtime design with libscyllasband native + an experimental LiteRT backend sits at a budget most neural TTS systems don't target — the 8-step Heun default and 512/512/640 fixed graph budget keep the model usable on CPU and mobile, and the inference-only release surface (training data + checkpoints not distributed) is the complement of the latency / mobile inference focus.

Links:

· · · · · · · · · · · · · ·

sanoTTS

Description: sanoTTS is the smallest known neural text-to-speech family. The name sano (सानो) is Nepali for "small". Each voice weighs 294k to 2.3M parameters — smaller than the smallest voice in prior families (TinyTTS at 1.62M; Inflect Nano at 4.63M; Kokoro at 82M) and the family fits in under 4 MB per voice with zero runtime dependencies (the espeak-ng phonemizer is bundled). Voices run real-time on a ~$3 ESP32-S3 microcontroller (output through a GPIO into an LM386 and a speaker) and live in the browser via WebAssembly — no server, no upload, no NPU. The full neural stack is duration → acoustic → decoder, quantized to int8, with the espeak-ng phonemizer included. 11 voices across 6 languages ship: English, Nepali, Hindi, Vietnamese, Indonesian, and Chinese — including the 294k heart-nano voice (337 KB) and the mel-based heart / heart-nano pair that predicts a 100-band spectrogram rendered by a noise-fed ConvNeXt + iSTFT decoder at 24 kHz. The inference runtime was relicensed MIT (September 2026); the project as a whole remains GPLv3 via espeak-ng. The project page at ampixa.github.io/sanoTTS hosts a live browser synthesis demo for every voice.

Release Date: July 13, 2026

Feature	Value
Parameters	294k–2.3M per voice (smallest = the 294k "heart-nano" voice, 337 KB)
Voice Cloning	❌
Asr	❌
Languages	English, Nepali, Hindi, Vietnamese, Indonesian, Chinese (6 languages, 11 voices)
Streaming	❌
License
Architecture	full neural stack — duration model → acoustic model → decoder
Quantization	int8 (W8/A12, corr 0.9995+; piperlite portable C99)
Runtime Microcontroller	ESP32-S3 (real-time RTF 0.41, GPIO → LM386 → speaker)
Runtime Browser	WebAssembly (no server, no upload, no NPU)
Runtime Footprint	under 4 MB per voice, zero dependencies
Voices	11 (English: amy / kristin / hfc / amy-1p1m / amy-1p8m / robot / heart / heart-nano; one voice each for NE / VI / ID / ZH + shared lang voices)
Phonemizer	espeak-ng (bundled)
License Split	inference runtime MIT; project overall GPL-3.0 (copyleft from espeak-ng)
Library Name	sanotts
Training Method	distillation (per voice)

Features: The hard constraint — be the smallest neural TTS family known, real-time on a $3 microcontroller — drives the entire stack. Conventional sub-100M TTS systems are too large for an ESP32's flash and RAM. sanoTTS keeps the full duration → acoustic → decoder neural pipeline (no espeak-NG-only fallback, no concatenative hybrid), quantizes everything to int8, and bundles the phonemizer so the whole voice ships in under 4 MB with zero runtime dependencies. The newest heart / heart-nano voices switch to a mel-based recipe (100-band spectrogram + noise-fed ConvNeXt + iSTFT decoder at 24 kHz), bringing the smallest voice down to 294k parameters / 337 KB — a per-voice footprint 100× smaller than Kokoro and 2× smaller than TinyTTS while still leading SCOREQ / UTMOS in the sub-15M class — and the demo synthesizes every voice live in the browser via WASM, so the smallest-known neural TTS is also the only one that runs unattended on a $3 chip and a $0 web page.

Links:

· · · · · · · · · · · · · ·

WordVoice-base-0.5B

Description: WordVoice reframes LLM-based TTS from implicit to explicit control: instead of asking the model to "just sound right", the user states what should happen acoustically, word by word. Built on the CosyVoice3 framework, it supports independent, decoupled control of five acoustic dimensions per input word: ⏱️ duration (per-word pronunciation length), ⏸️ boundary (a 5-level pause class b0–b4), 🔊 energy (loudness, 0–1), 🎵 pitch (core f0, −1–1) and 📈 tone (7 prosodic morphologies — flat, rise, strong rise, fall, strong fall, peak, valley). The mechanism is an "acoustic thinking" bound token (<b>): before the autoregressive model emits the speech tokens for a given word, it first predicts that word's acoustic attributes — plan the prosody, then generate the sound. Two inference modes are shipped: Free Mode, where the LLM performs the prosodic planning itself for reference-conditioned zero-shot speech, and Control Mode, where the caller supplies the attributes. Weights are released per language — wordvoice_llm_en.pt and wordvoice_llm_zh.pt alongside a shared flow-matching module — trained on the WordVoice-5A dataset, whose five "dimensions" are the five controllable attributes.

Release Date: July 8, 2026

Feature	Value
Parameters	0.5B (base and name); shipped as three checkpoints — flow matching 1.33 GB plus separate EN and ZH LLM weights at 2.03 GB each
Voice Cloning	✅
Asr	❌
Languages	2 (Chinese and English, with separate per-language LLM checkpoints)
License
Base Model	FunAudioLLM/Fun-CosyVoice3-0.5B-2512 (Qwen2.5-family LLM backbone per the project README)
Control Dimensions	duration, boundary (b0–b4), energy (0–1), pitch (−1 to 1), tone (7 categories)
Tone Categories	flat, rise, strong rise, fall, strong fall, peak, valley
Mechanism	bound token (`<b>`) — the AR LLM predicts a word's acoustic attributes before generating its speech tokens
Modes	Free Mode (model plans prosody for zero-shot speech) and Control Mode (caller supplies per-word attributes)
Training Data	WordVoice-5A (released alongside the model, with a separate preprocessing pipeline at XXH333/WordVoice-5A-Pipeline)
Datasets Note	speaker count, hours and utterance counts are not stated in the released documentation
Benchmarks	none — no WER, MOS, speaker-similarity or control-accuracy numbers are published; quality claims are qualitative
Sample Rate	not stated in the released documentation
Install	conda environment, `pip install -e .`, then `bash download_models.sh` (pulls CosyVoice3 and MMS-FA dependencies)
Inference	`python wordvoice_infer.py` for the out-of-the-box Free Mode / Control Mode demo
Release Notes	open-sourced with paper, model, dataset and pipeline on 2026/07/08; interactive Hugging Face Space demo added 2026/07/29

Features: Nearly every LLM-TTS system makes prosody a latent: the language model is asked to produce fluent speech and the timing, pauses, energy and intonation emerge as a side effect, so the user can influence them only through prompt text — and cannot ask for "a long pause after this word" or "flat and quiet here" in any reproducible way. WordVoice's move is to make those attributes first-class, discrete, per-word outputs that the user writes down. The subtle part is where the control lives: the bound token means the attributes are predicted by the autoregressive LLM itself, autoregressively, before the corresponding speech tokens are generated — "acoustic thinking". That ordering is what makes the control compositional rather than a separate classifier bolted on afterward; the model reasons about the prosodic shape of the word it is about to say, in context with the words around it. The five-dimension decomposition (duration / boundary / energy / pitch / tone) is also a deliberate interface choice — boundary is a category while energy and pitch are scalars, which is not how a generic "emotion" tag behaves, and is far more useful for subtitle and dubbing work. Worth weighing alongside the controls is what the release does not include: no objective results, no MOS, no speaker-similarity figure, no stated reference-audio protocol, and no sample rate. The controls are clearly specified and the code is open; the accuracy of the control and the fidelity of the zero-shot cloning are unmeasured in the published artifacts.

Links:

· · · · · · · · · · · · · ·

FreyaTTS

Description: FreyaTTS is a 183M-parameter Turkish text-to-speech model. It is tokenizer-free at the character level — 92 symbols in its Turkish vocabulary — so there is no phonemizer or G2P step in either training or inference. Speech is generated with a non-autoregressive conditional flow-matching DiT in the frozen AudioVAE2 latent space (25 Hz, 64-dim latents, 16 kHz encode / 48 kHz decode). Training runs from scratch on Turkish speech: a pretraining stage followed by SFT stage 1/2 for voice lock and short-utterance coverage. Output is 48 kHz mono. On the project's Freya-TR-Eval benchmark the model reports WER 8.0% / CER 3.0%, ranking 3rd of 7 among open sub-1B Turkish TTS systems — a deliberate single-target-speaker, no-cloning design choice for a focused foundation release. The evaluation dataset is freya-tr-eval.

Release Date: July 7, 2026

Feature	Value
Parameters	183.2M
Voice Cloning	❌
Asr	❌
Languages	Turkish (tr)
Streaming	❌
License
Architecture	conditional flow-matching diffusion transformer (DiT), non-autoregressive, 32-step Euler ODE, no CFG
Tokenizer	character-level (92 Turkish symbols; no phonemizer, no G2P)
Latent Space	frozen AudioVAE2 (Apache-2.0, openbmb/VoxCPM2), 64-dim at 25 Hz
Codec Io	16 kHz encode / 48 kHz decode
Sample Rate	48,000 Hz
Training	from scratch on Turkish speech; pretraining + SFT stage 1/2 (voice lock + short-utterance coverage)
Evaluation	Freya-TR-Eval — WER 8.0% / CER 3.0%, 3rd of 7 open sub-1B Turkish TTS
Library Name	freyatts

Features: Two design choices are worth flagging. First, tokenizer-free character-level Turkish: by training directly on the 92-symbol Turkish alphabet with no phonemizer or G2P grapheme-to-phoneme step, the model removes a dependency that is fragile for agglutinative Turkish morphology and that often degrades quality when ported to low-resource Turkic relatives. Second, non-autoregressive conditional flow-matching in a frozen AudioVAE2 latent space: the 25 Hz / 64-dim bottleneck keeps the DiT small (183M) while inheriting a separately-trained audio codec's representation, letting a focused single-language-non-multilingual release ship at a fraction of the parameter budget of multilingual foundation TTS systems. The deliberate "no cloning, single target speaker" choice is a scope-lowering move that lets the foundation release put all its capacity into Turkish speech quality rather than spread it across zero-shot speaker adaptation.

Links:

· · · · · · · · · · · · · ·

Inflect-Nano-v2

Description: Inflect-Nano-v2 is a complete local text-to-waveform speech synthesis model with 3,966,721 deployable parameters — under 4M total. It is a VITS-architecture fixed-voice English TTS designed for CPU or CUDA inference with deterministic seeds, long-text handling, and 24 kHz mono output. The full FP32 checkpoint is 15.97 MB, making it one of the smallest complete neural TTS systems that produces natural-sounding speech without a separate vocoder or phonemizer dependency. The model ships with a public adaptation toolkit for preparing data, auditing train/validation splits, adapting a fixed voice or language, resuming training, evaluating checkpoints, and exporting PyTorch or ONNX packages. A sibling Inflect-Micro-v2 (9.36M parameters) prioritizes quality below 10M; Nano prioritizes footprint below 4M. Both share one public API.

Release Date: June 25, 2026

Feature	Value
Parameters	3,966,721 (3.97M deployable)
Voice Cloning	❌
Asr	❌
Languages	English
Streaming	❌
License
Architecture	VITS (end-to-end text-to-waveform)
Sample Rate	24,000 Hz
Footprint	15.97 MB FP32
Inference	CPU or CUDA; PyTorch + ONNX export
Determinism	deterministic seeds for reproducible generation
Long Text	automatic text splitting and handling
Input Format	text (no phonemizer or system dependencies)
Adaptation Toolkit	data prep, split auditing, voice/language adaptation, training resume, checkpoint eval, PyTorch/ONNX export
Sibling Model	Inflect-Micro-v2 (9.36M params, quality-prioritized below 10M)
Api	one public API across Micro and Nano sizes
Library	pytorch
Metrics	WER
Inference False On Hf	yes (no hosted HF inference endpoint; local-only)

Features: Inflect-Nano-v2's defining constraint is completeness under 4M parameters: the entire text-to-waveform pipeline — no separate vocoder, no phonemizer, no system dependencies — fits in 3.97M deployable parameters and a 15.97 MB FP32 checkpoint. This is smaller than even sanoTTS's smallest voice (745k) when measured by complete-pipeline footprint, though sanoTTS ships per-voice weights rather than a single fixed-voice checkpoint. The VITS end-to-end architecture is the enabler: by folding the acoustic model and vocoder into a single jointly-trained network, Inflect avoids the multi-stage pipeline overhead that makes most neural TTS systems larger. The public adaptation toolkit extends the fixed-voice design into a customizable platform — users can prepare data, adapt a voice or language, resume training, and export PyTorch or ONNX packages — making the 4M-parameter footprint a starting point for domain-specific TTS rather than a dead-end fixed-voice release.

Links:

· · · · · · · · · · · · · ·

Gepard

Description: GEnerative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue. Gepard is built for low-latency, high-throughput streaming conversation: the model starts speaking the moment text begins arriving, generating audio piece by piece instead of waiting for a full sentence. It is a single decoder-only autoregressive language model built on Qwen3.5 (14 layers, hidden 1024, 8 heads) with ≈556M total parameters (backbone + audio interface + voice-cloning compressor). Audio is produced through NVIDIA NeMo NanoCodec — Finite Scalar Quantization at 22.05 kHz, 21.5 frames/s, 1.89 kbps — with the full 32-channel FSQ frame sampled in one step. Reports ~25× real time on a single RTX 5090 with first-audio-chunk latency around 50 ms; a 96 GB Blackwell card serves up to 256 concurrent conversations. CFG refinement is baked into the weights so quality gain comes at no extra two-pass cost at inference, though the two-pass mode is still selectable as a quality dial.

Release Date: June 22, 2026

Feature	Value
Parameters	~556M (555,694,169; Qwen3.5 backbone + audio interface + voice-cloning compressor)
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	❌
Languages	English (US/UK), Spanish (es-MX), Portuguese (pt-BR), Dutch (NL)
Streaming	✅
License
Audio Codec	NVIDIA NeMo NanoCodec (FSQ, 22.05 kHz, 21.5 fps, 1.89 kbps; NVIDIA Open Model License)
Sample Rate	22,050 Hz
Backbone	Qwen3.5 full-attention transformer (14 layers, hidden 1024, 8 heads; ~500M params)
Inference	vLLM
Throughput	256 conversations on one 96 GB Blackwell (RTX Pro 6000) GPU
Benchmark	Seed-TTS-eval leader on perceived quality (NISQA-MOS 4.25, NOI 4.16, COL 4.16, DIS 4.51) trading some WER/SIM

Features: A prosody-aware autoregressive single-pass frame generator: the whole 32-channel FSQ audio frame is sampled in one step (no depth transformer), and CFG quality refinement is baked into the weights rather than incurred at inference as a two-pass cost — so the publicly reported TTFA of ~50 ms and 25× real time on a single RTX 5090 represent the quality-on path, not a cheap-fast preview. Voice cloning is decoupled into a separate up-front compressor, which means cloning is "free" at run-time once the reference clip is encoded — a structural choice that supports serving hundreds of conversations per GPU. A stop-head weight update (2026-08-06) fixed premature stopping at sentence boundaries and lifted the effective duration ceiling; on Seed-TTS-eval Gepard leads the compared systems on perceived quality (NISQA-MOS 4.25) while trading some speaker similarity and WER for its streaming-first design.

Links:

· · · · · · · · · · · · · ·

Higgs Audio v3 TTS

Description: Boson AI's flagship conversational TTS: an ~4B autoregressive decoder over interleaved text and audio tokens from the Higgs Tokenizer (8 codebooks at 25 fps / 24 kHz). Built for voice chat rather than narration, it covers 102 languages with zero-shot voice cloning and inline control over emotion, style, prosody, pauses, and sound effects.

Release Date: June 4, 2026

Feature	Value
Parameters	4B (BF16, 36 layers, hidden=2560, GQA 32/8)
Architecture	Autoregressive decoder (Qwen3-style)
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	102 (85 with WER/CER <5, 17 between 5-10)
Streaming	✅
Audio Output	24 kHz
License

Features: Interleaved text/audio token modelling with a delay-pattern multi-codebook embedding/head: a single autoregressive stack emits both modalities and supports inline <|category:value|> control tags (emotion/style/sfx/prosody) inserted at any point in the target text.

Links:

· · · · · · · · · · · · · ·

dots.tts

Description: dots.tts is a 2B-parameter fully continuous, end-to-end autoregressive TTS system from Rednote-HiLab. The backbone pairs a semantic encoder, an LLM, and an autoregressive flow-matching acoustic head over a 48 kHz AudioVAE, with no discrete tokens anywhere in the pipeline. It achieves the best average performance on Seed-TTS-Eval (WER 0.94 / 1.30 / 6.60 on zh / en / zh-hard) and the highest speaker similarity on a 24-language MiniMax multilingual benchmark, with broad cross-lingual voice cloning.

Release Date: June 3, 2026

Feature	Value
Parameters	2B (semantic encoder + LLM + AR flow-matching acoustic head)
Voice Cloning	✅
Asr	❌
Languages	Multilingual (24+ languages; zh / en focus)
Streaming	❌
License
Sample Rate	48 kHz
Tokenizer	48 kHz AudioVAE (continuous, no discrete tokens)

Features: A fully continuous autoregressive pipeline that keeps generation in waveform-latent space end-to-end (no discrete-code phase), pairing an LLM-side semantic encoder with an autoregressive flow-matching acoustic head over a 48 kHz AudioVAE — yielding SOTA seed-TTS-Eval scores and the strongest speaker-similarity number (83.9 avg) on the 24-language MiniMax multilingual benchmark.

Links:

· · · · · · · · · · · · · ·

Confucius4-TTS

Description: Confucius4-TTS is an LLM-based text-to-speech system from NetEase Youdao designed for multilingual and cross-lingual synthesis. It uses a speech encoder + LLM (Text2Semantic) + flow-matching Semantic2Acoustic architecture that allows zero-shot voice cloning without a required reference transcript and explicit cross-lingual voice transfer with unaccented output across languages. Covers Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, and Vietnamese with code-switching and emotion transfer.

Release Date: June 2, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	14 (zh, en, ja, ko, de, fr, es, id, vi, th, pt, it, ru, ms)
Streaming	❌
License
Architecture	speech encoder + LLM (T2S) + flow-matching head (S2A)

Features: Cross-lingual voice transfer without accent drift: the same reference voice stays consistent when the speaker switches languages — backed by a speech encoder + LLM backbone pipeline (T2S) with a flow-matching acoustic decoder (S2A) and training that bundles 14 languages with code-switched, emotion-preserving decoding.

Links:

· · · · · · · · · · · · · ·

WavTTS

Description: WavTTS is an end-to-end zero-shot TTS framework that synthesizes speech directly in the raw waveform space — explicitly skipping the intermediate mel-spectrogram, VAE-latent, or codec-token representations that most modern TTS stacks use. It is built on a flow-matching diffusion transformer (DiT) with waveform patchification, multi-scale mel-spectrogram supervision, and an optimized noise schedule. Forked from F5-TTS at the codebase level but replaces the whole acoustic pipeline.

Release Date: May 28, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	English, Chinese
Streaming	❌
License
Sample Rate	16 kHz
Training Data	Emilia
Architecture	Flow-matching DiT, raw waveform patchification, multi-scale mel supervision
Training Steps	1.2M

Features: Skip every intermediate waveform representation (no mel, no VAE, no codec tokens): a flow-matching DiT produces raw-waveform patches directly, supervised at multiple mel scales and an optimized noise schedule — yielding high-quality zero-shot TTS at 16 kHz from a single end-to-end stack.

Links:

· · · · · · · · · · · · · ·

MOSS-TTS

Description: MOSS-TTS is a production-grade Text-to-Speech foundation model developed by the OpenMOSS Team and MOSI.AI. The current public v1.5 release preserves the original 1.0 capabilities — zero-shot voice cloning, long-form speech generation, token-level duration control, Pinyin/IPA pronunciation supervision, multilingual synthesis, and code-switching — and extends multilingual continued training from 20 languages to 31 languages including Cantonese, Dutch, Finnish, Hindi, Macedonian, Malay, Romanian, Swahili, Tagalog, Thai, and Vietnamese. v1.5 improves speaker similarity, reduces cloning variance on long-reference / short-text scenarios, follows punctuation-driven prosody more reliably, and adds explicit inline pause markers (e.g., [pause 3.2s]).

Release Date: May 25, 2026

Feature	Value
Parameters	8B (Delay), 1.7B (Local)
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	31 (extended from v1.0's 20)
Streaming	✅
License
Max Duration	1 hour
Pause Control	yes (inline markers like `[pause 3.2s]`)
Lang Tag Control	yes (set `language=` in user message)

Features: v1.5 widens MOSS-TTS from 20 → 31 languages with stronger per-language multilingual synthesis (control via a language tag in the user message), more stable cloning under long-reference / short-text conditions, punctuation-driven prosody that holds up across long sentences, and explicit inline pause tokens ([pause 3.2s]) for scripted narration control.

Links:

· · · · · · · · · · · · · ·

VoxFlash-TTS

Description: VoxFlash-TTS is a zero-shot voice-cloning text-to-speech engine built around extreme latent compression. The VAE encodes 24 kHz waveforms into a 9 frames/s latent space — roughly 8× more compressed than EnCodec (75 fps) and 2.4× more than Stable Audio (21.5 fps). Generating 10 s of audio therefore requires the diffusion model to produce just 90 latent vectors rather than hundreds or thousands of tokens, with downstream quadratic savings in attention cost. A ConvNeXtV2-based phoneme encoder followed by a novel coarse-alignment algorithm (cheaper than cross-attention) maps text into the latent sequence; a modern diffusion head then iteratively refines speech latents that the lightweight VAE decoder renders back to waveforms. The architecture targets low-latency, low-resource deployment — consumer-grade GPUs and edge devices — with Chinese and English zero-shot cloning. The project card lists inference: false on HF (no hosted inference endpoint), but the project page at voxflash.github.io carries the abstract, demo examples, and ablations.

Release Date: May 22, 2026

Feature	Value
Parameters	not stated (ConvNeXtV2 phoneme encoder + diffusion head + lightweight VAE decoder)
Voice Cloning	✅
Asr	❌
Languages	Chinese, English
Streaming	❌
License
Audio Codec	VoxFlash VAE (9 Hz / 9 fps latent, 24 kHz input)
Compression Ratio	~8× tighter than EnCodec (75 fps), ~2.4× tighter than Stable Audio (21.5 fps)
Phoneme Encoder	ConvNeXtV2 + coarse-alignment algorithm (no cross-attention)
Diffusion Head	modern multi-step iterative refinement
Decoder	lightweight VAE decoder
Sample Rate	24,000 Hz
Inference	local CUDA ≥ 12.3.2; no HF hosted endpoint
Training Dataset	seed-tts-eval
Metrics	word_error_rate, speaker_similarity

Features: The central technical move is compressing the audio latent space to 9 frames/s instead of the conventional 75 fps (EnCodec) or 21.5 fps (Stable Audio). This is not a quantization tweak — it is a temporal-decimation architectural choice that shrinks the sequence length the diffusion model has to traverse, and because attention cost scales quadratically with sequence length the end-to-end compute drops by orders of magnitude. Combined with a coarse-alignment phoneme-to-latent map that avoids cross-attention entirely (using a ConvNeXtV2 phoneme encoder instead), VoxFlash hits millisecond-level inference latency on consumer-grade and edge hardware for zero-shot Chinese + English cloning, where conventional latent-diffusion TTS systems are too slow for real-time edge deployment.

Links:

· · · · · · · · · · · · · ·

Miso TTS

Description: Miso TTS 8B is a text-to-speech model from Miso Labs built on the Sesame Conversational Speech Model (CSM) architecture. A large Llama-3.2-style backbone consumes text/audio-frame embeddings and predicts codebook 0 of the Mimi audio token stream, while a smaller 300M autoregressive audio decoder predicts codebooks 1–31 in codebook depth. The model is designed for high-quality conversational speech and voice continuation from a short prompt audio clip.

Release Date: May 21, 2026

Feature	Value
Parameters	8B (backbone llama-3.2-style) + 300M (audio decoder) = 8.3B
Voice Cloning	✅
Asr	❌
Languages	English
Streaming	❌
License
Architecture	Sesame-style CSM (two transformer stack: backbone + audio decoder)
Audio Tokenizer	Mimi (32 codebooks, vocab 2051, max seq 2048)
Library	pytorch

Features: A two-transformer Sesame-style CSM (Llama 8B backbone consumes text + audio frames and produces backbone codebook-0 prediction; a 300M audio decoder autoregresses over codebook depth via Mimi's 32-codebook stack) — letting the larger backbone spend capacity on linguistic / speaker conditioning while a leaner decoder handles fine-grained codebook-by-codebook generation.

Links:

· · · · · · · · · · · · · ·

Raon-OpenTTS-1B

Description: Raon-OpenTTS is an open-data, open-weight zero-shot TTS system from KRAFTON that performs on par with state-of-the-art closed-data models. This is the 1B variant (1048M parameters). Both model weights and training data are public: Raon-OpenTTS-Core is 510.1K hours of English speech, quality-filtered from the 615K-hour public Raon-OpenTTS-Pool using combined DNSMOS, WER, and VAD rank-based filtering. It ranks 1st or 2nd in WER and SIM among recent zero-shot TTS models on Seed-TTS-Eval and CV3-Eval, and achieves the best average WER/SIM on Raon-OpenTTS-Eval across Clean, Noisy, Wild, and Expressive regimes. A smaller Raon-OpenTTS-0.3B variant is also available.

Release Date: May 21, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	English only (trained on 11 English speech datasets)
License
Parameters	1048M
Architecture	DiT (Diffusion Transformer) based on F5-TTS with flow matching; dim=1408, depth=28, heads=24
Audio Output	80-ch mel-spectrogram at 16 kHz, HiFi-GAN vocoder (LibriTTS)
Training Data	Raon-OpenTTS-Core (510.1K hours), 520K updates on 48× B200

Features: Demonstrates that fully open data + open weights can match proprietary SOTA: on Seed-TTS-Eval it reaches 1.78 WER / 0.749 SIM (vs Qwen3-TTS 1.46/0.715 at 1.7B), and best overall robustness (WER 2.81 / SIM 0.695) across four acoustic regimes on its own Raon-OpenTTS-Eval benchmark. The pipeline pairs large-scale rank-based data curation (DNSMOS + WER + VAD filtering of a 615K-hour pool) with an efficient F5-TTS-derived DiT.

Links:

Additional Tools:

Tool	Type	Link
ComfyUI-Raon-OpenTTS	ComfyUI node	ComfyUI-Raon-OpenTTS

· · · · · · · · · · · · · ·

OronTTS

Description: OronTTS is a non-autoregressive text-to-speech model from btsee, an F5-TTS fork specialized for Mongolian (Khalkha Cyrillic) and Kazakh (Cyrillic). It uses Flow Matching + Diffusion Transformer + Vocos (dim 1024, depth 22, 16 heads, vocab 65, 24 kHz sample rate), trained on the btsee/mbspeech_mn corpus (3,846 Mongolian speech samples) and outputs zero-shot synthesis from a short reference audio + language tag.

Release Date: May 16, 2026

Feature	Value
Parameters	(not stated)
Voice Cloning	✅
Asr	❌
Languages	Mongolian (Khalkha Cyrillic), Kazakh (Cyrillic)
Streaming	❌
License
Architecture	F5-TTS (OT-CFM + DiT + Vocos)
Dim	1024
Depth	22
Heads	16
Vocab Size	65
Sample Rate	24000 Hz
Mel Bins	100
Training Data	btsee/mbspeech_mn (3,846 Mongolian speech samples)

Features: F5-TTS re-purposed for low-resource Cyrillic languages (Mongolian + Kazakh) — non-autoregressive flow-matching DiT over a tight 65-word vocab. Trained on a small (~3.8k sample) Mongolian corpus; the architecture is small enough that Khalkha Cyrillic and Kazakh Cyrillic share the same checkpoint via the lang tag at inference time.

Links:

· · · · · · · · · · · · · ·

Supertonic 3

Description: Supertonic 3 is the third-generation open-weight release from Supertone. It is a lightweight, on-device text-to-speech system that runs with ONNX Runtime entirely on the user's machine (no network, no API call) and ships as a Python SDK (pip install supertonic). Compared with the Supertonic 2 base (5 languages, 66 M params), v3 expands to 31 languages and adds expression tags (<laugh>, <breath>, <sigh>), more stable reading on long utterances, and higher speaker similarity across the core language set.

Release Date: May 6, 2026

Feature	Value
Parameters	(not stated on card; Supertonic 2 baseline 66 M — likely similar or smaller weight class)
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	31 (expanded from Supertonic 2's 5)
Streaming	✅
License
On Device	yes (ONNX Runtime, no cloud call)
Expression Tags	yes (`<laugh>`, `<breath>`, `<sigh>`)

Features: A more compact on-device multilingual TTS: ONNX-Runtime inference everywhere, 31 languages from a single small open-weight encoder, and discrete expression tags that the decoder interprets inline — without a separate speaker-emotion control path or a cloud-rendered audio round-trip.

Links:

· · · · · · · · · · · · · ·

Scenema Audio

Description: Scenema Audio is a zero-shot expressive voice cloning and speech generation model from ScenemaAI. It is built on an audio diffusion transformer extracted from the audio branch of Lightricks' LTX 2.3 (a 22B audiovisual model) — keeping the in-the-wild acoustic quality the bigger model learned while specializing for speech output. Generation is prompt-driven: a <speak> tag carries a voice description, gender, optional scene (ambient audio around the voice), and language; an <action> tag shifts emotional state mid-generation. Action tags cover rage, grief, joy, fear, exhaustion; voice prompt can describe timbre/pitch/breathiness/rasp/resonance plus character archetypes ("Tony Soprano having a breakdown"). Supports zero-shot voice cloning from 10-20 seconds of reference audio with some emotional variability, automatic long-form narration by splitting text and maintaining voice continuity, and 13 multilingual built-ins.

Release Date: April 26, 2026

Feature	Value
Parameters	(audio diffusion transformer of LTX 2.3, weights ~9.8 GB bf16 / ~4.9 GB INT8 + ~6.7 GB pipeline)
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	13 (en, de, fr, es, it, pt, ja, zh, ko, ru, ar, hi, sw)
Streaming	❌
License
Parent Model	Lightricks LTX-2.3 (audio branch)
Prompt Format	`<speak voice=… gender=… scene=… language=…>` XML with `<action>` tag for shifting emotion
Long Form Narration	yes (auto-splits text while preserving voice continuity)
Quantized	yes (INT8 weights at ~4.9 GB, identical quality)

Features: A standalone audio diffusion transformer extracted from a much bigger multimodal source: the model inherits how people actually sound in real scenes (angry, laughing, whispering, crying, exhausted, terrified) and exposes that capacity through a <speak> + <action> prompt interface — emotional state shifts within a single generation, instead of being a token-level or speaker-level conditioning problem.

Links:

· · · · · · · · · · · · · ·

Dramabox

Description: Dramabox is Resemble AI's expressive TTS, distributed under the LTX-2 Community License. It is an IC-LoRA fine-tune of the LTX-2.3 3.3B audio-only branch (Diffusion Transformer + flow matching), conditioned on Gemma 3 12B text embeddings. Generation is prompt-driven: speaker identity, emotion, delivery, laughs, sighs, breaths, pauses, and transitions are all expressed inside a natural-language description, with an optional 10-second voice reference that clones the target timbre.

Release Date: April 17, 2026

Feature	Value
Parameters	3.3B (LTX-2.3 audio backbone, IC-LoRA fine-tune) + 12B Gemma 3 text encoder (conditioning only)
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	English
Streaming	❌
License
Base Model	Lightricks/LTX-2.3 (audio branch)
Architecture	DiT + flow matching, IC-LoRA fine-tune, Gemma 3 12B text embeddings
Inference Time	~2.5 s / generation (warm server)

Features: IC-LoRA fine-tune of LTX-2.3's audio branch leaves the heavy text-understanding work to Gemma 3 12B and lets the DiT do the expressive rendering — so what's normally multimodal-stage orchestration collapses into a single prompt-driven TTS where speaker identity, emotion, and delivery are encoded in the prompt itself, and the timbre comes from a 10-second voice reference when present.

Links:

· · · · · · · · · · · · · ·

Sarashina2.2-TTS

Description: Sarashina2.2-TTS is a Japanese-centric text-to-speech system from SB Intuitions built on a large language model. It supports both Japanese and English, delivers strong pronunciation accuracy on Japanese text through large-scale end-to-end training, and reproduces a speaker's voice, speaking style, and acoustic characteristics from a short reference clip (zero-shot). Training data is sourced exclusively from legitimately acquired, properly licensed speech archives per the Sarashina Model NonCommercial License Agreement v2.0 (released April 24, 2026).

Release Date: April 16, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	Japanese, English
Streaming	❌
License
Base Model	sbintuitions/sarashina2.2-0.5b-instruct-v0.1
Cross Lingual	yes (Japanese ↔ English, code switching)

Features: Japanese-optimized TTS fine-tuned on responsibly-licensed Japanese training corpora with explicit cross-lingual code-switching to English in a single utterance; reference audio carries speaking style and speaker identity together, so the same prompt yields narration, broadcast, conversation, or customer-service delivery without separate style conditioning.

Links:

· · · · · · · · · · · · · ·

LongCat-AudioDiT

Description: State-of-the-art diffusion-based TTS model operating directly in waveform latent space. Developed by Meituan's LongCat team, it requires only a Waveform VAE and Diffusion backbone, effectively mitigating compounding errors.

Release Date: March 30, 2026

Feature	Value
Parameters	1B / 3.5B
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	❌
Languages	Chinese, English
Streaming	❌
Audio Output	24000 Hz
License

Features: Adaptive Projection Guidance (APG) replaces traditional classifier-free guidance for elevated generation quality. Outperforms Seed-TTS on zero-shot voice cloning benchmarks.

Links:

· · · · · · · · · · · · · ·

VoXtream2

Description: VoXtream2 is a zero-shot full-stream TTS model from the Department of Speech, Music and Hearing at KTH Royal Institute of Technology (Nikita Torgashov, Gustav Eje Henter, Gabriel Skantze). Its headline feature is dynamic speaking-rate control: distribution matching plus classifier-free guidance give fine-grained control of speech rate that can be changed while the model is generating — the paper demonstrates gradual speed-ups/slow-downs and sharp control-signal changes mid-utterance. It runs 4x faster than real time with a 74 ms first-packet latency in full-stream mode on a consumer GPU ("full-stream" meaning it begins speaking from the first word rather than waiting for a text chunk). A translingual capability comes from prompt text masking, so the 3–10 s reference clip can be in any language even though text input is English. The backbone follows the phone/temp/dep former stack of its predecessor VoXtream (arXiv 2509.15969, ICASSP 2026), keeping sink attention for streaming, and was trained on Emilia and HiFiTTS2. Code is Apache-2.0; weights are CC-BY-NC 4.0, and the card prohibits generating someone's speech without consent.

Release Date: March 17, 2026

Feature	Value
Parameters	639,104,000 (462,870,528 model + 176,233,472 depth former, FP32)
Voice Cloning	✅
Asr	❌
Languages	English (acoustic prompts in any language via prompt text masking)
Streaming	✅
License
Architecture	phone_former + temp_former + dep_former_csm (depth former, CSM-style), 16 codebooks, audio vocab 2050, embedding dim 1024, speaker embedding dim 192, sink attention
Speaking Rate	dynamic, adjustable mid-utterance in syllables per second (distribution matching + classifier-free guidance)
Training Data	Emilia + HiFiTTS2 (preprocessed: herimor/voxtream2-train)
Constraints	text max 1000 chars; prompt audio 3-10 s; out-of-scope use forbids generating anyone's speech without consent
Usage	pip install "voxtream>=0.2" (espeak-ng phonemizer required)

Features: Dynamic speaking-rate control is the novelty: instead of a fixed speed set at inference, the target rate (in syllables per second) acts as a continuous control signal handled with distribution matching and classifier-free guidance, and can be updated between generated frames. Combined with full-stream generation — first audio packet in 74 ms at 4x real time on a consumer GPU — and prompt text masking that makes any-language reference audio usable for English synthesis, VoXtream2 extends the low-latency streaming TTS line started by VoXtream into flexibly paced, voice-controllable territory.

Links:

· · · · · · · · · · · · · ·

SILMA TTS

Description: SILMA TTS v1 is a high-performance, 150M-parameter bilingual (Arabic/English) TTS model developed by SILMA AI. Built on the F5-TTS diffusion architecture, it was pretrained from scratch using tens of thousands of hours of high-quality public and proprietary data. It supports instant voice cloning with less than 8 seconds of reference audio (the reference transcript can also be left empty — it is transcribed on the fly), full support for Arabic Tashkeel (diacritics, auto-enriched via CATT when absent), NeMo-based text normalization, and an RTF around 0.12 on an RTX 4090. Released under a commercial-friendly license: code MIT, model weights Apache-2.0. The model is 100% compatible with F5-TTS v1.1.7 tooling for inference and fine-tuning.

Release Date: March 13, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Languages	2 (Arabic MSA/Fusha + English)
License
Parameters	150M
Architecture	F5-TTS Diffusion Transformer with flow matching (pretrained from scratch, F5-TTS v1.1.7-compatible)
Pronunciation	✅
Cost	RTF ≈ 0.12 (RTX 4090)

Features: Brings native-level Arabic synthesis to a 150M footprint: one of the smallest open F5-TTS-family models pretrained from scratch rather than fine-tuned, with first-class Arabic handling (Tashkeel-aware pronunciation via CATT enrichment, NeMo text normalization) alongside English, plus instant zero-shot cloning under fully permissive licensing (Apache-2.0 weights / MIT code).

Links:

· · · · · · · · · · · · · ·

Fish Audio S2 Pro

Description: Fish Audio S2 Pro is a leading text-to-speech model with fine-grained inline control of prosody and emotion. It combines reinforcement learning alignment with a dual-autoregressive architecture for high-quality speech synthesis.

Release Date: March 10, 2026

Feature	Value
Parameters	~10 GB (BF16)
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	80+ (Tier 1: En, Zh, Jp)
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

LongCat-Next

Description: Native multimodal foundation model by Meituan LongCat Team processing text, vision, and audio under a single autoregressive objective. Industrial-strength model with strong speech synthesis and voice cloning.

Release Date: March 2026

Feature	Value
Parameters	3B (MoE A3B)
Voice Cloning	✅
Asr	✅
Pronunciation	✅
Emotion Control	✅
Languages	Chinese, English
Streaming	✅
Audio Output	24 kHz
License

Features: Discrete Native Autoregression Paradigm (DiNA) unifying modalities in shared discrete token space. Combines visual understanding, generation, and audio processing in single model.

Links:

· · · · · · · · · · · · · ·

Voxtral-4B-TTS

Description: Frontier, open-weights text-to-speech model developed by Mistral AI. Designed to be fast, instantly adaptable, and produces lifelike speech with natural prosody and emotional range.

Release Date: March 2026

Feature	Value
Parameters	4B
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	✅
Languages	9 (En, Fr, Es, De, It, Pt, Nl, Ar, Hi)
Streaming	✅
Audio Output	24 kHz
License

Links:

· · · · · · · · · · · · · ·

Blue (Light Blue) TTS

Description: BlueTTS (project page: lightbluetts.com) is a multilingual text-to-speech library. Built around slim ONNX graphs that run on ONNX Runtime with first-class CPU support and optional accelerators — OpenVINO (Intel), CUDA ORT (NVIDIA), TensorRT, and ONNX Runtime stock CPU. Targets five languages — Hebrew, English, Spanish, Italian, German — including inline mixed-language with XML-style tags in the text prompt. Inference is deliverable as a PyPI package (blue-onnx), a Rust crate, or directly from the pinned ONNX graphs on the HF Hub; the v2 release ships a slimmed opset-17 ONNX bundle (notmax123/blue-onnx-v2) that's intended for both FP32 production and the experimental INT8 weight-only fallback.

Release Date: February 27, 2026

Feature	Value
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	❌
Languages	Hebrew, English, Spanish, Italian, German
Streaming	❌
License
Runtime	ONNX Runtime (stock CPU; OpenVINO / CUDA / TensorRT optional)
Speed	"fastest open-source TTS" (per project description)
Graph Format	ONNX opset 17 (slim, full-precision; experimental weight-only INT8 fallback)
Distribution	PyPI + HuggingFace + Rust

Features: A CPU-first multilingual TTS that ships both slimmed ONNX graphs and a Python package where the same code path runs on stock CPU ONNX Runtime by default — and optionally accelerates on OpenVINO / CUDA ORT / TensorRT — so deployment doesn't gate on GPU availability. Languages include Hebrew (with explicit G2P normalization) — a comparatively rare open-source TTS target — plus standard European languages, all from MIT-licensed weights distributed via both Hugging Face and PyPI.

Links:

· · · · · · · · · · · · · ·

KittenTTS

Description: KittenTTS is an open-source realistic text-to-speech model designed for lightweight deployment. It is a state-of-the-art TTS model under 25MB with just 15 million parameters, running without GPU on any device.

Release Date: February 24, 2026 (v0.8.1)

Feature	Value
Parameters	15M-80M
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	✅
Languages	English, Multiple
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Ming-omni-tts

Description: Ming-omni-tts is a high-performance unified audio generation model in the Ming 2.0 series. It uses a custom 12.5 Hz continuous tokenizer and a Patch-by-Patch compression strategy that drives the LLM inference frame rate down to 3.1 Hz, enabling fine-grained control over speech rate, pitch, volume, emotion, and dialect (notably Cantonese at ~93 % accuracy). It supports 100+ premium built-in voices plus zero-shot voice design from natural-language prompts and is the first autoregressive model that jointly generates speech, ambient sound, and music in a single channel.

Release Date: February 11, 2026

Feature	Value
Parameters	16.8B (3B active, MoE; A3B)
Voice Cloning	✅
Asr	❌
Emotion Control	✅
Languages	Chinese, English, Cantonese
Streaming	❌
License

Features: Patch-by-Patch compression drives the inference frame rate to 3.1 Hz, drastically cutting LLM-side latency for podcast-style audio while preserving naturalness. A custom 12.5 Hz continuous tokenizer plus a DiT head jointly produce speech, ambient sound, and music in a single output channel — an "in-the-scene" listening experience rather than TTS-on-top-of-a-track.

Links:

· · · · · · · · · · · · · ·

SoulX-Singer

Description: SoulX-Singer is a high-fidelity, zero-shot singing voice synthesis model for generating realistic singing voices for unseen singers without fine-tuning.

Release Date: February 6, 2026

Feature	Value
Parameters	-
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Mandarin, English, Cantonese
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

SoproTTS

Description: SoproTTS is a lightweight English text-to-speech model with zero-shot voice cloning. It uses dilated convolutions (WaveNet-style) and lightweight cross-attention layers instead of the common Transformer architecture.

Release Date: February 4, 2026 (v1.5)

Feature	Value
Parameters	135M
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	✅
Languages	English
Streaming	✅
License
Rtf	0.05 (CPU M3)
Training-Cost	~$100

Links:

· · · · · · · · · · · · · ·

Qwen3-TTS

Description: Qwen3-TTS is an open-source series of Text-to-Speech models developed by Alibaba Cloud. Supports stable, expressive, and streaming speech generation with free-form voice design.

Release Date: January 22, 2026

Feature	Value
Parameters	0.6B-1.7B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	10 (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian)
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

TADA

Description: TADA is a unified speech-language model from Hume AI built around a Text-Acoustic Dual-Alignment tokenizer: for every text/subword token there is exactly one corresponding speech vector, so the audio stream stays 1:1 aligned with text. As a TTS model, each autoregressive step covers one text token and dynamically determines the duration and prosody for that token, breaking the fixed-frames-per-second constraint that drives most modern TTS backbones. As a speech-language model, it generates a text token and the speech for the preceding token in the same dual step.

Release Date: January 12, 2026

Feature	Value
Parameters	1B (Llama 3.2 1B base)
Voice Cloning	❌
Asr	❌
Emotion Control	✅
Languages	English
Streaming	❌
License
Base Model	meta-llama/Llama-3.2-1B
Tokenization	1:1 text–acoustic dual alignment (one speech vector per text token)
Dynamic Duration	yes (each autoregressive step covers one text token, duration is determined per-token)

Features: A dual-alignment speech–text tokenizer that decouples autoregression from a fixed audio frame rate: each text token owns exactly one speech vector, and the model synthesizes the whole segment for that token in one step, regardless of how long the spoken form is — eliminating transcript hallucination and the latency overhead of constant-frame-rate codecs while staying as compact as Llama 3.2 1B.

Links:

· · · · · · · · · · · · · ·

Irodori-TTS-500M-v2

Description: Japanese Text-to-Speech model based on Rectified Flow Diffusion Transformer. Features emoji-based style and sound effect control by embedding emojis in input text for expressive speech generation.

Release Date: 2026

Feature	Value
Parameters	500M
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	✅
Languages	Japanese
Streaming	❌
Audio Output	48kHz waveform
License

Features: Key Feature: Emoji annotation control - insert specific emojis into text to control speaking styles, emotions, and sound effects.

Links:

· · · · · · · · · · · · · ·

KugelAudio

Description: Open-source TTS for European languages with 7B parameters. Outperformed ElevenLabs in human preference testing.

Release Date: Early 2026

Feature	Value
Parameters	7B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	23 European languages
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

LEMAS-TTS

Description: Part of the LEMAS (Large-scale Extensible Multilingual Audio Suite) project. Zero-shot multilingual TTS with 0.3B parameters supporting 10 languages with word-level precise editing capabilities.

Release Date: 2026

Feature	Value
Parameters	0.3B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	10 (zh/en/de/fr/es/pt/it/ru/id/vi)
Streaming	❌
License
Special-Feature	Word-level editing (LEMAS-Edit)

Features: Built on 150,000+ hours of multilingual speech data with word-level timestamps. Includes LEMAS-Edit for precise word-level speech editing via masked token infilling.

Links:

· · · · · · · · · · · · · ·

MioTTS-2.6B

Description: Lightweight, high-speed LLM-based TTS model for English and Japanese with minimal resource usage.

Release Date: 2026

Feature	Value
Parameters	2.6B
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	❌
Languages	English, Japanese
Streaming	✅
License
Rtf	0.135-0.145

Links:

· · · · · · · · · · · · · ·

MOSS-TTS-Nano

Description: Ultra-lightweight open-source multilingual speech generation model with only 0.1B parameters. Designed for realtime speech generation that runs directly on CPU without GPU.

Release Date: 2026

Feature	Value
Parameters	0.1B
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	❌
Languages	20
Streaming	✅
Audio Output	48 kHz Stereo
License

Features: Pure autoregressive architecture with MOSS-Audio-Tokenizer-Nano. Compresses audio to 12.5 Hz token stream using RVQ with 16 codebooks. Runs on 4-core CPU.

Links:

· · · · · · · · · · · · · ·

NeuTTS

Description: NeuTTS is a collection of open-source on-device TTS models with instant voice cloning. Built off LLM backbones with GGUF format quantizations for efficient on-device deployment.

Release Date: Early 2026

Feature	Value
Parameters	360M (Air), 120M (Nano)
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	❌
Languages	English, Spanish, German, French
Streaming	✅
License
On-Device	yes (GGUF quantizations)

Links:

· · · · · · · · · · · · · ·

OmniVoice

Description: Massive multilingual zero-shot TTS model scaling to 600+ languages. Uses diffusion language model-style discrete non-autoregressive architecture with single-stage text-to-acoustic mapping.

Release Date: 2026

Feature	Value
Parameters	-
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	600+
Streaming	❌
License
Training-Data	581k hours

Features: Simplified single-stage architecture vs conventional two-stage pipelines. Full-codebook random masking strategy with LLM initialization for superior intelligibility. Noise-robust prompt processing.

Links:

· · · · · · · · · · · · · ·

T5Gemma-TTS

Description: Multilingual TTS model with voice cloning and duration control, built on the T5Gemma encoder-decoder LLM architecture. Supports batch generation for multiple audio variations.

Release Date: 2026

Feature	Value
Parameters	2B-2B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	❌
Languages	English, Chinese, Japanese
Streaming	❌
License
Vram	7.6-10.6 GB

Features: PM-RoPE positional encoding with XCodec2 audio codec. Low-VRAM options with CPU offloading. Batch inference efficiency with single encoder pass.

Links:

· · · · · · · · · · · · · ·

TinyTTS

Description: The smallest English TTS model with only 1.6 million parameters. End-to-end neural network achieving ~53x real-time synthesis speed on CPU via ONNX optimization.

Release Date: 2026

Feature	Value
Parameters	~3.4 MB (ONNX FP16)
Voice Cloning	❌
Asr	❌
Pronunciation	✅
Emotion Control	❌
Languages	English
Streaming	✅
License

Features: Ultra-compact architecture optimized for CPU-only deployment. Multi-platform support via Python and Node.js APIs. Works on laptops, edge devices, and embedded systems.

Links:

· · · · · · · · · · · · · ·

VoxCPM2

Description: OpenBMB's next-generation tokenizer-free diffusion autoregressive TTS model with 2 billion parameters. Supports 30 languages with automatic detection, voice design from text descriptions, and high-fidelity voice cloning.

Release Date: 2026

Feature	Value
Parameters	2B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	30 (+ 9 Chinese dialects)
Streaming	✅
Audio Output	48 kHz
License

Features: Tokenizer-free design with LocEnc → TSLM → RALM → LocDiT pipeline. Built-in super-resolution via AudioVAE V2 for 48kHz output.

Links:

· · · · · · · · · · · · · ·

Soprano

Description: Soprano is an ultra-lightweight, on-device text-to-speech (TTS) model designed for expressive, high-fidelity speech synthesis at unprecedented speed. The 1.1 release ships an 80M-parameter backbone that achieves up to 20× real-time generation on CPU and 2000× real-time on GPU, with lossless streaming (<250 ms latency on CPU, <15 ms on GPU), <1 GB memory usage at inference, and infinite generation length (automatic text splitting). Output sample rate is 32 kHz, with widespread device support (CUDA / CPU / MPS on Windows, Linux, and Mac). Inference is production-ready through an OpenAI-compatible endpoint, ONNX, WebUI, CLI, and ComfyUI nodes. The base 1.1 model is ekwek/Soprano-1.1-80M on HuggingFace; a fine-tuning toolkit (soprano-factory) was released January 13 2026 alongside the 1.1 weights — the 1.1 release reports 95% fewer hallucinations and a 63% preference rate over 1.0 (Soprano-80M). A live demo runs on ekwek/Soprano-TTS HF Space.

Release Date: December 22, 2025

Feature	Value
Parameters	80M (Soprano-1.1-80M)
Voice Cloning	❌
Asr	❌
Languages	English (US/UK family voices, per HF Space)
Streaming	✅
License
Sample Rate	32,000 Hz
Inference Targets	OpenAI-compatible endpoint, ONNX, WebUI, CLI, ComfyUI
Performance Cpu	up to 20× real-time
Performance Gpu	up to 2000× real-time
Memory	<1 GB at inference
Text Length	infinite (automatic text splitting)
Devices	CUDA, CPU, MPS (Windows, Linux, Mac)
Training Toolkit	soprano-factory (https://github.com/ekwek1/soprano-factory)
History 1 1	Soprano-1.1-80M released 2026-01-14 (95% fewer hallucinations; 63% preference over 1.0)
History 1 0	Soprano-80M released 2025-12-22

Features: The defining trade-off of this release is extreme on-device efficiency at sub-100M scale: an 80M-parameter backbone hits <250 ms CPU latency and <15 ms GPU for lossless streaming while keeping inference within <1 GB of memory — well under the multi-billion-parameter budget that newer conversational TTS systems require. The release pairs the model with soprano-factory (open-source training/fine-tuning toolkit) so users can build their own voices on top of the same backbone, and one installation can drive OpenAI-compatible / ONNX / WebUI / CLI / ComfyUI inference. The 1.1 update is a measured iteration: 95% fewer hallucinations and a 63% preference over 1.0 at the same parameter budget, so the measurable quality jump ships with no added inference cost.

Links:

· · · · · · · · · · · · · ·

GLM-TTS

Description: High-quality TTS synthesis system based on LLMs from ZhipuAI, supporting zero-shot voice cloning with Multi-Reward Reinforcement Learning.

Release Date: December 11, 2025

Feature	Value
Parameters	-
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Chinese, English
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Echo-TTS

Description: Echo is a 2.4B-parameter diffusion-based diffusion transformer (DiT) text-to-speech model. It conditions on target text and up to two minutes of speaker reference audio, generates Fish Speech S1-DAC latents, and decodes to 44.1 kHz audio. Output length is up to 30 seconds per segment. The model is fast at single-sample generation: on one A100, generating 30 seconds of audio from a 120-second prompt takes ~1.45 seconds (RTF < 0.05) — substantially faster than frontier autoregressive approaches at similar quality. The architecture is a deliberate pivot from the author's prior autoregressive-in-DAC-space model Parakeet, which struggled with semantic-consistency retries and weak voice cloning; Echo's diffusion approach trades off real-time interactivity for fast, high-fidelity zero-shot voice cloning in offline synthesis. Trained via the TPU Research Cloud (TRC). Demo (preview) hosted on jordand/echo-tts-preview HF Space; base model on jordand/echo-tts-base.

Release Date: December 4, 2025

Feature	Value
Parameters	2.4B (DiT)
Voice Cloning	✅
Asr	❌
Languages	English (per demo samples)
Streaming	❌
License
Architecture	diffusion transformer (DiT) in Fish Speech S1-DAC latent space
Max Segment Duration	30 s
Sample Rate	44,100 Hz
Speaker Reference Max	120 s
Performance A100 Rt	30 s output in ~1.45 s (RTF < 0.05)
Audio Codec	Fish Speech S1-DAC
Prior Model	Parakeet (autoregressive in DAC space)
Training Infrastructure	TPU Research Cloud (TRC)
Inference Requirements	CUDA-capable GPU with at least 8 GB VRAM; Python 3.10+
Sampler	euler CFG with independent guidances for text (3.0) and speaker (8.0); 40 steps; sequence_length 640 default
License Clarification	MIT (per GH repo license)

Features: Echo is a deliberate next-step pivot from autoregressive-in-DAC-space TTS to a full diffusion approach. The author's prior model, Parakeet, generated DAC tokens autoregressively but suffered the classic AR weakness — semantic-consistency retries — and weak voice cloning. Echo keeps Fish Speech S1-DAC latents (so the audio representation is the same proven codec) but moves the generator upstream to a 2.4B DiT operating directly on those latents, conditioned on a long (up to 2-minute) speaker reference. The result: 30-second outputs in ~1.45 s on a single A100 (RTF < 0.05) with high-fidelity zero-shot cloning — fast enough that the "diffusion is too slow" objection no longer applies at the segment length that matters for offline content generation, while the AR class's retry-induced inconsistency is gone by construction.

Links:

· · · · · · · · · · · · · ·

VibeVoice-Realtime

Description: Real-time TTS model from Microsoft with streaming text input and ultra-low latency (~300ms).

Release Date: December 3, 2025

Feature	Value
Parameters	0.5B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Multilingual
Streaming	✅
License
Max Duration	~10 minutes

Links:

· · · · · · · · · · · · · ·

Fun-CosyVoice 3.0

Description: Advanced TTS system based on LLMs for zero-shot multilingual speech synthesis from FunAudioLLM.

Release Date: December 2025

Feature	Value
Parameters	0.5B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	9 + 18+ Chinese dialects
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

LFM2-Audio-1.5B

Description: Liquid AI's first end-to-end audio foundation model with low latency and real-time conversation.

Release Date: November 28, 2025

Feature	Value
Parameters	1.5B
Voice Cloning	✅
Asr	✅
Emotion Control	✅
Languages	English
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Marvis-TTS

Description: Marvis is a conversational real-time streaming TTS from Marvis-Labs. The architecture inherits Sesame's CSM-1B (Conversational Speech Model): a 250M-parameter multimodal backbone that processes interleaved text + audio tokens and a smaller 60M-parameter audio decoder that models the remaining 31 RVQ codebook levels to reconstruct high-quality speech from the backbone's representations. Audio tokens come from Kyutai's mimi codec (RVQ tokens). The dual-transformer split — semantic backbone + small decoder — yields sub-second latency, and the model is built for on-edge / on-device deployment (Apple Silicon / iPad / iPhone / Mac). Two operational choices distinguish Marvis:

Release Date: November 6, 2025

Feature	Value
Parameters	250M (multimodal backbone) + 60M (audio decoder) = 310M total
Voice Cloning	✅
Asr	❌
Languages	English, French, German
Streaming	✅
License
Architecture	dual-transformer CSM-1B (Conversational Speech Model) — multimodal backbone + audio decoder
Audio Codec	Kyutai mimi codec (RVQ tokens; backbone models codebook 0, decoder models codebook 1–31)
Quantized Size	~500 MB (4-bit MLX)
Training Dataset	amphion/Emilia-Dataset
Library Name	transformers, mlx, mlx-audio
Inference Cli	`mlx_audio.tts.generate --model Marvis-AI/marvis-tts-250m-v0.2 --stream --text "..." [--ref_audio ./x.wav]`
Variants In Collection	250m-v0.2, 250m-v0.2-MLX-{4bit,6bit,8bit}, 100m-v0.2 (+ MLX variants), 250m-v0.2-transformers
Emits Text Chunking	no (full-sequence contextual processing)

Features: Two operational choices make Marvis stand out among conversational TTS. First, no regex chunking: most streaming TTS engines pre-split sentences by regex patterns before feeding them to the generator, which can disrupt flow / intonation; Marvis processes the entire text contextually, treating the text as a single interleaved multimodal sequence. Second, the dual-transformer CSM-1B design — a 250M backbone for codebook 0 (semantic) and a smaller 60M audio decoder for codebooks 1-31 (acoustic) — produces a quantized footprint of ~500 MB, enabling on-device Apple-Silicon inference (iPad / iPhone / Mac) with real-time streaming. The architecture makes a high-quality CSM-style TTS with zero-shot cloning actually deployable at the edge, while the official collection's 4 / 6 / 8-bit MLX variants let users trade footprint for fidelity on a per-device basis.

Links:

· · · · · · · · · · · · · ·

IndexTTS2

Description: AI-Enhanced Text-to-Speech System with Intelligent Optimization and self-learning capabilities.

Release Date: November 2025

Feature	Value
Parameters	-
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Chinese, English
Streaming	✅
License
Multi-Speaker	yes (1-4 speakers)

Links:

· · · · · · · · · · · · · ·

Maya1

Description: State-of-the-art speech model for expressive voice generation with natural language voice control.

Release Date: November 2025

Feature	Value
Parameters	3B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	English (Multi-accent)
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Step-Audio-EditX

Description: 3B-parameter LLM-based RL audio model specialized in expressive and iterative audio editing.

Release Date: November 2025

Feature	Value
Parameters	3B (4B BF16)
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Mandarin, English, Sichuanese, Cantonese, Japanese, Korean
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

KaniTTS

Description: KaniTTS is a 370M-parameter two-stage text-to-speech model from nineninesix-ai. The architecture pairs an LFM-2 backbone LLM (Liquid Foundation Model v2 — a non-transformer, structured state-space architecture) with a neural audio codec for output waveform synthesis. The LLM generates compressed audio-token representations and the codec renders them to 22 kHz waveforms, yielding low-latency generation: ~1 s to produce 15 s of audio on a single RTX 5080, with 2 GB GPU VRAM at inference, and MOS 4.3 / WER < 5% quality on the project's benchmarks. Languages covered: English, German, Chinese, Korean, Arabic, Spanish across multiple per-language voices (English, German, Chinese, Korean, Arabic, Spanish each ship a 400M checkpoint; Japanese ships a 370M "Expo-2025-Osaka" variant; a multilingual 370M checkpoint is also available). MLX variants exist for Apple-Silicon inference. The codec is the same author's nemo-nano-codec-22kHz-0.6kbps-12.5fps-MLX (NVIDIA NeMo NanoCodec, MLX-ported to ~12.5 fps / 0.6 kbps). The model is part of the nineninesix-ai family alongside Gepard.

Release Date: September 30, 2025

Feature	Value
Parameters	370M (kani-tts-370m multilingual); 400M per-language (en / de / zh / ko / ar / es); 370M (expo2025-osaka-ja)
Voice Cloning	❌
Asr	❌
Languages	English, German, Chinese, Korean, Arabic, Spanish (multilingual 370M checkpoint); Japanese (Expo-2025-Osaka variant)
Streaming	✅
License
Sample Rate	22,000 Hz
Backbone Llm	LFM-2 (Liquid Foundation Model; non-transformer structured state-space architecture)
Audio Codec	nineninesix/nemo-nano-codec-22khz-0.6kbps-12.5fps-MLX (NVIDIA NeMo NanoCodec, MLX-ported)
Performance Rt 5080	~1 s for 15 s audio on RTX 5080
Memory	2 GB GPU VRAM at inference
Quality Mos	4.3 / 5 (naturalness)
Quality Wer	<5% (accuracy)
Training Dataset	~80k hours (LibriTTS, Common Voice, Emilia)
Training Hardware	8x H100 GPUs, 45 hours on Lambda AI
Per Language Models	kani-tts-400m-{en,zh,de,ar,es,ko} on HuggingFace
Pretrained Checkpoints	0.2-pt (450M), 0.3-pt (400M) for custom posttraining / fine-tuning
Mlx Variants	kani-tts-370m-MLX (Apple Silicon)
Arxiv	2505.20506

Features: KaniTTS's design choice worth flagging: it pairs a non-transformer LFM-2 backbone (Liquid Foundation Model — structured state-space rather than attention) with a neural audio codec for output at the 370M scale. The choice lets the model hit a ~1 s / 15 s audio generation rate on a 2 GB GPU VRAM budget — sub-1B parameters, sub-entry-tier GPU requirement, but still multilingual across six languages. The two-stage approach (LLM → codec) is conventional; what's less conventional is the choice of a state-space backbone over the usual transformer decoder at this scale, hitting latency / VRAM numbers that open up sub-1B real-time TTS on consumer-grade hardware. The same author ships soprano-factory-style companion assets (pretrained v0.2-pt / v0.3-pt checkpoints, a NeMo NanoCodec MLX port) to lower the bar for fine-tuning on custom datasets.

Links:

· · · · · · · · · · · · · ·

VibeVoice-Finetuning

Description: This is an unofficial, work-in-progress LoRA fine-tuning toolkit for the VibeVoice TTS / speech model (1.5B-base and 7B-base checkpoints). The base VibeVoice checkpoints are the same ones covered in this list's a separate entry: audio-conditioned diffusion TTS for spoken dialogue, streaming, etc. This toolkit takes pretrained VibeVoice weights + a paired (text, audio, optional reference-audio) dataset and trains a LoRA adapter against two losses simultaneously:

Release Date: September 16, 2025

Feature	Value
Parameters	1.5B (LoRA-adapted) / 7B (LoRA-adapted)
Voice Cloning	❌
Asr	❌
Languages	inherits base VibeVoice coverage
Streaming	not directly (toolkit output is a LoRA adapter; the adapter inherits VibeVoice inference shape)
License
Loss Text	masked cross-entropy on text tokens
Loss Acoustic	diffusion MSE on acoustic latents
Hardware 1 5B	≥16 GB VRAM
Hardware 7B	≥48 GB VRAM
Transformers Version	4.51.3 (known-good; other versions may break on Qwen2 architecture)
Tested Docker Image	runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04
Audio Target Format	24 kHz audio (paired dataset of target-audio + transcripts + optional reference-audio prompts)
Training Entrypoint	`python -m src.finetune_vibevoice_lora --model_name_or_path aoi-ot/VibeVoice-Large --processor_name_or_path src/vibevoice/processor --dataset_name <your/dataset> --text_column_name text [--voice_column_name audio_ref]`
Supports Hf Dataset Loader	yes
Output	LoRA adapter compatible with VibeVoice base

Features: The dual-loss trick is the technical center of this toolkit. Naive LoRA fine-tuning of a unified TTS model often specializes the synthesis but silently damages the text LLM capability the base inherited from its Qwen-class backbone; the "masked CE on text tokens + diffusion MSE on acoustic latents" two-headed loss preserves both competencies at training time. Pair that with the careful pinning of Transformers 4.51.3 (other versions break on the Qwen2 architecture) and a documented minimum-VRAM budget per base size (16 GB for 1.5B, 48 GB for 7B), and you get a reproducible recipe for community fine-tuning of VibeVoice — something the official Microsoft VibeVoice release doesn't ship out-of-the-box.

Links:

· · · · · · · · · · · · · ·

VoxCPM

Description: Tokenizer-free TTS system for context-aware speech generation and true-to-life voice cloning.

Release Date: September 16, 2025

Feature	Value
Parameters	640M-800M
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Chinese, English
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

FireRedTTS2

Description: Long-form streaming TTS system for multi-speaker dialogue generation with stable, natural speech.

Release Date: September 2025

Feature	Value
Parameters	-
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	EN, ZH, JP, KO, FR, DE, RU
Streaming	✅
License
Multi-Speaker	yes (4 speakers)
Max Duration	3 minutes

Links:

· · · · · · · · · · · · · ·

Audio Flamingo 3 (AF3) / Audio Flamingo Next

Description: NVIDIA ADLR's fully open-source Large Audio Language Model with state-of-the-art audio understanding. Audio Flamingo Next (AF-Next) is the latest generation featuring stronger general audio understanding, longer context support, and timestamp-grounded reasoning.

Release Date: July 2025 (AF3), 2026 (AF-Next)

Feature	Value
Parameters	7B
Voice Cloning	❌
Asr	✅
Emotion Control	✅
Languages	Multi-lingual
Streaming	✅
License
Context	Up to 30 minutes

Features: Key Innovation (AF-Next): Staged curriculum training with GRPO-based RL post-training. Three specialized checkpoints: Instruct, Think (reasoning), and Captioner. Temporal Audio Chain-of-Thought grounding intermediate reasoning to timestamps.

Links:

· · · · · · · · · · · · · ·

ZipVoice

Description: Fast and high-quality zero-shot TTS models based on flow matching.

Release Date: June 16, 2025

Feature	Value
Parameters	123M
Languages	Chinese, English
License
Zero-Shot-Cloning	yes
Dialogue	yes

Links:

· · · · · · · · · · · · · ·

Fish Speech

Description: State-of-the-art open source TTS and voice cloning model that generates natural, realistic, and emotionally rich speech.

Release Date: May 31, 2025 (v1.5.1)

Feature	Value
Parameters	4B (S1), 0.5B (S1-mini)
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	8 (EN, JP, KO, ZH, FR, DE, AR, ES)
Streaming	✅
License
Rtf	~1:7

Links:

· · · · · · · · · · · · · ·

Chatterbox

Description: Family of SOTA open-source TTS models by Resemble AI, covering a single-language English line plus a multilingual V3 release that brings broader language coverage, more consistent speaker similarity, reduced hallucinations, and more natural conversational speech across 23+ languages.

Release Date: April 24, 2025

Feature	Value
Parameters	500M (Llama backbone, 0.5B)
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	23+
Streaming	❌
License

Features: First open-source TTS model with explicit emotion exaggeration control, plus an alignment-informed inference pipeline and a watermarked decoder. Multilingual V3 narrows the quality gap to closed systems like ElevenLabs on cross-language voice cloning while staying under 1B parameters.

Links:

· · · · · · · · · · · · · ·

Orpheus-TTS

Description: SOTA open-source TTS built on Llama-3b backbone demonstrating emergent capabilities of LLMs for speech synthesis.

Release Date: April 2025

Feature	Value
Parameters	3B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Multilingual
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

MegaTTS3

Description: Advanced zero-shot speech synthesis with Sparse Alignment Enhanced Latent Diffusion Transformer.

Release Date: March 22, 2025

Feature	Value
Parameters	0.45B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Chinese, English
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Spark-TTS

Description: Efficient LLM-Based TTS Model with Single-Stream Decoupled Speech Tokens, built on Qwen2.5.

Release Date: March 2025

Feature	Value
Parameters	0.5B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Chinese, English
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Step-Audio

Description: Production-ready open-source framework for intelligent speech interaction with unified speech comprehension and generation.

Release Date: February 17, 2025

Feature	Value
Parameters	130B (Chat), 3B (TTS)
Voice Cloning	✅
Asr	✅
Pronunciation	✅
Emotion Control	✅
Languages	Chinese, English, Japanese
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Kokoro-82M

Description: Kokoro is an open-weight Text-to-Speech model with 82 million parameters. Despite its lightweight architecture, it delivers comparable quality to larger models while being significantly faster and more cost-efficient. With Apache-licensed weights, Kokoro can be deployed anywhere from production environments to personal projects.

Release Date: January 27, 2025 (v1.0)

Feature	Value
Parameters	82M
Architecture	StyleTTS 2, ISTFTNet
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	8 (54 voices)
Streaming	✅
Cost	<$0.06 per hour of audio
License

Links:

· · · · · · · · · · · · · ·

KokoClone

Description: KokoClone is a fast, real-time compatible multilingual voice cloning system built on top of Kokoro-ONNX. It enables users to type text in multiple languages, provide a short 3-10 second reference audio clip, and instantly generate speech in that same voice.

Release Date: 2025

Feature	Value
Parameters	82M (Base: Kokoro-ONNX)
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	✅
Languages	7 (En, Hi, Fr, Ja, Zh, It, Pt, Es)
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

LuxTTS

Description: Lightweight ZipVoice-based TTS model for high quality voice cloning at speeds exceeding 150x realtime.

Release Date: 2025

Feature	Value
Parameters	-
Voice Cloning	✅
Asr	❌
Pronunciation	❌
Emotion Control	❌
Languages	-
Streaming	✅
License
Rtf	150x
Vram	1GB

Links:

· · · · · · · · · · · · · ·

MiMo-Audio

Description: Audio Language Model by Xiaomi functioning as a Few-Shot Learner with SOTA audio understanding.

Release Date: 2025

Feature	Value
Parameters	7B
Voice Cloning	✅
Asr	✅
Emotion Control	✅
Languages	Multi-lingual
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

SoulX-Podcast

Description: SOTA Multi-Speaker TTS model for generating realistic long-form podcasts with dialectal diversity.

Release Date: 2025

Feature	Value
Parameters	-
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	Mandarin, English, Cantonese, Sichuanese, Henanese
Streaming	✅
License
Max Duration	90+ minutes

Links:

· · · · · · · · · · · · · ·

VieNeu-TTS

Description: Advanced on-device Vietnamese TTS model with instant voice cloning from 3-5 seconds of reference audio.

Release Date: 2025

Feature	Value
Parameters	0.3B-0.6B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	❌
Languages	Vietnamese
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Dia

Description: 1.6B parameter TTS model by Nari Labs for generating ultra-realistic dialogue in one pass.

Release Date: June 27, 2024

Feature	Value
Parameters	1.6B
Voice Cloning	✅
Asr	❌
Pronunciation	✅
Emotion Control	✅
Languages	English
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

MeloTTS

Description: MeloTTS is a high-quality multi-lingual text-to-speech library from MyShell.ai in collaboration with MIT, supporting English (American, British, Indian, Australian, and a default accent), Spanish, French, Chinese (with mixed Chinese–English capability), Japanese, and Korean. Built on VITS / VITS2 / Bert-VITS2 family work and packaged with both a Python API and a Web UI, it runs fast enough for CPU real-time inference.

Release Date: February 19, 2024

Feature	Value
Voice Cloning	❌
Asr	❌
Languages	English (American, British, Indian, Australian, Default), Spanish, French, Chinese, Japanese, Korean
Streaming	❌
License
Base	VITS / VITS2 / Bert-VITS2 family
Mixed Chinese English	yes

Features: A multi-accent multilingual TTS library that ships both a Python API and a Web UI on top of the VITS-style architecture, with explicit English-accent coverage (American, British, Indian, Australian, Default) and mixed Chinese–English output — designed for fast CPU real-time inference without requiring GPU servers.

Links:

· · · · · · · · · · · · · ·

Kimi-Audio

Description: Open-source audio foundation model by Moonshot AI for audio understanding, generation, and conversation.

Release Date: 2024

Feature	Value
Parameters	7B
Voice Cloning	✅
Asr	✅
Emotion Control	✅
Languages	Multi-lingual
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

eSpeak-NG

Description: eSpeak NG is a compact open-source software text-to-speech synthesizer for Linux, Windows, Android, and other operating systems. It supports more than 100 languages and accents, is a fork of Jonathan Duddington's original eSpeak engine, and uses the formant synthesis method: the model produces speech by explicitly computing the acoustic resonances (formants) of each phoneme, not by concatenating human-speech recordings. The trade is well-known — the speech is clear and usable at high playback speeds, but not as natural or smooth as larger neural or concatenative synthesizers. The compensation is size: the program and its data, including many languages, total a few megabytes. Other synthesis methods supported: Klatt formant synthesis and MBROLA diphone back-end via the documented integration.

Release Date: December 8, 2015

Feature	Value
Parameters	n/a (formant-synthesis engine; not a neural model)
Voice Cloning	❌
Asr	❌
Languages	100+ languages and accents (see docs/languages.md)
Streaming	✅
License
Synthesis Method	formant synthesis (primary); Klatt formant synthesis (secondary); MBROLA diphone backend (optional)
Footprint	a few MB (program + data + many languages)
Audio Output	WAV file (CLI), direct playback, or shared-library API
Input Formats	text from file / stdin (CLI), SSML (partial), HTML (partial)
Packages	CLI (espeak-ng man page), shared library (libespeak-ng), SAPI5 Windows module
Supersedes	eSpeak (Jonathan Duddington's original engine)
Downstream Usage	G2P / phonemizer for neural TTS pipelines (e.g. sanoTTS bundles espeak-ng for its duration model)
Platforms	Linux, Windows, Android, Solaris, Mac OS X

Features: eSpeak-NG is the canonical reference implementation of compact multi-language formant synthesis. Its 100+-language coverage in a few megabytes, plus SSML / SAPI5 / MBROLA / shared-library / CLI surfaces, are matched only by neural TTS systems that are orders of magnitude larger. The reason it belongs in a list whose other entries are neural TTS systems is its continued quiet role in the neural stack as a G2P / phonemizer front-end — the phoneme inventory and grapheme-to-phoneme rules that sanoTTS and similar sub-1B neural TTS engines bundle are often just a port of eSpeak-NG's language-data files. So even if the formant-synthesis audio output itself has been surpassed for naturalness, the phoneme infrastructure underneath many of the smaller neural TTS entries on this list still traces back to eSpeak-NG.

Links:

· · · · · · · · · · · · · ·

Anything to Audio

Models that can generate audio from multiple input modalities (video, text, image, audio). These are unified frameworks for multimodal audio synthesis.

Anything to Audio Quick Comparison

Model	Text	Video	Audio	Max Duration	Sample Rate
Ming-3.0-audio-flash	✅	❌	✅	—	—
Ming-3.0-audio-tiny	✅	❌	✅	—	—
MiDashengLM-Gen	✅	❌	❌	—	16 kHz
NVIDIA NemotronLabs VoiceChat 11B	✅	❌	✅	—	22.05 kHz
ScenA	✅	❌	✅	—	—
Nemotron-Labs-Audex-2B	✅	❌	✅	—	—
Nemotron-Labs-Audex-30B-A3B	✅	❌	✅	—	—
MOSS-SoundEffect	✅	—	—	30 s	48 kHz
Omni2Sound (Omni2Audio)	✅	✅	✅	—	—
ControlFoley	✅	✅	✅	—	44,100 Hz
Woosh	✅	✅	—	—	—
Chroma-4B	✅	❌	✅	—	—
Uni-MoE (Audio)	✅	✅	—	—	—
AudioX / Audio-Omni	✅	✅	✅	—	—
HunyuanVideo-Foley	✅	✅	—	—	48 kHz
PrismAudio	—	✅	—	—	—
ThinkSound	✅	—	✅	—	—
MMAudio	✅	✅	—	—	—

Ming-3.0-audio-flash

Description: Ming-3.0-audio-flash is inclusionAI's native audio-language Mixture-of-Experts model: 126B total parameters with 6.5B activated per inference step (128,181,010,400 measured across 25 BF16 shards). It accepts audio together with a text instruction and returns text, optionally with synthesized speech from a separate Talker, so a single model covers the whole loop from hearing to understanding to answering out loud. The backbone is a BailingMoeV3Audio MoE paired with a 32-layer, 1280-wide audio encoder over 128 mel bins with a 3000-token audio context; the Talker speaks with built-in IP voices (灵小柔, 灵楷, US_James, US_Stella) or clones a voice from a reference clip plus its transcript. Recognition is the headline: Chinese CER 3.45% averaged over AISHELL-1/2, Common Voice 15zh, FLEURS and WenetSpeech, English WER 3.69% averaged over LibriSpeech, Common Voice 15 en, FLEURS and VoxPopuli, and 9.40% CER across fifteen Mandarin dialects — with long-form meeting transcription (AA-Earnings22, AISHELL-4, AliMeeting, AMI) handled by the same model. On audio understanding it leads the compared open models on BigBenchAudio (99.40), MMAU (82.00), MMSU (78.82) and MMAU-Pro (66.41). It is served through vLLM-Omni, SGLang and SGLang-Omni — the last exposing OpenAI-compatible transcription, speech and chat endpoints from one server — and is released under MIT.

Release Date: October 19, 2026

Feature	Value
Parameters	126B total / 6.5B active per step (128,181,010,400 safetensors-measured, 25 BF16 shards)
License
Text	✅
Video	❌
Audio	✅
Voice Cloning	✅
Asr	✅
Languages	Chinese and English (varied accents, dialects, noisy and multi-speaker conditions)
Streaming	✅
Architecture	audio encoder (32 layers, hidden 1280, 20 heads, 128 mel bins, 3000 audio-context tokens) + projector + BailingMoeV3Audio MoE text backbone + optional Talker for speech output
Chinese Asr Avg	CER 3.45 (AISHELL-1 1.11, AISHELL-2 Android 2.48 / iOS 2.49, Common Voice 15zh 4.61, FLEURS zh 2.92, WenetSpeech meeting 5.54 / net 5.01)
English Asr Avg	WER 3.69 (LibriSpeech clean 1.20 / other 2.40, Multilingual LibriSpeech 3.80, Common Voice 15 en 6.51, FLEURS en 2.86, VoxPopuli 5.37)
Dialect Asr Avg	CER 9.40 across 15 Mandarin dialects (incl. Cantonese 4.09, Sichuanese/Chongqing 4.07, Anhui 5.76, Hokkien 12.84)
Long Form Asr	AA-Earnings22-clean WER 5.78, Earnings22 WER 11.60, AISHELL-4 CER 14.76 / DER 8.58, AliMeeting DER 8.66, AMI WER 19.06
Context Asr	ContextLASR WER 1.52 / NE-WER 5.21 / NE-FNR 1.06
Audio Intelligence	BigBenchAudio 99.40, MMAU 82.00, MMAR 74.40, MMSU 78.82, MMAU-Pro 66.41, AudioMarathon 68.41, LongAudioSpan rubric 41.12, MuchoMusic 65.65
Spoken Interaction	URO-Bench-Pro 81.92, VoiceBench 86.74
Tool Calling	BFCL-V3 Audio 81.40, SpeechTCEval 82.60
Voices	built-in IP voices 灵小柔 (default), 灵楷, US_James, US_Stella; reference-audio cloning with prompt text
Serving	vLLM-Omni (OpenAI-compatible, 128K profile on 8x 80 GB H800, TP8), SGLang (text output, 4 GPUs), SGLang-Omni (2 GPUs; /v1/audio/transcriptions, /v1/audio/speech, /v1/chat/completions), Transformers with custom code (trust_remote_code)
Sibling Model	Ming-3.0-audio-tiny (9.3B total / 2.7B active, single-GPU profile)

Features: The design bet is that one audio-language model should absorb the whole assistant stack — recognition, understanding, reasoning and spoken output — instead of chaining an ASR model into an LLM into a TTS model, each hop adding latency and losing acoustic context. Ming-3.0-audio-flash implements that as a MoE thinker (6.5B active out of 126B) behind a 32-layer audio encoder, with a Talker that can speak the answer in a built-in IP voice or a voice cloned from a reference clip. The evidence that the bet is paying off is in the numbers rather than the slogan: 3.45% Chinese CER and 3.69% English WER including fifteen Mandarin dialects and long-form multi-speaker meeting audio, 9.40% dialect-average CER, and best-in-class BigBenchAudio (99.40) and MMSU (78.82) against Qwen3-Omni, MiMo-V2.6 and Gemini 3.8 Flash. It also keeps the agent shape: tool calling (BFCL-V3 Audio 81.40) and spoken interaction (VoiceBench 86.74) come from the same weights, with an OpenAI-compatible server that serves transcription, chat and speech from one port.

Links:

· · · · · · · · · · · · · ·

Ming-3.0-audio-tiny

Description: Ming-3.0-audio-tiny is the compact member of inclusionAI's Ming-3.0 audio family: a 9.3B-parameter Mixture-of-Experts model with 2.7B activated per inference step (8,871,306,016 measured across 2 BF16 shards), roughly a fourteenth of Flash's footprint. It keeps the same architecture and the same task surface — audio plus a text instruction in, text or text-plus-speech out through an optional Talker — but the whole checkpoint fits on a single GPU, which is the whole point: the SGLang-Omni profile colocates the thinker, the audio encoder and speech synthesis on one device (--talker.gpu 0), and the validated SGLang text profile uses one GPU. Recognition quality tracks Flash closely but sits below it: Chinese CER 3.79% averaged over AISHELL-1/2, Common Voice 15zh, FLEURS and WenetSpeech, English WER 4.36% over LibriSpeech, Common Voice 15 en, FLEURS and VoxPopuli, and 10.29% CER across fifteen Mandarin dialects. On audio understanding it trails Flash across the board but stays usable — BigBenchAudio 99.20, MMAU 75.90, MMSU 73.06, MMAR 67.70, MMAU-Pro 56.42, VoiceBench 74.29, URO-Bench-Pro 54.15, BFCL-V3 Audio 79.80. It is released under MIT and shares the flash repository, benchmark tables and serving scripts.

Release Date: October 19, 2026

Feature	Value
Parameters	9.3B total / 2.7B active per step (8,871,306,016 safetensors-measured, 2 BF16 shards)
License
Text	✅
Video	❌
Audio	✅
Voice Cloning	✅
Asr	✅
Languages	Chinese and English (varied accents, dialects, noisy and multi-speaker conditions)
Streaming	✅
Architecture	audio encoder (32 layers, hidden 1280, 20 heads, 128 mel bins, 3000 audio-context tokens) + projector + BailingMoeV3Audio MoE text backbone + optional Talker for speech output
Chinese Asr Avg	CER 3.79 (AISHELL-1 1.47, AISHELL-2 Android 2.68 / iOS 2.60, Common Voice 15zh 5.64, FLEURS zh 2.99, WenetSpeech meeting 6.04 / net 5.14)
English Asr Avg	WER 4.36 (LibriSpeech clean 1.42 / other 3.06, Multilingual LibriSpeech 4.78, Common Voice 15 en 7.89, FLEURS en 3.30, VoxPopuli 5.68)
Dialect Asr Avg	CER 10.29 across 15 Mandarin dialects (incl. Cantonese 4.49, Sichuanese/Chongqing 4.46, Anhui 5.42)
Long Form Asr	AA-Earnings22-clean WER 6.30, Earnings22 WER 18.29, AISHELL-4 CER 21.94, AMI WER 24.77
Context Asr	ContextLASR WER 2.12 / NE-WER 6.27 / NE-FNR 2.02
Audio Intelligence	BigBenchAudio 99.20, MMAU 75.90, MMAR 67.70, MMSU 73.06, MMAU-Pro 56.42, AudioMarathon 64.54, LongAudioSpan rubric 35.45, MuchoMusic 58.07
Spoken Interaction	URO-Bench-Pro 54.15, VoiceBench 74.29
Tool Calling	BFCL-V3 Audio 79.80, SpeechTCEval 69.80
Voices	built-in IP voices 灵小柔 (default), 灵楷, US_James, US_Stella; reference-audio cloning with prompt text
Serving	SGLang-Omni (single GPU, `--talker.gpu 0` colocates speech synthesis), SGLang (text output, 1 GPU), vLLM-Omni (MING_MODEL_VARIANT=tiny), Transformers with custom code (trust_remote_code)
Sibling Model	Ming-3.0-audio-flash (126B total / 6.5B active) — same architecture, larger, stronger

Features: Ming-3.0-audio-tiny is what makes the Ming-3.0 audio line deployable rather than merely capable: the same BailingMoeV3Audio MoE thinker, the same 32-layer audio encoder and the same optional Talker as the 126B Flash checkpoint, distilled down to 9.3B total and 2.7B active, and small enough that the entire pipeline — audio encoder, thinker and speech synthesis — runs on one GPU behind an OpenAI-compatible server offering /v1/audio/transcriptions, /v1/audio/speech and /v1/chat/completions side by side. The quality gap to Flash is real but modest where it matters most: Chinese CER 3.79% vs 3.45%, English WER 4.36% vs 3.69%, dialect CER 10.29% vs 9.40%, while BigBenchAudio is essentially tied (99.20 vs 99.40). The gap widens on the harder audio-reasoning and spoken-interaction axes (MMAU-Pro 56.42 vs 66.41, VoiceBench 74.29 vs 86.74), which is the honest dividing line between the two tiers: Tiny for on-prem transcription and spoken chat, Flash when the audio reasoning has to be right.

Links:

· · · · · · · · · · · · · ·

MiDashengLM-Gen

Description: MiDashengLM-Gen (MiDasheng Language Model for Generation) is an end-to-end framework for unified audio-scene generation from Xiaomi. Built on a pre-trained LLM and the Dasheng audio tokenizer, it couples per-token conditional flow matching with autoregressive generation to produce coherent 16 kHz audio that simultaneously blends speech, music, sound effects and environmental acoustics from a structured text description. It supports 9 languages with emotion control and approaches dedicated TTS intelligibility on speech (Seed-TTS English WER drops from 12.15% to 2.79%) while retaining mixed-audio scene capability, and extends competitively to multilingual settings.

Release Date: August 12, 2026

Feature	Value
Text	✅
Video	❌
Image	❌
Audio	❌
Sample Rate	16 kHz
Languages	9
Emotion Control	✅
License
Parameters	1.7B (Qwen3-1.7B backbone)
Architecture	DashengTokenizer (768-dim @25Hz) + Qwen3-1.7B + flow-matching DiT (16 layers, hidden 2048)

Features: LLM-conditioned high-dimensional (768-dim @25Hz) audio latents generated without quantization artifacts; audio-text alignment pre-training maps latents into the LLM token space before generation; a learned stop head enables variable-length truncation. First end-to-end trained model for general text-to-audio-scene generation.

Links:

· · · · · · · · · · · · · ·

NVIDIA NemotronLabs VoiceChat 11B

Description: NVIDIA NemotronLabs VoiceChat is an 11B end-to-end, real-time speech full-duplex (FD) model for conversational AI that jointly performs streaming speech understanding and speech generation in one unified architecture — no ASR → LLM → TTS cascade, no API handoffs between models, and therefore no stacked end-to-end latency. It listens and speaks simultaneously, so it handles natural turn-taking (~450 ms response), barge-in (the user cuts in and the model yields instantly) and mid-turn pauses without a rigid request/response cycle. Architecturally it encodes audio with a Fast Conformer streaming encoder, feeds the audio tokens into a Nemotron Nano V2 9B LLM backbone to predict text tokens, and decodes those to audio codes with an NVIDIA TTS decoder and codec — while a separate output channel emits tool-calling scripts. That separate channel is what makes VoiceChat the first open full-duplex model to support tool calling: each tool can define an "on-hold" message spoken the moment the LLM emits the triggering text, so the conversation keeps flowing naturally while the tool executes. It ranks #2 among all open full-duplex models on VoiceBench and #2 among all open models on Full-Duplex-Bench 1.0.

Release Date: August 3, 2026

Feature	Value
Parameters	11B (11,095,371,252 safetensors-measured)
License
Text	✅
Video	❌
Audio	✅
Max Duration	not stated
Sample Rate	22.05 kHz output / 16 kHz input
Architecture	hybrid Mamba/Transformer — Fast Conformer streaming speech encoder + Nemotron Nano V2 9B LLM backbone + NVIDIA TTS decoder and codec + a separate tool-calling output channel
Encoder	Fast Conformer from nvidia/nemotron-speech-streaming-en-0.6b
Base Model	nvidia/NVIDIA-Nemotron-Nano-9B-v2
Duplex	full duplex (simultaneous listen and speak)
Turn Taking Latency	~450 ms (448 ms measured, smooth turn-taking)
Barge In	yes (user interruption TOR 1.0, 480 ms latency)
Tool Calling	yes — first open full-duplex model with tool calling; per-tool "on-hold" speech covers tool execution; `<TOOLCALL>[{...}]</TOOLCALL>` control format
Voicebench	#2 among all open full-duplex models
Full Duplex Bench 1	#2 among all open models
Pause Handling Tor	0.153 (Synthetic) / 0.255 (Candor), lower is better
Smooth Turn Taking	TOR 0.82 / latency 448 ms
User Interruption	TOR 1.0 / latency 480 ms / GPT-4o judge 4.33
Au Harness Bfcl V3	simple 58.5% / multiple 62.5% / parallel 42.5% / parallel-multiple 27.5% / irrelevance 89.6% / average 56.1%
Full Duplex Bench V3	tool selection 82.5% / argument accuracy 42.2% / pass@1 33%
Training Data	~550k hours of audio plus text, real and synthetic
Training Sources	Nemotron 5.5 pre-training and SFT text, Brainy-mantis, Greteal AI v1/v2, UltraChat, Blackwell studio recordings, Fisher, LibriVox, LibriTTS, HiFi-TTS, Riva Speakers (internal), internet-scale public data, PromptTTS, VCTK, Voxmovies, JL-Corpus, Nemotron Nano v3 function-calling data, PersonaPlex training data
Runtime	vLLM
Hardware	NVIDIA A100, H100, H200, B100, B200, RTX-6000
Os	Linux only
Deployment	offline batch speech-to-speech from the HF checkpoint; interactive low-latency streaming via the NVIDIA inference container (bidirectional WebSocket with function calling)
Dependencies	NeMo Speech repo (`nemotron-labs-voicechat` branch), torch 2.10.0, transformers 4.56.0, mamba-ssm 2.3.2.post1, causal-conv1d 1.6.2.post1
Version	v1.0
Intended Use	ASR, TTS and voice-assistant development for researchers and developers

Features: The cascade being replaced here is the whole point: ASR → LLM → TTS stacks serialize three models' latencies and hand audio off as text, which makes natural back-and-forth structurally impossible. VoiceChat collapses that into a single model that listens and speaks at once, and the measured consequences are the interesting part — 448 ms smooth turn-taking, and on user interruption a TOR of 1.0 with a 480 ms yield, meaning the model does not finish its sentence over the top of the user. The genuinely novel piece is tool calling in a duplex setting. A tool call is a latency spike in a conversation: the user hears silence while an API resolves. VoiceChat's fix is architectural rather than a prompt trick — a dedicated output channel predicts the tool-call script separately from the spoken channel, and each tool can declare an "on-hold" message that is spoken as soon as the triggering text is generated, so the agent talks over its own tool latency instead of stalling. That is what makes it the first open full-duplex model to do this. The benchmark table also deserves an honest read rather than a headline: tool selection is strong (82.5% on Full-Duplex-Bench v3, 89.6% irrelevance rejection on AU-Harness), but argument accuracy is 42.2% and pass@1 is 33% — spoken tool arguments are a genuinely hard problem (names, numbers and IDs must survive the audio round-trip), so the capability is real and the remaining gap is visible. The hybrid Mamba/Transformer backbone and reuse of NVIDIA's own streaming Conformer encoder and Nemotron Nano V2 LLM show the design is assembled from components the lab had already validated, which is why the release reads as an integration milestone for open voice agents rather than a single-model result.

Links:

· · · · · · · · · · · · · ·

ScenA

Description: ScenA generates multi-speaker audio scenes — dialogue and conversation with sound effects and ambience — from a text prompt, conditioned on one or more reference-audio clips that set the speakers' voices. Unlike prior multi-speaker dialogue systems it uses no per-turn tags, multi-stream transcripts, or speaker embeddings: a free-form natural-language prompt alone describes the scene. The text prompt determines which reference voice speaks where, allowing overlapping speech, spontaneous paralinguistic events, and scene-level ambient sound — all inherited from the in-the-wild text-to-audio pretraining distribution. The architecture is an audio-only, reference-conditioned flow-matching DiT built on the LTX-2 backbone (~4B parameters, 48 layers). Reference latents are concatenated into the token sequence and distinguished by lightweight identity-aware positional encodings. The training tackles a specifically identified "Reference Shortcut" failure mode — under standard noise schedules the model can identify the matching reference by noisy-target acoustic similarity, bypassing the text prompt — by using a high-noise-biased timestep distribution that forces reliance on the prompt for speaker assignment. Evaluator: CoVoMix2-Dialogue benchmark. Project page, code, paper, and HuggingFace checkpoint are linked below.

Release Date: July 7, 2026

Feature	Value
Parameters	~4B (DiT, 48 layers; built on LTX-2 architecture)
Text	✅
Video	❌
Audio	✅
Max Duration	not stated (scene-level generation)
Sample Rate	(not stated; inherits LTX-2 audio VAE)
Voice Cloning	✅
Multi Speaker	yes
Ambient Sound	yes (SFX, room acoustics, overlapping speech)
Architecture	flow-matching DiT (LTX-2 backbone, audio-only)
Speaker Assignment	natural language (no per-turn tags / identity encoders)
Training Fix	high-noise-biased timestep distribution (defeats Reference Shortcut)
Text Encoder	google/gemma-3-12b-it
Audio Vae	bundled (~365 MB; encodes+decodes so full LTX-2 not needed)
Checkpoint Size	~8.2 GB (scena.safetensors) + ~365 MB (audio_vae.safetensors)
License
Training Data	in-the-wild text-to-audio pretrained, then reference-conditioned fine-tune
Evaluation	CoVoMix2-Dialogue (speaker-binding metrics)

Features: The "Reference Shortcut" failure-mode identification is the technical center of the work: under standard diffusion noise schedules, a multi-speaker reference-conditioned model can match each reference to the noisy-target segment by acoustic similarity alone, bypassing the text prompt entirely. ScenA's high-noise-biased timestep distribution forces the model to rely on the prompt for speaker assignment at training time. Combined with the absence of any per-turn speaker structure (tags / transcripts / identity encoders) and the prompt's role as the only speaker-routing signal, this yields multi-speaker conversational scenes with overlapping speech, paralinguistic events, and ambient texture that previous structured-supervision multi-speaker systems filter out by design.

Links:

· · · · · · · · · · · · · ·

Nemotron-Labs-Audex-2B

Description: Nemotron-Labs-Audex-2B is NVIDIA's smaller sibling of the Audex unified audio-text LLM. Like the 30B-A3B flagship, the 2B is a single model family that both understands audio (audio QA, speech recognition, speech translation) and generates audio (text-to-speech, text-to-audio, speech-to-speech). It is built on the same audio-vocabulary-extended transformer stack as the 30B-A3B but at a densely-parameterized 2B scale (no MoE), so the compute and memory footprint are lowered to a budget tractable on more modest hardware. The 2B checkpoint is the project-tagged SFT variant in the Audex collection — instruction-tuned and ready for inference. Both sizes preserve text reasoning, alignment, knowledge, long-context, and agentic capabilities of the text backbone while adding discrete-token audio I/O.

Release Date: July 6, 2026

Feature	Value
Parameters	2B (dense; SFT fine-tune, instruct + reasoning-ready)
Text	✅
Video	❌
Audio	✅
Modalities	text + audio (input and output)
Max Duration	not stated
Sample Rate	not stated (decoder output)
Voice Cloning	❌
Audio Understanding	yes (audio QA, classification)
Asr	✅
Speech Translation	yes
Text To Speech	yes
Text To Audio	yes
Speech To Speech Generation	yes
Reasoning Mode	yes (thinking + instruct modes inherited from text backbone)
License
Pipeline Tag	text-generation
Library Name	transformers
Derived From	same family as Nemotron-Labs-Audex-30B-A3B
Companion 30B	nvidia/Nemotron-Labs-Audex-30B-A3B (MoE: 30B total, 3B active)
Spaces	nvidia/Nemotron-Labs-Audex, WaveCut/Nemotron-Labs-Audex, hugging-apps/nemotron-labs-audex-2b
Createdat	2026-07-06T16:21:07Z
Downloads	~2.4k

Features: The 2B sibling matters because it preserves the central thesis of the Audex paper — unified audio-text LLM intelligence without regressing on text intelligence — while dropping the parameter budget substantially. The 30B-A3B MoE hits a 1M-context, agentic flagship tier; the 2B dense version is the same audio-aware architecture extended down to a budget that doesn't require a high-end MoE serving stack. The pair lets users choose on deployment cost rather than on capability sub-selection: the 2B ships the same audio-to-audio + text-to-audio + audio-understanding

ASR + speech-translation coverage as the MoE flagship, just at the cost of longer-context / reasoning depth that the MoE was specifically tuned for. Both share the discrete-audio-token vocabulary extension of the text backbone so they can be reasoned about interchangeably.

Links:

· · · · · · · · · · · · · ·

Nemotron-Labs-Audex-30B-A3B

Description: Nemotron-Labs-Audex-30B-A3B is NVIDIA's unified audio-text LLM — a single model that both understands audio (audio QA, speech recognition, speech translation) and generates audio (text-to-speech, text-to-audio, speech-to-speech). Built on Nemotron-Cascade-2-30B-A3B (text-only MoE: 30B parameters, 3B active), Audex extends the vocabulary with discrete audio tokens for speech / general-audio output and adds an audio encoder for speech / general-audio input. Runs in thinking and instruct (non-thinking) modes and supports up to a 1M-token context length — preserving text-reasoning, alignment, knowledge, long-context, and agentic capabilities of the backbone while gaining audio tasks.

Release Date: July 6, 2026

Feature	Value
Parameters	30B MoE (3B active)
Modalities	audio (input and output)
Audio Understanding	yes
Asr	✅
Speech Translation	yes
Text To Speech	yes
Text To Audio	yes
Speech To Speech Generation	yes
Voice Cloning	❌
License
Languages	English
Modes	thinking, instruct (non-thinking)
Context Length	1M tokens
Template	ChatML (with `<think>…</think>` for thinking mode)
Inference	vLLM 0.20.0 (recommended) or transformers >= 4.53.0 (mamba-ssm + causal-conv1d required)

Features: First-class audio I/O for a 30B/3B-active text LLM: extended vocabulary with discrete audio tokens for outputting speech and general audio, plus an audio encoder for input — so the same backbone keeps its strong text reasoning (alignment, knowledge, long-context) and adds ASR + speech translation + TTS + audio generation + S2S without retraining. The MoE form (30B routes, 3B active) keeps inference tractable for a single pipeline that does both.

Links:

· · · · · · · · · · · · · ·

MOSS-SoundEffect

Description: MOSS-SoundEffect is the dedicated text-to-sound model in the OpenMOSS / MOSI.AI MOSS-TTS family. It turns natural-language captions into high-fidelity non-speech audio (ambience, urban scenes, creatures, human actions, and short music-like clips).

Release Date: May 25, 2026

Feature	Value
Type	Text-to-Sound / SFX generation
Conditioning	Text
Max Duration	30 seconds
Sample Rate	48 kHz
License
Architecture	DiT + Flow Matching + DAC VAE + Qwen3 text encoder
Parameters	1.3B (DiT variant 1.3B)
Languages	English, Chinese
Inference Defaults	100 flow-match steps, cfg 4.0, sigma_shift 5.0
Library	diffusers

Features: Replaces the discrete-token autoregressive v1 (which bottlenecked on vocabulary) with a continuous-latent DiT + Flow Matching paired with a DAC VAE — yielding 30 s stable audio, bilingual English + Chinese prompts, and a clean CFG/sigma-shift inference schedule (cfg 4.0, shift 5.0) that works straight out of the box on the diffusers library.

Links:

· · · · · · · · · · · · · ·

Omni2Sound (Omni2Audio)

Description: Omni2Sound — also written Omni2Audio on the project page — is a unified VT2A / V2A / T2A framework and a CVPR 2026 Highlight. A single Diffusion Transformer (DiT) backbone with a decoupled two-branch conditioning design:

Release Date: April 20, 2026

Feature	Value
Conditioning	Text / Video / Text+Video
Modalities	Video, Audio
Asr	❌
Voice Cloning	❌
Text	✅
Video	✅
Image	❌
Audio	✅
License
Tasks	VT2A, V2A, T2A (single model)
Architecture	DiT + decoupled Semantic / Temporal branches + 3-stage progressive training
Pipeline Tag	text-to-audio

Features: One single model that is SOTA on three distinct tasks (VT2A, V2A, T2A) without a separate model per mode — decoupled semantic and temporal conditioning let the same DiT backbone handle text-only, video-only, and text+video conditioning by cleanly omitting the missing modality rather than padding it, which is what most prior unified VA models had to do.

Links:

· · · · · · · · · · · · · ·

ControlFoley

Description: ControlFoley (Xiaomi MiLM Plus) is a unified controllable video-to-audio (foley) generation model. It supports four conditioning combinations under one architecture:

Release Date: April 13, 2026

Feature	Value
Conditioning	Text / Video / Text + Video / Video + Reference Audio
Modalities	Video (visual), Audio (foley)
Asr	❌
Voice Cloning	❌
Text	✅
Video	✅
Image	❌
Audio	✅
Sample Rate	44,100 Hz
License
Pipeline Tag	text-to-audio
Library	diffusers
Cross Modal Conflict	handled via modality-specific control (no explicit router)
Inference Skill	ClawHub ControlFoley Audio Generator
Upcoming	ComfyUI nodes (in preparation, expanding to V2A / TV2A / TC-V2A / AC-V2A / T2A)

Features: Modality-specific cross-modal conflict resolution in a single generative stack: text governs semantics, reference audio governs timbre/acoustic style, and video governs temporal synchronization. Rather than routing to a single user-trusted modality, the model decouples control axes so an input disagreement (video shows a dog barking, text asks for a cat) is decomposed into a coherent output that respects each modality's responsibility. Trained with all-modality dropout for modality-robustness, ControlFoley is the first foley system that brings all four conditioning modes — T2A, V2A, TV2A, AC-V2A — under one model.

Links:

· · · · · · · · · · · · · ·

Woosh

Description: Sony AI's sound effect foundation model for text-to-audio and video-to-audio generation. Includes Woosh-AE (audio encoder/decoder), Woosh-Flow/DFlow (T2A), and Woosh-VFlow/DVFlow (V2A) with distilled fast inference variants.

Release Date: 2026

Feature	Value
Architecture	Flow-based generative models
Text	✅
Video	✅
License
Audio-Encoding	yes
Fast-Inference	yes (Distilled models)

Features: Optimized for sound effects (not general audio) with both public and private model versions. Video-conditioned generation without requiring captions. Competitive with Stable Audio Open and TangoFlux.

Links:

· · · · · · · · · · · · · ·

Chroma-4B

Description: Chroma 1.0 (FlashLabs' Chroma-4B on HuggingFace) is the first open-source, real-time, end-to-end spoken dialogue model that achieves both sub-second end-to-end latency and high-fidelity personalized voice cloning. The pipeline is end-to-end — no separate ASR → LLM → TTS stitch — speech goes in, speech comes out. The architectural centerpiece is an interleaved text-audio token schedule (1 text : 2 audio) that supports streaming generation, so the model can begin emitting audio while the user is still talking (broken-off turns / barge-in handled). Experimental results from the project's paper:

Release Date: November 28, 2025

Feature	Value
Parameters	4B
Voice Cloning	✅
Asr	✅
Languages	English (per benchmark reporting)
Streaming	✅
License
Architecture	end-to-end spoken-dialogue LLM; interleaved text-audio token schedule (1:2); custom_code modules
Pipeline Tag	any-to-any (HF classification)
Audio Tokenization	chroma tokenizer (RVQ-style per project's tag)
Latency Rtf	0.43 (speech out ~2.3× wall-clock)
Speaker Similarity	+10.96% relative improvement over human baseline
Inference Library	transformers (custom_code)
Correlations With Larger Class	matches full dialogue turn at streaming latency
Pretrained	yes (safetensors weights)
Hf Space Demos	hysts/Chroma-4B, Pnevka/Chroma-4B
History	paper arXiv 2601.11141 (2026-01)

Features: Two bets together produce the dual property that no prior open-source spoken-dialogue model has hit simultaneously. First, an interleaved text-audio token schedule (1:2) — text tokens and audio tokens are interleaved at a fixed 1:2 ratio through the sequence, which gives the model a structured place to emit audio while still consuming user audio + text context, supporting sub-second end-to-end latency without a separate ASR / LLM / TTS pipeline. Second, personalized voice cloning baked into the spoke-dialogue model — the cloned voice is not bolted on top by a separate TTS stage (as is the default pattern), it's in-model at the audio-token-generation layer. The empirical payoff is a 10.96% relative speaker-similarity gain over the human baseline (i.e. the cloned voice is closer to the reference speaker than two of the same human speaker's recordings are to each other), while hitting RTF 0.43 — a floor that prior systems exceeded either in latency (no streaming) or in cloning fidelity (parrot the speaker poorly), rarely both.

Links:

· · · · · · · · · · · · · ·

Uni-MoE (Audio)

Description: MoE-based omnimodal model with voice cloning, TTS, T2M (text-to-music), and V2M (video-to-music).

Release Date: October 16, 2025 (Uni-MoE-Audio)

Feature	Value
Parameters	-
Voice Cloning	✅
Text	✅
Video	✅
License
Dynamic-Routing	yes

Links:

· · · · · · · · · · · · · ·

AudioX / Audio-Omni

Description: Audio-Omni is the first end-to-end framework unifying understanding, generation, and editing across general sound, music, and speech domains. Presented at SIGGRAPH 2026. AudioX is a unified framework integrating text, video, image, and audio conditions.

Release Date: March 2025 (AudioX), 2026 (Audio-Omni)

Feature	Value
Parameters	-
Text	✅
Video	✅
Audio	✅
License

Features: First unified framework covering all three audio domains. Combines frozen multimodal LLM (Qwen2.5-Omni) with trainable Diffusion Transformer for high-fidelity synthesis. Any-to-any audio processing.

Links:

· · · · · · · · · · · · · ·

HunyuanVideo-Foley

Description: Tencent's end-to-end video sound effect generation model for professional-grade AI Foley sound generation. Analyzes footage and creates immersive audio that matches the visual content perfectly.

Release Date: 2025

Feature	Value
Parameters	-
Sample Rate	48 kHz
Text	✅
Video	✅
License
High-Quality-Foley	yes
Context-Aware	yes

Links:

· · · · · · · · · · · · · ·

PrismAudio

Description: Video-to-Audio generation framework with Reinforcement Learning and specialized Chain-of-Thought (CoT) planning. Decomposes reasoning into four specialized modules (Semantic, Temporal, Aesthetic, Spatial CoT) for comprehensive video understanding. Built upon ThinkSound.

Release Date: 2025 (ICLR 2026)

Feature	Value
Parameters	518M
Video	✅
License
Cot-Planning	yes (4 modules)
Multi-Dimensional-Rl	yes
Fast-Grpo	yes (Hybrid ODE-SDE)
Inference-Time	0.63 seconds

Features: Performance Benchmarks:

Metric	VGGSound	AudioCanvas
Semantic (CLAP)	0.47	0.52
Temporal (DeSync↓)	0.41	0.36
Aesthetic (MOS-Q)	4.21±0.35	4.12±0.28

Links:

· · · · · · · · · · · · · ·

ThinkSound

Description: Unified Any2Audio generation framework with flow matching guided by Chain-of-Thought (CoT) reasoning. Supports generating or editing audio from video, text, audio, or their combinations. Accepted to NeurIPS 2025.

Release Date: 2025

Feature	Value
Parameters	-
Text	✅
Audio	✅
License
Cot-Driven-Reasoning	yes
Interactive-Object-Centric-Editing	yes

Links:

· · · · · · · · · · · · · ·

MMAudio

Description: Multimodal joint training framework for high-quality synchronized audio generation from video and/or text inputs. State-of-the-art open source model for generating sounds for videos, images, and text prompts.

Release Date: December 2024 (CVPR 2025)

Feature	Value
Parameters	-
Text	✅
Video	✅
Image	✅
License
Synchronized-Audio	yes
Multimodal-Joint-Training	yes

Links:

· · · · · · · · · · · · · ·

Audio Restoration & Enhancement

Audio Restoration & Enhancement Quick Comparison

Model	Type	Bandwidth Extension	Inpainting
RE-USE	Universal Speech Enhancement	✅	❌
NovaSR	Audio Super-Resolution	✅	❌
QuarkAudio-UniSE	Universal Speech Enhancement	❌	❌
PASE	Speech Enhancement	❌	❌
DTT-BSR	Music Source Restoration	❌	❌
NVIDIA A2SB (Audio-to-Audio Schrodinger Bridges)	High-Resolution Audio Restoration	✅	✅
ZipEnhancer	Acoustic Noise Suppression	❌	❌
AudioSR	Audio Super-Resolution	✅	❌

RE-USE

Description: RE-USE (RE-…), NVIDIA's multilingual universal speech enhancement model, targets distortion–perception trade-off by training a single model that balances listening quality against fidelity to the underlying linguistic / speaker / emotional content. Designed to restore diverse degraded speech while leaving everything else (content, identity, prosody, accent, paralinguistic attributes) intact.

Release Date: March 17, 2026

Feature	Value
Type	Universal Speech Enhancement
Bandwidth Extension	✅
Inpainting	❌
Sample Rate	8 / 16 / 22.05 / 24 / 32 / 44.1 / 48 kHz (multi-rate input)
Architecture	Mamba-SSM backbone
Degradation Coverage	additive noise, reverberation, clipping, bandwidth limit, codec artifacts, packet loss, low-quality mics
Language Agnostic	yes
License

Features: A single Mamba-SSM model that handles seven different input sample rates (no resampling pre-step), covers a broad degradation menu in one checkpoint, stays language-agnostic without per-language training, and explicitly balances distortion reduction against fidelity to the input speech — addressing the universal-SE trade-off that earlier single-purpose enhancers couldn't.

Links:

· · · · · · · · · · · · · ·

NovaSR

Description: NovaSR is a tiny audio upsampler (~52 kB parameter count) that bandwidth-extends 16 kHz input up to 48 kHz. Public card on YatharthS/NovaSR advertises realtime factors around 3500× on A100, making it a candidate for real-time on-device super-resolution where model size dominates latency. Inference path is small enough to fit in CPU memory; the use case is speech-bandwidth extension without GPU.

Release Date: January 6, 2026

Feature	Value
Type	Audio Super-Resolution (16 kHz → 48 kHz)
Bandwidth Extension	✅
Inpainting	❌
Channels	mono
License
Parameters	52 kB
Streamable	yes (low VRAM / runs without GPU)
Realtime Factor	~3500× (A100)

Features: A 52 kB-parameter Upsampler that hits ~3500× realtime on GPU and runs on CPU — pushing bandwidth extension below the size / latency envelope where a typical neural upsampler is unacceptable (real-time on-device speech enhancement).

Links:

· · · · · · · · · · · · · ·

QuarkAudio-UniSE

Description: UniSE is a unified, prompt-free autoregressive speech-enhancement framework built on a decoder-only language model. A single model performs multiple speech-enhancement tasks — speech restoration (SR / denoising), target-speaker extraction (TSE), source separation (SS), and acoustic echo cancellation (AEC, in development) — without explicit task-specific instructions or prompt conditioning; the language model infers the task from the input context. Stack: WavLM as the feature extractor, BiCodec as the discrete codec, and a decoder-only LM as the middle autoregressive backbone. Outputs reconstructed waveform from predicted discrete token sequences.

Release Date: December 22, 2025

Feature	Value
Voice Cloning	❌
Asr	❌
Streaming	❌
Languages	English (paper demo)
License
Tasks	Speech Restoration, Target Speaker Extraction, Source Separation, AEC (developing)
Architecture	WavLM (feature extractor) + BiCodec (discrete codec) + decoder-only AR-LM
Unified	yes (single model handles SE, SR, TSE, SS without explicit task prompts)
Prompt Free	yes (LM infers task from input context)
Dataset Signals	noise + reverb + packet-loss + clean (configurable per task)
Training	Speech-enhancement SFT, then multitask joint training

Features: A single decoder-only LM that learns the speech-enhancement task distribution and infers which task to perform from the input context — eliminating the need for task-specific prompts, modules, or fine-tuning when switching between denoising, target-speaker extraction, and separation. Built as an autoregressive discrete-token predictor over a WavLM-extracted / BiCodec-quantised representation, it moves the speech-enhancement workflow from a zoo of specialist models into one generalist.

Links:

· · · · · · · · · · · · · ·

PASE

Description: PASE (Phonologically Anchored Speech Enhancer) is a generative speech-enhancement model from Cisco Collaboration AI that removes noise and reverberation while preserving linguistic content and speaker identity. It uses two fine-tuned WavLM-derived components:

Release Date: November 8, 2025

Feature	Value
Type	Speech Enhancement
Bandwidth Extension	❌
Inpainting	❌
Sample Rate	16 kHz mono
Architecture	Denoising WavLM (DRD from WavLM-Large) + Dual-Stream Vocoder (phonetic + acoustic)
Finetuned From	WavLM-Large
Training Data	DN5/DNS5 challenge clean + noise, LibriTTS, VCTK, OpenSLR26+28 RIRs
License

Features: Anchors enhancement to phonology instead of spectrum: by reconstructing from a phonetic stream and a separate acoustic stream (per DeWavLM's two representations), PASE keeps the words intact even when the spectrum is severely degraded — substantially lowering hallucinations while still regaining perceptual quality.

Links:

· · · · · · · · · · · · · ·

DTT-BSR

Description: DTT-BSR (DTTNet with BandSequence and RoPE) is a music-source-restoration challenge submission from team AC/DC (Wuhan University) to ICASSP 2026. It is built inside the official MSR-Kit GAN framework, where the baseline generator is replaced by a DTTNet-style time-frequency U-Net and augmented at the bottleneck with:

Release Date: October 16, 2025

Feature	Value
Type	Music Source Restoration
Bandwidth Extension	❌
Inpainting	❌
Architecture	DTTNet TFC-TDF U-Net (complex STFT) + Improved Dual-Path BandSplitRNN block + RoPE-Transformer
Input	complex STFT (real + imag channels; n_fft=2048, hop=512)
Discriminator	Multi-Frequency Discriminator (baseline)
Framework	MSR-Kit GAN (reconstruction + adversarial + feature-matching losses)
License

Features: Treats music-source restoration as a complex-STFT time-frequency U-Net enhancement at the bottleneck: keep the strong DTTNet dual-path TFC-TDF structure for local spectral patterns, then layer in BandSplitRNN-style sub-band recurrence + RoPE self-attention so the generator can model long-range, cross-band harmonic structure that ordinary GAN baselines miss — critical for restoring non-vocal stems cleanly.

Links:

· · · · · · · · · · · · · ·

NVIDIA A2SB (Audio-to-Audio Schrodinger Bridges)

Description: A2SB is NVIDIA's audio-to-audio Schrödinger Bridge diffusion model for high-resolution (44.1 kHz) music restoration. It is the first long-audio restoration model that can restore hour-long inputs without boundary artifacts, and it's end-to-end — predicting waveform outputs directly without a separate vocoder. A single trained checkpoint handles both bandwidth extension (predicting high-frequency components) and inpainting (re-generating missing segments), training on permissively-licensed subsets of FMA, Medley-Solos-DB, MUSAN, Musical Instrument, MusicNet, Slakh, FreeSound, FSD50K, GTZAN, and NSynth.

Release Date: August 1, 2025

Feature	Value
Type	High-Resolution Audio Restoration
Bandwidth Extension	✅
Inpainting	✅
License
Channels	stereo
Sample Rate	44.1 kHz
Architecture	End-to-end vocoder-free diffusion Schrödinger Bridge (factorized audio representation)
Long Audio	yes (hour-scale restoration, no boundary artifacts)
Multi Task	yes (single model, joint bandwidth-extension + inpainting)
Training Data	FMA, Medley-Solos-DB, MUSAN, Musical Instrument, MusicNet, Slakh, FreeSound, FSD50K, GTZAN, NSynth (permissive subsets)

Features: A diffusion Schrödinger Bridge formulation that is end-to-end vocoder-free: instead of generating a mel / MFCC / latent and then re-synthesising, the model predicts the waveform directly, which is what lets it stay boundary-free over hour-long inputs. The same checkpoint carries both bandwidth extension and inpainting — two distinct restoration tasks trained jointly on permissive-licensed music data and rolled out under NVIDIA NC.

Links:

· · · · · · · · · · · · · ·

ZipEnhancer

Description: ZipEnhancer (speech_zipenhancer_ans_multiloss_16k_base) is Alibaba DAMO Academy's next-generation single-channel speech intelligent denoising model. It is a 16 kHz-in → 16 kHz-out acoustic-noise-suppression (ANS) system that:

Release Date: October 12, 2024

Feature	Value
Type	Acoustic Noise Suppression / Speech Enhancement (single-channel denoising)
Bandwidth Extension	❌
Inpainting	❌
License
Model Type	dual-path (per ModelScope metadata)
Training Objective	multi-loss (per model id `..._multiloss_...`)
Input Sample Rate	16,000 Hz
Output Sample Rate	16,000 Hz
Channels	single-channel (mono)
Task	acoustic-noise-suppression (ANS)
Use Cases	denoise any audio source; improve acoustic quality; vocal extraction from background
Host Org	Alibaba DAMO Academy (iic on ModelScope)
Ecosystem	ModelScope; Hugging Face Spaces not hosted on HF, demo available through ModelScope's widget
Release Date Evidence	ModelScope CreatedTime = unix 1728713079 = 2024-10-12 06:04 UTC

Features: Alibaba's ZipEnhancer is among the few open single-channel speech denoising models that ships both an explicit dual-path architecture and a published multi-loss training objective at the 16 kHz single-channel baseline. The dual-path split — a common modern architectural pattern in speech enhancement — separates the model's processing path for short-time-frame local acoustic features from its longer-context path, which lets the system keep a sharp response to sudden transient noise while still modeling the slow envelope of stationary backgrounds. Combined with the multi-loss objective (the model id suffix _multiloss indicates multiple supervision signals during training, e.g. spectral / waveform / perceptual losses), the result is one of the few openly hosted ANS checkpoints that handles arbitrary-source audio quality improvement, not just narrow-band microphone cleanup — the documented use case extends to vocal extraction from full mixes, which most ANS-only pipelines can't do without a separate source-separation model.

Links:

· · · · · · · · · · · · · ·

AudioSR

Description: AudioSR is a versatile audio super-resolution model from Haoheliu Liu and collaborators, published alongside the arXiv paper 2309.07314. It is a latent-diffusion model that takes arbitrary low-resolution input audio and reconstructs a 48 kHz waveform — bandwidth extension in one model. Designed to be input-rate-agnostic (8 kHz, 16 kHz, 24 kHz, 32 kHz, 44.1 kHz, 48 kHz), produces a single fixed 48 kHz output regardless of input rate, and supports both mono and stereo material. The HF model id haoheliu/audiosr_basic ships the Apache-2.0-licensed basic checkpoint the README links out to; the underlying implementation lives in the GitHub repo haoheliu/versatile_audio_super_resolution.

Release Date: September 6, 2023

Feature	Value
Type	Audio Super-Resolution (any → 48 kHz)
Bandwidth Extension	✅
Inpainting	❌
Channels	mono, stereo
License
Vram	6 GB minimum
Long Audio	yes
Architecture	latent diffusion (audio LDM)

Features: Versatile bandwidth extension over a wide range of input rates (8–48 kHz) under one fixed 48 kHz output, with monaural-only and stereo-only inference paths working through the same latent-diffusion pipeline — letting a single light checkpoint cover multiple BWE tasks instead of separate per-rate models.

Links:

· · · · · · · · · · · · · ·

Speech Recognition (ASR)

ASR Quick Comparison

Model	Languages	Streaming
Xiaomi-CocktailASR-1	Chinese, English	❌
BuzzASR	102	❌
VibeVoice-ASR-Streaming-7B	10	✅
Ruby-ASR-1.7B	Japanese	❌
Granite-Speech-5.0-470M-TurboCTC	English	❌
kodama-ja-streaming-small	Japanese	✅
GigaAM-Multilingual	70+	❌
GigaChat3.1-Audio	Russian, English	❌
Audio8-ASR-0.1B	7	❌
MOSS-Transcribe-Diarize	Multilingual	✅
Typhoon ASR Streaming	Thai	✅
ARK-ASR-3B	19	❌
Mega-ASR	English, Chinese	❌
Higgs-Audio-v3-8B-STT-v2	English	❌
Higgs-Audio-v3-STT	English	❌
Cohere Transcribe	14	✅
VibeVoice-ASR	50+	✅
SYMPHONY-ASR	English, Korean	❌
Fun-ASR	31	✅
GLM-ASR-Nano-2512	17	❌
SYMPHONY	English, Korean	❌
Moonshine	English	❌
SenseVoice	Multilingual	✅
FunASR	50+	✅

Xiaomi-CocktailASR-1

Description: Xiaomi-CocktailASR-1 is a target-speaker ASR (TS-ASR) SpeechLLM. Given a reference clip of the speaker you want and either mixed or single-speaker audio, it transcribes only that speaker — using the reference as a voiceprint prompt, with no explicit speech separation stage. It is built on large-scale multi-speaker data and addresses the two failure modes the paper calls out in prior TS-ASR work: degraded single-speaker accuracy (a dedicated TS-ASR model that breaks on ordinary audio) and the inability to reject when the target speaker is absent. Xiaomi-CocktailASR-1 does both in one architecture — it stays comparable to mainstream ASR on single-speaker sets, and it outputs empty text when the reference speaker is not in the mixture. It also supports a Chain-of-Thought mode that emits explicit reasoning steps. On simulated multi-speaker benchmarks it reaches 4.11% WER on LibriMix 2mix and 2.90% on LibriSpeechMix 2mix, where Qwen3-ASR scores 68.75% and 92.17% respectively.

Release Date: September 10, 2026

Feature	Value
Languages	Chinese, English
Streaming	❌
License
Architecture	MicAsrModel SpeechLLM — inline D2V2 audio encoder + adapter + LLM, loaded via `trust_remote_code`
Parameters	not stated (single `pytorch_model.bin` holding D2V2 + adapter + LLM)
Input	reference speaker clip + target audio, auto-concatenated internally as reference + 1 s silence + target
Audio Format	16 kHz mono (other rates are resampled automatically)
Capabilities	target-speaker ASR, negative-sample rejection, chain-of-thought reasoning
Cot	`<think>...</think>` reasoning plus `<answer>...</answer>` output, enabled with `cot=True`
Wer Librimix	2mix 4.11 / 3mix 12.29
Wer Librispeechmix	2mix 2.90 / 3mix 4.91
Wer Real Multispeaker	AMI SDM 21.81 / AliMeeting Far 20.63
Wer Single Speaker	LibriSpeech 1.73 / AliMeeting-near 6.57 / AMI-ihm 8.89 / WenetSpeech-meeting 5.81 / CommonVoice-zh 4.95
Frr Single Speaker	LibriSpeech 0.36 / AliMeeting-near 0.38 / AMI-ihm 0.01 / WenetSpeech 0 / CommonVoice-zh 0.73
Negative Rejection Rate	LibriSpeech neg 79.59 / Aishell neg 75.35 / Chinese in-house neg 68.54
Cot Gain	LibriMix 2mix 4.11 → 3.87 WER with CoT
Baselines Beaten	Qwen3-ASR, Gemini, StepAudio, Whisper Large-v2, SQ-Whisper, prior Conformer TS-ASR
Usage	`AutoModel.from_pretrained("Ease3/Xiaomi-CocktailASR-1", trust_remote_code=True, torch_dtype="bfloat16")` then `model("target.wav", "ref_speaker.wav")`
Batch	`tools/test_batch_scp.py` over a 5-column TSV, with `--cot` for reasoning mode
Runtime	`pip install torch torchaudio transformers soundfile`
Weights Host	HuggingFace repo `Ease3/Xiaomi-CocktailASR-1` (linked from the official xiaomi-research repo)
Upstream License	Apache-2.0 per the xiaomi-research repository LICENSE

Features: The architectural bet is prompt-based speaker conditioning instead of separation: rather than running a speech-separation front-end and then transcribing the extracted stream, Xiaomi-CocktailASR-1 feeds the reference voiceprint into the LLM's prompt path and lets attention do the selection, so one model serves both cocktail-party and ordinary single-speaker audio with no model switch. That unification is the point — the paper's complaint about prior TS-ASR is precisely that specializing for mixtures costs you single-speaker accuracy, and the single-speaker numbers here (1.73% on LibriSpeech, 8.89% on AMI-ihm) sit alongside mainstream ASR rather than behind it. The second, less common capability is negative-sample rejection: because the model can emit empty output when the reference speaker is absent, it becomes usable as a verification component — 79.59% rejection on LibriSpeech negative pairs, where Qwen3-ASR and StepAudio score 0%. That is not free, and the report is candid about the cost: rejection introduces a small false-rejection rate on true positives (0.36% on LibriSpeech, 0.73% on CommonVoice-zh), which is why FRR is published next to every WER number. CoT mode adds interpretability and a small accuracy gain (4.11 → 3.87 WER on LibriMix 2mix) but not a transformation — the honest framing for a reasoning mode whose value is legibility rather than raw score.

Links:

· · · · · · · · · · · · · ·

BuzzASR

Description: BuzzASR is a swarm of 102 monolingual speech recognizers — one fine-tune of openai/whisper-large-v3 per FLEURS language, MIT-licensed and shipped as fp16 safetensors so each drops into any standard Whisper pipeline. The project scales up two language-adaptation recipes that had previously been tried on only a handful of languages: Simple (SFT), plain fine-tuning, and Full (FFT), full fine-tuning behind a native tokenizer whose embeddings are warm-started from Whisper's before training. The native tokenizer is the crux — Whisper's BPE was built for English-heavy data and shreds other languages into many small pieces, so replacing it both raises accuracy and shortens the token stream, which means FFT also decodes faster. Results: 89 of 102 languages beat Whisper-large-v3 zero-shot on FLEURS-test CER (77 in the paper with an average CER reduction of more than 2.8×), 31 of 102 hold the lowest CER of every open system compared on FLEURS-test, and 41 of 102 on the combined FLEURS + Common Voice set. Compression improves 3.3× on average in characters per token, up to 21.7×.

Release Date: September 8, 2026

Feature	Value
Languages	102 (one monolingual model per FLEURS language)
Streaming	❌
License
Parameters	1.5B per model (1,545,548,800 F16 in each safetensors checkpoint)
Base Model	openai/whisper-large-v3
Models Released	102 (the best recipe per language, chosen on validation — never on test)
Recipes	Simple (SFT, plain fine-tuning) and Full (FFT, native tokenizer + fine-tuning)
Fft Tokenizer	language-native tokenizer replacing Whisper's English-heavy BPE, with embeddings warm-started from Whisper's
Compression Gain	3.3× average improvement in characters per token, up to 21.7×
Results Vs Whisper Zs	89/102 languages better on FLEURS-test CER (paper reports 77/102 with >2.8× average CER reduction)
Sota Open Systems	31/102 lowest CER on FLEURS-test; 41/102 on FLEURS + Common Voice (paper counts 27/102 on the combined set)
Compared Against	Whisper-large-v3, Omnilingual 1B/7B, MMS-1B, Qwen3-ASR, Cohere
Decoding Speed	FFT emits fewer tokens than Whisper/SFT, so it finishes first at equal time-per-token
Usage	`WhisperForConditionalGeneration.from_pretrained("BuzzASR/<language>")` with `WhisperProcessor`
Prompt	language/task prompt baked into each model's generation config, so no `language=` argument is needed
Compute Partner	EleutherAI
Venue	Findings of EMNLP 2026
Format	fp16 safetensors, compatible with the standard Transformers Whisper path

Features: Most multilingual ASR work tries to make one model cover everything; BuzzASR bets the opposite way and mass-produces specialists. The contribution is not the idea — monolingual fine-tuning is old news — but the demonstration that it holds up at 102-language scale with a uniform, reproducible recipe, which converts a folk practice into a baseline others have to beat. The genuinely technical part is the tokenizer swap. Because decode latency is token count × time-per-token, and because Whisper's tokenizer fragments non-English text, a language-native tokenizer pays twice: fewer tokens to emit and better accuracy from keeping whole morphemes. The engineering trick that makes it cheap is warm-starting the new tokenizer's embeddings from Whisper's before fine-tuning, so the model does not have to relearn its input representation from scratch — that is what turns a from-scratch tokenizer change into an incremental fine-tune. The reporting is also notably careful: recipes are selected on validation and never on test, and the project publishes both the FLEURS-test count (31) and the combined FLEURS + Common Voice count (41) rather than quoting the flattering one alone.

Links:

· · · · · · · · · · · · · ·

VibeVoice-ASR-Streaming-7B

Description: VibeVoice-ASR-Streaming is Microsoft's unified streaming speech-to-text model that transcribes Who (Speaker) said What (Content) continuously as speech arrives, with support for customized hotwords (names, technical terms) that improve recognition of domain-specific content. It covers 10 languages: Chinese, English, French, German, Italian, Japanese, Korean, Portuguese, Russian, and Spanish. The checkpoint weighs 8.67B parameters (BF16) and loads with transformers (VibeVoiceForASRStreamingTraining). A technical report is on arXiv (2609.02812). Released under the MIT License by Microsoft Research.

Release Date: September 2, 2026

Feature	Value
Languages	10 (zh, en, fr, de, it, ja, ko, pt, ru, es)
Streaming	✅
License
Parameters	8.67B (8,674,021,857 BF16)
Architecture	VibeVoice (transformers, VibeVoiceForASRStreamingTraining)
Highlights	speaker-attributed transcription; customized hotword boosting

Features: Unifies three streaming capabilities in one pass — speaker attribution ("who"), content transcription ("what"), and user-supplied hotword boosting for domain-specific vocabulary — instead of chaining diarization + ASR + post-correction, and does it continuously as audio arrives rather than in offline batches.

Links:

· · · · · · · · · · · · · ·

Ruby-ASR-1.7B

Description: Ruby-ASR-1.7B is a pair of Japanese fine-tunes of Qwen/Qwen3-ASR-1.7B that carry their own pronunciation evidence. The standard Japanese ASR target is an orthographic transcript, but the same written span can be realized with different readings — so acoustically different utterances get identical targets, and the reading distinction is destroyed before the model ever sees it. Ruby-ASR refines the target into a span-bound orthography↔reading sequence, binding each written span to the reading actually realized in speech (七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、), which lets both the orthographic and the lexical-reading view be recovered deterministically. Two variants ship in the repo: Ruby-ASR-sub (subtitle/, subtitle-style) and Ruby-ASR-ver (verbatim/). Each adds two complementary outputs: (1) the ruby transcript with inline furigana, and (2) a mora reading produced by CTC directly from the audio encoder — no LLM decoding at all. On a character-count-weighted average over B5K, CSJ, JSUT-Book, Common Voice 8 and TEDx, Ruby-ASR-sub posts the best Raw CER at 8.51% (21.1% relative below Qwen3-ASR) and Ruby-ASR-ver the best SA-CER at 6.59% and Kana CER at 3.75% (23.3% and 34.7% relative reductions).

Release Date: August 27, 2026

Feature	Value
Parameters	1.7B (fine-tune of Qwen3-ASR-1.7B; weights shipped as a stock Qwen3-ASR checkpoint)
Languages	Japanese only (ja)
Streaming	❌
License
Base Model	Qwen/Qwen3-ASR-1.7B
Variants	Ruby-ASR-sub (`subtitle/` — concise orthographic transcription) and Ruby-ASR-ver (`verbatim/` — script-aware and lexical-reading accuracy)
Output Modes	Mode 1 mora reading via encoder + CTC (no LLM decoding); Mode 2 ruby transcription with inline furigana
Ruby Output	every annotated word carries its reading, e.g. `七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、`
Mora Output	actual pronunciation as katakana morae, e.g. `ナナネンカンデロスニモトモダチデキタシ`
Mora Alphabet	katakana morae (キョ, ッ, ン, ー are single tokens) plus a–z for embedded English; long vowels unified to ー per the pyopenjtalk `g2p(kana=True)` convention
Ctc Branch	2-layer Transformer adapter (nhead 8) + linear head on the audio encoder, trained jointly with CTC loss, stored separately under `ctc/`
Training Objective	joint loss 0.3·CTC + 0.7·CE (cross-entropy over the ruby transcript plus CTC over mora targets)
Decoupled Paths	the seq2seq path never sees the CTC adapter, so ruby decoding is bit-identical to a checkpoint without the branch
Benchmarks	character-count-weighted averages over B5K, CSJ, JSUT-Book, Common Voice 8 and TEDx
Raw Cer	Ruby-ASR-sub 8.51 (best) / Ruby-ASR-ver 9.10 / ReazonSpeech-k2 10.35 / Qwen3-ASR-1.7B 10.78
Sa Cer	Ruby-ASR-ver 6.59 (best) / Ruby-ASR-sub 6.64 / ReazonSpeech-k2 8.51 / Qwen3-ASR-1.7B 8.59
Kana Cer	Ruby-ASR-ver 3.75 (best) / Ruby-ASR-sub 4.01 / Kana-Whisper 4.66 / Qwen3-ASR-1.7B 5.74
Relative Gains	21.1% Raw CER below Qwen3-ASR (sub); 23.3% SA-CER and 34.7% Kana CER below Qwen3-ASR (ver); 19.5% Kana CER below Kana-Whisper (ver)
Metric Definitions	Raw CER (orthographic, original script); SA-CER (kana scored by reading, kanji by orthography); Kana CER (lexical-reading transcription)
Serving	unmodified vLLM (>= 0.14) or the official qwen-asr toolkit for Mode 2; the ruby_asr package for Mode 1
Postprocessing	`to_surface`, `to_reading`, `to_html`, `brackets_to_parens` helpers emit plain text, kana-only text or HTML `<ruby>` markup
Vllm Note	use /v1/chat/completions, not /v1/audio/transcriptions — the transcriptions endpoint builds a different prompt and returns brackets-free text
Limitations	Japanese-only fine-tune (other Qwen3-ASR languages untested); Mode 1 returns pronunciation kana with no kanji, punctuation or word boundaries; furigana is learned behaviour rather than constrained decoding, so rare formatting slips are possible
Paper	Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition

Features: The idea is a target-representation fix, not an architecture change, and it is the kind of intervention that usually matters more than either. In Japanese ASR, supervising on orthography throws away the acoustic evidence that disambiguates readings — two utterances with identical kanji but different pronunciations become one target, and no post-hoc text-only G2P can recover what the label discarded. Ruby-ASR keeps that evidence by making the target span-bound: reading is written next to the span that produced it rather than as a separate parallel sentence, so orthography and reading stay locally aligned and both views are deterministically recoverable. The interesting second move is what they do with the auxiliary CTC branch: it hangs off the audio encoder rather than the decoder, and its weights are shipped in a separate ctc/ subfolder so the main checkpoint stays a stock Qwen3-ASR file. That is a deliberate systems compromise — reading extraction becomes a fast encoder-side pass with no LLM decoding, while the primary path remains byte-identical to a checkpoint without the branch and still runs on unmodified vLLM. The joint 0.3·CTC + 0.7·CE weighting plus the explicit claim that the seq2seq path never sees the adapter is what makes that decoupling honest rather than lossy. Benchmark reporting is careful in a way most releases are not: all three metrics are defined, the two variants are compared on all three axes (and each wins a different one), and the comparison set includes both an orthographic ASR baseline and a reading-oriented one (Kana-Whisper) so the gain cannot be explained by metric choice.

Links:

· · · · · · · · · · · · · ·

Granite-Speech-5.0-470M-TurboCTC

Description: Granite Speech 5.0 TurboCTC is a compact 470 million parameter English ASR model whose explicit design target is the speed axis of the speed–accuracy trade-off. It is an encoder-only Conformer trained with CTC — no autoregressive decoder, no language-model head — and inference is a single non-autoregressive pass with greedy decoding. The architecture runs 16 Conformer blocks (hidden 1024, 8 heads × 128, conv kernel 7) with block-diagonal (chunk-wise) self-attention over 128-frame blocks, self-conditioned CTC (middle-layer predictions feed back as conditioning), and a 16,384 BPE output head. Aggressive pyramidal temporal subsampling takes the frame rate from 100 Hz to 12.5 Hz (2× frame stacking/skipping, then 4× strided depthwise convolutions with pooled residuals in the first two blocks). Training used only public data — ~60,000 hours of English audio plus synthetic multi-speaker and entity-rich subsets — with a Muon optimizer and balanced sampling, over 10 days on 8× H100. The result: on the speed–accuracy Pareto frontier of the Open ASR leaderboard while being twice as fast as the fastest competitor, with Apache-2.0 weights suited to laptops, smartphones and other edge devices.

Release Date: August 25, 2026

Feature	Value
Parameters	473.0M (472,993,792 safetensors-measured; card rounds to 470M)
Languages	English only (en)
Streaming	❌
License
Architecture	encoder-only Conformer CTC — 16 blocks, hidden 1024, 8 attention heads × 128, conv kernel 7
Output Head	16,384 BPE units
Input Dimension	320 = (80 log-mels + 80 deltas) × 2
Temporal Subsampling	8× (100 Hz → 12.5 Hz) via frame stacking/skipping (2×) then strided depthwise convolutions with pooled residuals in the first two Conformer blocks (4×)
Attention	block-diagonal (chunk-wise) self-attention, 128-frame blocks
Self Conditioning	CTC predictions from the middle layer feed back as conditioning
Training Data	~60,000 hours of English audio, public corpora only
Training Breakdown	MLS 44,600 h; YODAS 8,900 h; CommonVoice-17 2,500 h; VoxPopuli 500 h; LibriSpeech 960 h; AMI 150 h; Earnings-22 100 h
Synthetic Data	2,000 h multi-speaker concatenations (MLS/YODAS/CV-17/VoxPopuli/AMI); 500 h multi-speaker Earnings-22; 240 h numbers, currencies, URLs, phone numbers and addresses generated with gpt-oss-120b or gpt-oss-20b and synthesized with StyleTTS2
Optimizer	Muon (novel use for ASR training)
Sampling	balanced data sampling
Inference Speedups	1×1 convolutions replaced with linear layers; optimized Conformer attention computation
Speed Claim	on the speed–accuracy Pareto frontier of the Open ASR leaderboard, 2× faster than the fastest competitor
Evaluations	Open ASR leaderboard (short-form English, RTFx on 1× H200) and FFASR leaderboard (noisy/reverberant speech, RTFx on 1× L4) — official results as of August 25, 2026
Deployment	transformers >= 5.16.0 native (`AutoModelForCTC`); mlx-audio >= 0.5.1 on Apple Silicon; transcribe.cpp GGUF (Q8_0) on Metal, Vulkan, CUDA, ROCm or CPU
Usage	`AutoProcessor` + `AutoModelForCTC`, then `model.generate(**inputs)` with greedy decoding
Training Compute	10 days on 8× H100 (IBM Blue Vela)
Venue	technical report submitted to ICASSP 2027
Intended Use	enterprise low-latency / high-throughput English speech-to-text

Features: The contrarian move is deploying plain CTC in 2026, against a leaderboard landscape dominated by LLM-based SpeechLLMs — and winning on the axis CTC was always good at. An encoder-only model with greedy, fully parallel decoding has no autoregressive token loop to pay for, and IBM then spends the entire architecture budget making the encoder itself cheap: 8× pyramidal subsampling to a 12.5 Hz frame rate collapses the sequence the encoder must traverse (the same low-frame-rate logic the audio-codec side of this list converges on), block-diagonal attention caps quadratic cost at chunk scale, self-conditioned CTC buys back accuracy lost to the small output head, and inference-level micro-optimizations (1×1 convs → linear layers, attention restructuring) squeeze the runtime further. Two quieter choices matter for reproducibility: training on public data only (with synthetic multi-speaker and entity-rich augmentation, including StyleTTS2-synthesized numbers and addresses) makes the recipe auditable, and the first ASR use of the Muon optimizer is a concrete, copyable training contribution rather than a benchmark flex. The claim worth remembering is calibrated honestly — not "best WER" but on the Pareto frontier while 2× faster than the fastest competitor, which is the right pitch for edge deployment where RTFx dominates.

Links:

Additional Tools:

Tool	Type	Link
transcribe.cpp	GGUF inference engine (Metal, Vulkan, CUDA, ROCm, CPU)	transcribe.cpp
mlx-audio	Apple Silicon (MLX) runtime	mlx-audio

· · · · · · · · · · · · · ·

kodama-ja-streaming-small

Description: kodama-ja-streaming-small is a Japanese streaming speech-recognition model, fully fine-tuned from moonshine-ai/moonshine-streaming-small (MIT, English) on the full 35,000 hours of ReazonSpeech v2. It targets offline CPU operation and low-latency streaming, shipping a 3-graph ONNX/ORT deployment set (324 MB): a float acoustic encoder (207.9 MB; int8 quantization degrades accuracy so it is kept in float), a cross-attention KV prefill graph, and an int8 dynamic-quantized single-step decoder (83.1 MB, no CER regression). Streaming uses bounded-tail revision (local-agreement) with a 500 ms block; the adopted setting trades +0.0538 CER vs offline for ~883–957 ms finalization latency. It is ~5.3× faster to first partial than Vosk small. The card notes the checkpoint measures 140,135,225 parameters from the safetensors header (the base model's card says 123M).

Release Date: August 22, 2026

Feature	Value
Languages	Japanese only (ja)
Streaming	✅
License
Parameters	140.1M (safetensors measured; base card claims 123M)
Architecture	Moonshine Streaming encoder-decoder (10+10 layers, vocab 32,768, 16 kHz)
Base Model	moonshine-ai/moonshine-streaming-small (full fine-tune, not LoRA)
Training Data	ReazonSpeech v2 (35,000 hours, CDLA-Sharing-1.0)
Deployment	ONNX/ORT assets bundled (encoder float + int8 decoder step), CPU offline

Features: Brings native streaming Japanese ASR to CPU-class devices by porting Moonshine's bounded-tail revision scheme through a full-language fine-tune and quantizing only where it is safe (the per-step decoder, keeping the acoustic encoder in float). The card is unusually transparent about the streaming/offline trade-off (+0.0538 CER at the adopted latency setting vs +0.0230 for a slower frozen-free variant) and about the parameter-count discrepancy versus the upstream card.

Links:

· · · · · · · · · · · · · ·

GigaAM-Multilingual

Description: GigaAM Multilingual is a family of Conformer-based ASR foundation models from ai-sage (Sber). Two parameter scales ship (ssl / ctc at 220M and large_ssl / large_ctc at 600M). The encoder is pretrained with a HuBERT-style self-supervised objective on 2M hours of speech across 70+ languages, then fine-tuned for speech recognition with character-wise CTC decoders on 50K hours. The ssl/large_ssl variants are encoder-only checkpoints for downstream fine-tuning; the ctc/large_ctc variants add a character-wise CTC decoder ready for inference. The model reports best-in-class open-source quality on Russian, Kazakh, Kyrgyz, and Uzbek ASR, and moderate quality on English — beating Seamless M4T large v2 and Omnilingual 1B on Russian Common Voice (7.1%/5.1% WER vs 9.2%/13.6%). A live demo Space is hosted at hugging-apps/gigaam-multilingual-asr.

Release Date: July 14, 2026

Feature	Value
Parameters	220M (`ssl` / `ctc`); 600M (`large_ssl` / `large_ctc`)
Asr	✅
Languages	70+ pretrained, fine-tuned for ru / en / kk / ky / uz
Streaming	❌
License
Architecture	Conformer encoder + (optional) character-wise CTC decoder
Pretraining Objective	HuBERT-style self-supervised
Pretraining Hours	2,000,000 hours
Fine Tuning Hours	50,000 hours
Variants	ssl (encoder-only), ctc (encoder + CTC decoder), large_ssl (600M encoder-only), large_ctc (600M + CTC decoder)
Evaluation Cv Russian	WER 7.1% (220M) / 5.1% (600M)
Evaluation Cv Kazakh	best-in-class (see paper)
Evaluation Fleurs English	WER 12.2% (220M) / 9.4% (600M)
Training Library	pytorch, custom_code

Features: Two things set GigaAM-Multilingual apart from prior multilingual ASR foundation work. First, the scale of pretraining — 2M hours across 70+ languages under a HuBERT-style self-supervised objective — paired with a character-wise CTC decoder fine-tune (no word-piece lexicon dependency) lets a single checkpoint cover languages with very different phonotactics and orthographies (Turkic + Slavic + Indo-Aryan families) without a language-tagged decoding head per family. Second, the two-tier release — encoder-only (ssl/large_ssl) for downstream fine-tuning and encoder+CTC (ctc/large_ctc) ready for inference — gives the community both a drop-in recognizer and a foundation for task-specific adaptation. On Russian Common Voice it beats Seamless M4T large v2 and Omnilingual 1B, the closest open baselines.

Links:

· · · · · · · · · · · · · ·

GigaChat3.1-Audio

Description: GigaChat Audio 10B is an audio-native LLM built on top of the GigaChat 3.1 Lightning text model (10B total parameters, A1.8B active). A Conformer speech encoder and a modality adapter feed audio embeddings directly into the existing Mixture-of-Experts decoder, so the model retains the text quality of its base while adding speech understanding. Capabilities include audio question answering and classification, temporal grounding (event localization in long audio with timestamped descriptions and timestamped audio summarization), emotion recognition (Dusha benchmarks), tool-use, and text-only tasks. The temporal-grounding skills are trained on the purpose-built TimeGround-1M dataset of long-form audio paired with time-aligned annotations. On Russian ASR (Golos crowd), the model reports WER ≈14.7 versus Whisper-large-v3 ≈9.1; on temporal localization it scores mIoU 40.3 / 48.3 on ≤10min / 20–60min clips — substantially above Voxtral (3B) and Phi-4 (4B). Multiple HF Spaces (e.g. hugging-apps/gigaam-multilingual-asr) host live demos of the related GigaAM-Multilingual family.

Release Date: July 13, 2026

Feature	Value
Parameters	10B total (A1.8B active, MoE)
Asr	✅
Languages	Russian, English
Streaming	❌
License
Architecture	audio-native MoE LLM — Conformer speech encoder + modality adapter + MoE decoder
Base Model	ai-sage/GigaChat3.1-10B-A1.8B (GigaChat 3.1 Lightning text model)
Capabilities	audio QA, classification, temporal grounding, emotion recognition, speech translation, tool-use, text-only
Temporal Grounding	yes (timestamped event localization in long audio)
Training Data Audio Grounding	ai-sage/TimeGround-1M (1M long-form audio + time-aligned annotations)
Evaluation Mmau	62.2 (vs Voxtral 3B 59.8, Phi-4 4B 68.3)
Evaluation Mmlu Speech	50.3
Evaluation Audio Math Mqa	72.5
Evaluation Emotion Dusha Crowd	90.0 acc
Evaluation Emotion Dusha Podcast	92.4 acc
Evaluation Asr Ru Golos	WER 14.7
Evaluation Temporal Localization 10M	mIoU 40.3
Evaluation Temporal Localization 20 60M	mIoU 48.3

Features: Two things make GigaChat3.1-Audio stand out among audio LLMs. First, modality-injection into an existing MoE: rather than training a smaller speech-text model from scratch, it taps a 10B-A1.8B MoE text LLM (GigaChat 3.1 Lightning) by inserting a Conformer encoder + linear adapter that project audio embeddings into the existing decoder sequence — so text quality is inherited rather than re-learned. Second, timestamped temporal grounding on long audio: trained on the TimeGround-1M dataset, the model produces event-localization mIoU scores (40.3 / 48.3 on ≤10min / 20–60min clips) that are an order of magnitude above comparable open models (Voxtral 3B 3.4 mIoU on ≤10min; 0.1 on 20–60min), and its audio-summarization output is timestamped rather than just a flat transcript.

Links:

· · · · · · · · · · · · · ·

Audio8-ASR-0.1B

Description: Audio8-ASR-0.1B is a compact autoregressive ASR model whose language-model component has only 0.1B parameters. It supports multilingual speech recognition for Chinese, English, French, German, Japanese, Korean, and Cantonese. The project positions it as one of the smallest usable performance ASR models in the LLM era — and backs that claim with Open ASR Leaderboard results: a composite WER of 7.03% across seven English splits at RTFx 741.15 on H200 (LibriSpeech test-clean 2.70%, GigaSpeech 8.48%, AMI 10.99%, Earnings22 12.31%). On Chinese internal evals it reports CER 8.84% (WenetSpeech meeting) and 7.98% (WenetSpeech net).

Release Date: July 10, 2026

Feature	Value
Parameters	0.1B (LM component)
Asr	✅
Languages	Chinese, English, French, German, Japanese, Korean, Cantonese (7)
Streaming	❌
License
Architecture	autoregressive ASR (Whisper-style encoder + MLP adapter + Qwen decoder, per shared paper)
Pipeline Tag	automatic-speech-recognition
Library Name	transformers (custom_code)
Open Asr Leaderboard Composite Wer	7.03%
Open Asr Leaderboard Rtfx	741.15 (H200, BF16, eager attention, greedy decoding)
Eval Librispeech Clean	WER 2.70%
Eval Librispeech Other	WER 6.59%
Eval Gigaspeech Clean	WER 8.48%
Eval Ami Cleaned	WER 10.99%
Eval Earnings22	WER 12.31%
Eval Spgispeech	WER 3.73%
Eval Voxpopuli	WER 4.39%
Eval Wenetspeech Meeting	CER 8.84%
Eval Wenetspeech Net	CER 7.98%
Deployment Variants	ONNX Runtime (~1.1 GB peak), iOS ANE (~200 MB peak), base Transformers
Hotword Support	yes (tagged in HF metadata)
Companion Model	ARK-ASR-3B (3B-scale sibling, same paper + repo)
Createdat	2026-07-10T06:57:05Z
Downloads	~527

Features: Audio8-ASR-0.1B's pitch is usable ASR quality at 0.1B LM parameters — a scale where most LLM-era ASR systems are either too large for edge deployment or too low-quality for production. The composite WER of 7.03% on the Open ASR Leaderboard (with LibriSpeech clean at 2.70%) demonstrates that a 0.1B-parameter autoregressive decoder, paired with a Whisper-style encoder and MLP adapter, can hit competitive accuracy. The deployment-variant strategy is the other half of the story: the same model ships as ONNX Runtime (~1.1 GB peak for edge devices) and iOS ANE (~200 MB peak for local iPhone transcription) — the latter being a footprint that makes on-device ASR practical on consumer phones without server roundtrips. The hotword support (tagged in HF metadata) adds contextual biasing for domain-specific vocabulary, which is critical for meeting / medical / legal transcription at this parameter scale.

Links:

· · · · · · · · · · · · · ·

MOSS-Transcribe-Diarize

Description: MOSS-Transcribe-Diarize 0.9B is OpenMOSS's open-source SOTA end-to-end audio-understanding model for long-form multi-speaker transcription. Instead of stitching a separate ASR and diarization system, the model jointly transcribes speech and assigns speaker labels, producing time-aligned text with precise timestamps and consistent speaker tags such as [S01], [S02] in a single pass. Built for meetings, calls, podcasts, interviews, lectures, and video content. Optional acoustic-event annotations extend the output to "[start][Sxx]text[end]" segments plus [event] tags for non-speech sounds. Architecture: Qwen3-style 0.6B text backbone + Whisper-Medium audio encoder (16 kHz, 80 mel bins, 30 s chunks) joined by a 4× temporal merge + MLP audio-text adaptor.

Release Date: July 9, 2026

Feature	Value
Parameters	0.9B total
Voice Cloning	❌
Asr	✅
Pronunciation	✅
Emotion Control	❌
Languages	Multilingual (model card: long-form speech unspecified languages; English demo)
Streaming	✅
License
Output Format	"[start_seconds][Sxx]text[end_seconds]" segments
Diarization	built-in ([S01], [S02], ...)
Acoustic Event Tags	yes (optional)
Text Backbone	Qwen3-0.6B causal decoder
Audio Encoder	Whisper-Medium, 16 kHz, 80 mel bins, 30 s chunks
Audio Text Bridge	4× temporal merge + MLP adaptor
Inference	vLLM, SGLang Omni, transformers, HF inference API

Features: Joint speech transcription + speaker diarization in a single end-to-end checkpoint with the canonical [Sxx] output format embedded in the transcript. Most production pipelines stitch a separate ASR model to a separate diarization model (and reconcile speaker turns post-hoc) — MOSS-Transcribe-Diarize folds them into one model so the timestamps, speaker labels, and event tags come out of the same forward pass. Adding a Qwen3-style text decoder on top of Whisper-Medium audio features lets the model explicitly reason about speaker continuity in the textual stream rather than as a postprocess.

Links:

· · · · · · · · · · · · · ·

Typhoon ASR Streaming

Description: Typhoon ASR Streaming is SCB DataX's system for steerable, low-latency Thai speech recognition. Open Thai ASR had been dominated by offline Whisper-style models that read the whole utterance before transcribing; this release replaces them with cache-aware streaming encoders that stay accurate at low latency, plus a decode-time shallow-fusion layer (GPU 4-gram LM + phrase boosting) that steers the vocabulary toward names and domain terms at runtime — no retraining, and under a 3% change in the real-time factor. At a 1040 ms look-ahead, forcing the old full-context Typhoon ASR Real-time model to stream collapses to 62.8% CER on TVSpeech; the cache-aware models reach 19.4% (115M) and 14.1% (0.6B) — a 4.5× CER reduction. The release ships the two acoustic checkpoints, paired sub-word 4-gram fusion LMs, an OpenAI-Realtime-style WebSocket ASR server, a LiveKit Agents STT adapter, a Gradio demo and a Next.js steering front-end.

Release Date: July 5, 2026

Feature	Value
Languages	Thai (th-TH); the 0.6B tokenizer also carries English sub-word units for code-switched terms
Streaming	✅
License
Parameters	115M (converted) / 0.6B (adapted, best accuracy, default)
Architecture	FastConformer-Transducer with cache-aware streaming encoders
Variants	typhoon-asr-streaming-115m (converted from Typhoon ASR Real-time, single 2048-token Thai BPE, no prompt) and typhoon-asr-streaming-nemotron-0.6b (adapted from NVIDIA Nemotron streaming ASR, 15,135-token multilingual + Thai tokenizer, requires `target_lang="th-TH"`)
Fusion	GPU 4-gram LM shallow fusion + phrase boosting, applied at decode time (alpha = beta = 0.5)
Fusion Cost	< 3% RTF
Latency Options	1040 / 480 / 80 ms look-ahead (minimum 80 ms)
Cer Streaming 1040Ms Tvspeech	full-context forced to stream 62.8 → 115M 19.4 → 0.6B 14.1
Cer Streaming 1040Ms Gigaspeech2	full-context forced to stream 39.8 → 115M 10.4 → 0.6B 9.3
Steering 0.6B	baseline 14.4% CER / 16.6% keyword recall → +n-gram 14.6 / 20.4 → +phrase boost 14.1 / 20.7
Efficiency	batch-1 RTF 0.020–0.024 (42–50× faster than real time); batch-16 ≈300 concurrent streams per H100
First Token	≈ look-ahead + ~25 ms
Serving	OpenAI Realtime-style WebSocket server, LiveKit Agents STT adapter, Gradio demo, Next.js front-end
Dependency	NVIDIA NeMo pinned to source commit `907edfd`
Evaluation Data	TVSpeech and GigaSpeech2-Thai
Ngram Rebuild	`scripts/` rebuilds the 0.6B n-gram from your own corpus
Provenance	the contaminated n-gram used as a memorization upper bound in the paper is not distributed
Venue	IEEE SLT 2026 system demo (SCB DataX, Bangkok)

Features: The interesting move is treating latency as a runtime dial rather than a training-time commitment. Cache-aware streaming encoders make the look-ahead a decode parameter, so one checkpoint spans 1040 ms down to 80 ms — and the release exposes exactly that trade-off in an interactive demo that streams the same Thai clip through the full-context baseline and both cache-aware models side by side. The baseline is the argument: at 80 ms look-ahead it "collapses into garble", which is the honest version of the claim that cache-aware training is not optional for low-latency streaming. The second bet is decode-time vocabulary steering: shallow fusion with a GPU n-gram LM plus phrase boosting lets a user inject names like Betagro or กชพร mid-stream, which matters disproportionately in Thai broadcast speech where rare proper nouns and code-switched English jargon dominate the error budget. Crucially the paper reports both halves of the trade — phrase boosting pulls CER from 14.4% to 14.1% while lifting keyword recall from 16.6% to 20.7%, and the whole fusion layer costs under 3% RTF — rather than reporting only the accuracy win. The repo also declines to ship the contaminated n-gram it used as a memorization upper bound, keeping only the honest fusion artifacts.

Links:

· · · · · · · · · · · · · ·

ARK-ASR-3B

Description: ARK-ASR-3B is a 3B-scale audio-capable autoregressive Transformers model for automatic speech recognition. It achieves current state-of-the-art results on the Hugging Face Open ASR Leaderboard English short-form benchmark, with an average WER of 5.04% and RTFx of 490.98 across AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, and VoxPopuli. On Chinese benchmarks it reports CER 1.80% (AISHELL-1), 4.97% (WenetSpeech test meeting), and 4.58% (WenetSpeech test-net).

Release Date: June 22, 2026

Feature	Value
Parameters	3B (decoder LLM + Whisper-style encoder + MLP adapter)
Asr	✅
Languages	19 (Chinese, English, German, Japanese, French, Korean, Spanish, Polish, Italian, Romanian, Hungarian, Czech, Dutch, Finnish, Croatian, Slovak, Slovene, Estonian, Lithuanian)
Streaming	❌
License
Architecture	Whisper-style audio encoder (RoPE) + MLP adapter + Qwen decoder
Pipeline Tag	automatic-speech-recognition
Library Name	transformers (custom_code); vLLM serving supported
Input Sample Rate	16,000 Hz
Checkpoint Format	safetensors
Open Asr Leaderboard Avg Wer	5.04% (current SOTA on English short-form)
Open Asr Leaderboard Rtfx	490.98
Eval Ami	WER 8.79%
Eval Earnings22	WER 8.23%
Eval Gigaspeech	WER 6.98%
Eval Librispeech Clean	WER 1.03%
Eval Librispeech Other	WER 2.35%
Eval Spgispeech	WER 2.46%
Eval Voxpopuli	WER 5.47%
Eval Aishell1	CER 1.80%
Eval Wenetspeech Meeting	CER 4.97%
Eval Wenetspeech Net	CER 4.58%
Companion Model	Audio8-ASR-0.1B (0.1B-scale sibling, same paper + repo)
Createdat	2026-06-22T07:04:45Z
Downloads	~7,109

Features: ARK-ASR-3B's headline result is 5.04% average WER on the Open ASR Leaderboard — the current state of the art on that benchmark at the time of release, beating the project's own 0.6B variant (5.97%) and prior open ASR systems. The architecture is a straightforward but well-tuned combination: a Whisper-style encoder with RoPE (proven audio representation), an MLP adapter (dimension bridging), and a Qwen decoder (strong multilingual text LLM). The key design choice is injecting audio embeddings by replacing audio placeholder token embeddings in the Qwen decoder before transcript generation — this lets the model inherit the Qwen LLM's multilingual text capability (19 languages including low-resource Baltic and Slavic languages) while adding speech input through a minimal-modification adapter rather than a full architecture redesign. The vLLM serving support makes the 3B model practical for production batch transcription, and the shared training repo (AutoArk/open-audio-opd) with the 0.1B sibling lets users trade accuracy for footprint on the same code base.

Links:

· · · · · · · · · · · · · ·

Mega-ASR

Description: Mega-ASR is a robust ASR foundation model trained to withstand the full spectrum of real-world acoustic degradation: 7 atomic acoustic conditions (reverberation, echo, additive noise, far-field, frequency dropout, bandwidth limitation, clipping distortion) and 54 compound environmental scenarios built on top. The inference stack combines Qwen3-ASR-1.7B as backbone, Mega-ASR adaptation weights, and an audio-quality router that picks per-utterance between the robust Mega-ASR path and the base path to preserve clean-speech quality. Robustness is trained via A2S-SFT plus DG-WGPO reinforcement learning, with reported gains up to ~30 % over leading open- and closed-source SOTA models on adversarial acoustic samples.

Release Date: May 19, 2026

Feature	Value
Parameters	1.7B (Qwen3-ASR backbone) + Mega-ASR adapter + audio-quality router
Languages	English, Chinese
Streaming	❌
License
Architecture	Qwen3-ASR-1.7B base + adaptation weights + audio-quality router
Training Data	~2.4M samples (Voices-in-the-Wild-2M)
Robustness Atoms	7 (reverb, echo, additive noise, far-field, freq dropout, bandwidth limit, clipping)
Robustness Compound Scenarios	54
Training	A2S-SFT + DG-WGPO RL
Base Model	Qwen/Qwen3-ASR-1.7B

Features: The first foundation ASR explicitly trained to handle full-scenario in-the-wild acoustic conditions (7 atomic effects, 54 compound scenarios, 2.4 M samples), combined with a routed inference path that switches between a robust Mega-ASR adapter and the base Qwen3-ASR backbone via an audio-quality classifier — delivering up to ~30 % WER gains over SOTA while staying fully open under Apache-2.0.

Links:

· · · · · · · · · · · · · ·

Higgs-Audio-v3-8B-STT-v2

Description: Higgs Audio v3 8B STT v2 is the larger-scale speech-to-text model from Boson AI, combining a Whisper-Large-v3 encoder with a Qwen3-8B decoder (8.91B total parameters), fine-tuned with LoRA on diverse ASR benchmarks. It is the larger sibling of the 2.68B higgs-audio-v3-stt and shares the same custom architecture (trust_remote_code=True required). The model ships with full model-index benchmark results across 8 Open ASR Leaderboard splits:

Release Date: April 27, 2026

Feature	Value
Parameters	8.91B (Whisper-Large-v3 encoder + Qwen3-8B decoder, LoRA fine-tuned)
Asr	✅
Languages	English
Streaming	❌
License
Architecture	Whisper-Large-v3 encoder + Qwen3-8B decoder (custom `higgs_audio_3` code)
Pipeline Tag	automatic-speech-recognition
Library Name	transformers (custom_code)
Fine Tuning	LoRA on diverse ASR benchmarks
Base Model	bosonai/higgs-audio-v3-8b
Eval Librispeech Clean	WER 1.25%
Eval Librispeech Other	WER 2.38%
Eval Spgispeech	WER 3.60%
Eval Tedlium V3	WER 3.09%
Eval Voxpopuli	WER 5.92%
Eval Gigaspeech	WER 8.47%
Eval Earnings22	WER 8.73%
Eval Ami Ihm	WER 10.14%
Leaderboard	Open ASR Leaderboard (hf-audio/open_asr_leaderboard)
Companion Stt Small	bosonai/higgs-audio-v3-stt (2.68B variant)
Companion Tts	bosonai/higgs-audio-v3-tts-4b (the TTS sibling)
Createdat	2026-04-27T17:29:37Z
Downloads	~1,133

Features: The 8B STT v2 is the scaled-up sibling of the 2.68B base STT, pairing the same Whisper-Large-v3 encoder with a Qwen3-8B decoder (vs the base's smaller Qwen3 decoder) and fine-tuning with LoRA rather than full fine-tuning — keeping the 8B backbone's text capability while adapting the ASR head. The full model-index benchmark results (LibriSpeech clean WER 1.25%, GigaSpeech 8.47%) are published directly in the model card's model-index metadata, making the evaluation transparent and reproducible. The model shares the same custom higgs_audio_3 remote code as the base STT, so users can swap between the 2.68B and 8.91B variants without changing their inference pipeline.

Links:

· · · · · · · · · · · · · ·

Higgs-Audio-v3-STT

Description: Higgs Audio v3 STT is a speech-to-text model from Boson AI combining a Whisper-Large-v3 encoder with a Qwen3 decoder (2.68B total parameters). It is the ASR sibling of the Higgs Audio v3 TTS model (higgs-audio-v3-tts-4b), sharing the same Qwen3 decoder family but specialized for speech recognition rather than speech synthesis. The model uses a custom architecture (trust_remote_code=True required) and is fine-tuned on public train splits of AMI (IHM), VoxPopuli (en), SPGISpeech, LibriSpeech, TED-LIUM, GigaSpeech, and Earnings22 — with all rows from source recordings that appear in the ESB/Open-ASR test sets excluded from training to prevent contamination.

Release Date: March 25, 2026

Feature	Value
Parameters	2.68B (Whisper-Large-v3 encoder + Qwen3 decoder)
Asr	✅
Languages	English
Streaming	❌
License
Architecture	Whisper-Large-v3 encoder + Qwen3 decoder (custom `higgs_audio_3` code)
Pipeline Tag	automatic-speech-recognition
Library Name	transformers (custom_code)
Training Data	AMI (IHM), VoxPopuli (en), SPGISpeech, LibriSpeech, TED-LIUM, GigaSpeech, Earnings22 (public train splits; ESB/Open-ASR test-set rows excluded)
Repetition Handling	word-repetition cap + phrase-level repetition-loop collapse (deterministic, uniform)
Leaderboard	Open ASR Leaderboard (hf-audio/open_asr_leaderboard), FFASR (treble-technologies/ffasr)
Companion Tts	bosonai/higgs-audio-v3-tts-4b (the TTS sibling)
Createdat	2026-03-25T18:39:22Z
Downloads	~6,886
Checkpoint Update	June 2026 (refreshed fine-tuning data + repetition-loop collapse)

Features: Higgs Audio v3 STT pairs a Whisper-Large-v3 encoder (proven audio representation) with a Qwen3 decoder (strong text LLM) at 2.68B total parameters — a smaller-scale sibling of the 8B variant (higgs-audio-v3-8b-stt-v2). The June 2026 update added a phrase-level repetition-loop collapse alongside the existing word-repetition cap, addressing a common failure mode in autoregressive ASR where the decoder gets stuck repeating a phrase. Both corrections are deterministic and applied uniformly to every dataset, so the leaderboard results are directly comparable across the pre- and post-update checkpoints. The contamination-prevention protocol (excluding ESB/Open-ASR test-set source recordings from training) is a transparency measure that ensures the Open ASR Leaderboard results are clean.

Links:

· · · · · · · · · · · · · ·

Cohere Transcribe

Description: Open-source automatic speech recognition (ASR) model developed by Cohere. A 2 billion parameter dedicated audio-in, text-out model that ranks #1 on the English ASR leaderboard.

Release Date: March 2026

Feature	Value
Parameters	2B
Architecture	Conformer-based encoder-decoder
Asr	✅
Languages	14 (En, Fr, De, It, Es, Pt, Gr, Nl, Pl, Zh, Jp, Ko, Vi, Ar)
Streaming	✅
License
Rtfx	Up to 3x faster than comparable models

Features: - Long-form transcription with automatic chunking (>35 seconds)

Optional punctuation control
Batched inference support
vLLM integration for production serving
Apple Silicon support via mlx-audio
WebGPU browser deployment via transformers.js

Links:

· · · · · · · · · · · · · ·

VibeVoice-ASR

Description: Microsoft's unified speech-to-text model for 60-minute long-form audio processing with speaker diarization and timestamping.

Release Date: January 21, 2026

Feature	Value
Parameters	7B
Asr	✅
Languages	50+
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

SYMPHONY-ASR

Description: SYMPHONY-ASR is Okestro AI Lab's ASR-specialized speech-recognition model built from the earlier SYMPHONY base. It is an LLM-ASR design using Qwen3-4B as the language head, an HFQ-Former audio frontend (hierarchically compressing high-frame-rate audio features), and an audio-text-to-text interface that supports both Korean and English in a single model. The architecture is deliberately ASR-specialized rather than multi-task, allowing it to land anywhere the standard long-form benchmarks (AMI, Earnings-22, GigaSpeech, LibriSpeech clean & other, VoxPopuli, TedLium-3, SPGI Speech) are tracked, with public WER numbers reported on the model card.

Release Date: January 12, 2026

Feature	Value
Parameters	4B (Qwen3-4B backbone)
Languages	English, Korean
Streaming	❌
License
Architecture	HFQ-Former audio encoder + Qwen3-4B LLM adapter + ASR head
Base Model	Qwen/Qwen3-4B
Pipeline Tag	audio-text-to-text
Wer Benchmarks	AMI 9.56, Earnings-22 9.45, GigaSpeech 9.96, LS-clean 1.91, LS-other 4.43, VoxPopuli-en 6.30, TedLium-3 3.39, SPGI 2.29

Features: An ASR-specialized cut of LLM-ASR (Qwen3-4B backbone + HFQ-Former + Adapter) that explicitly trades generalist audio-text flexibility for tighter ASR performance: pinned public WER scores across the canonical long-form English benchmarks plus Korean support in the same model.

Links: []((not stated on this card))

· · · · · · · · · · · · · ·

Fun-ASR

Description: Fun-ASR (Fun-ASR-Nano-2512) is an end-to-end speech recognition large model from Tongyi Lab / FunAudioLLM, trained on tens of millions of hours of real speech data. The Nano variant covers 31 languages including Chinese with multiple dialects (Wu, Cantonese, Min, Hakka, Gan, Xiang, Jin) and 26 regional accents plus English/Japanese, with optional lyric / rap recognition. vLLM-based batch inference is 3-5× faster than baseline, and a llama.cpp / GGUF runtime brings the model to CPU/edge with built-in VAD.

Release Date: December 15, 2025

Feature	Value
Parameters	800M
Languages	31
Streaming	✅
Vad	✅
Punctuation	✅
Diarization	✅
Timestamps	yes
Hotwords	yes
Quant	GGUF (~484MB q-form)
License

Features: Far-field, high-noise end-to-end ASR (conference rooms, in-vehicle, industrial) without the "hallucination" generation and language confusion common to Whisper-class models, plus a vLLM WebSocket streaming SDK that delivers sub-second transcription at scale and a llama.cpp / GGUF edge runtime.

Links:

· · · · · · · · · · · · · ·

GLM-ASR-Nano-2512

Description: GLM-ASR-Nano-2512 is a 1.5B-parameter open-source speech recognition model from Zhipu AI. The "Nano" model is built for real-world complexity: it produces the lowest average error rate (4.10) among comparable open-source ASR models while keeping the checkpoint at the compact 1.5B scale. Key capabilities:

Release Date: December 9, 2025

Feature	Value
Parameters	1.5B
Asr	✅
Languages	17 supported (WER ≤ 20%); Mandarin, English, Cantonese specially optimized
Streaming	❌
License
Architecture	speech-encoder + text decoder (transformers framework)
Wer Avg	4.10 (lowest among comparable open-source ASR)
Benchmarks	Wenet Meeting (real-world meeting scenarios, noise / overlapping speech); Aishell-1 (Mandarin)
Licensed Channels	Hugging Face, ModelScope
Companion Repo	https://github.com/zai-org/GLM-ASR
Inference Lib	transformers (v4.x from source; v5 support planned; vLLM, SGLang planned)
Input Modes	audio URL (via processor.apply_transcription_request) or raw audio arrays
Download Count	~150k (as of mid-2026)
Model Class	GlmAsrForConditionalGeneration (custom_code) / AutoModelForSeq2SeqLM
Demo Spaces	YatharthS/GLM-ASR-Nano, Pevernow/GLM-ASR-Nano, yakoudev/GLM-ASR-Nano, naicoi/GLM-ASR-Nano

Features: Two design decisions set GLM-ASR-Nano apart from the Whisper-V3 class of multilingual ASR. First, explicit dialect-support training: rather than treating dialectal Mandarin as noise to be suppressed, the model was tuned for Cantonese and other regional variants as first-class targets — a capability gap that most prior ASR systems left untouched. Second, specifically trained robustness on low-volume / quiet-speech audio — the regime where conventional ASR often falls back on aggressive denoising that removes the very signal the model needs. The result: the lowest average error rate (4.10) among comparable open-source ASR models, with the strongest advantage on Chinese benchmarks (Wenet Meeting, Aishell-1) — in a 1.5B-parameter checkpoint that integrates directly with the transformers v4.x / v5.x stack.

Links:

· · · · · · · · · · · · · ·

SYMPHONY

Description: SYMPHONY is the earlier LLM-ASR design from Okestro AI Lab that pre-dates the ASR-specialized SYMPHONY-ASR. It uses the same Qwen3-4B backbone, an HFQ-Former audio frontend, and an audio-text-to-text interface for Korean + English. The model card self-identifies SYMPHONY-ASR as the new_version, and the SYMPHONY card itself is now minimal (auto-generated stub after the successor was published). Public WER numbers on the older release are slightly behind SYMPHONY-ASR — e.g. LibriSpeech-clean 2.26 vs 1.91, TedLium-3 3.97 vs 3.39.

Release Date: October 27, 2025

Feature	Value
Parameters	4B (Qwen3-4B backbone)
Languages	English, Korean
Streaming	❌
License
Architecture	HFQ-Former audio encoder + Qwen3-4B LLM adapter
Base Model	Qwen/Qwen3-4B
Pipeline Tag	audio-text-to-text
Status	superseded by SYMPHONY-ASR (Jan 12, 2026)

Features: An earlier general-purpose LLM-ASR recipe (Qwen3-4B + HFQ-Former + Adapter) on the Korean + English bilingual target — preserved here as the version-history parent of the ASR-specialized SYMPHONY-ASR successor.

Links:

· · · · · · · · · · · · · ·

Moonshine

Description: Moonshine is a tiny automatic-speech-recognition family from Useful Sensors, designed for real-time speech transcription on severely memory- and compute-constrained hardware. The release ships two checkpoints (tiny 27M, base 61M), both English-only ASR trained on 200,000 hours of audio + transcript pairs collected from the internet and open HuggingFace datasets. It is the smallest practical Wh-class model family for edge / microcontroller-class deployment among those with public benchmarks against standard ASR datasets.

Release Date: September 26, 2024

Feature	Value
Parameters	27M (tiny), 61M (base)
Asr	✅
Languages	English
Streaming	✅
License
Architecture	sequence-to-sequence ASR
Training Data	200,000 hours (internet + open HF datasets)
Target Hardware	low-cost / edge / MCU-class
Variants English Only	yes

Features: A purpose-built tiny ASR (27M–61M parameters) ranging far below the smallest Whisper-class models while remaining competitive on standard ASR datasets — built specifically so a microcontroller / low-cost hardware developer can run real-time English transcription with usable accuracy, rather than shipping a quantized-down Whisper clone.

Links:

· · · · · · · · · · · · · ·

SenseVoice

Description: SenseVoice is a speech foundation model that combines automatic speech recognition (ASR), spoken-language identification (LID), speech-emotion recognition (SER), and audio-event detection (AED) in a single non-autoregressive architecture. The Small variant processes 10 s of audio in roughly 70 ms, runs CPU/edge via a llama.cpp-compatible GGUF runtime, and is trained on over 400 k hours of data spanning 50+ languages.

Release Date: July 3, 2024

Feature	Value
Parameters	~230M (SenseVoice-Small)
Languages	50+ (multilingual)
Streaming	✅
Vad	✅
Punctuation	✅
Diarization	✅
Timestamps	yes
Emotion Recognition	yes
Audio Event Detection	yes
License

Features: A non-autoregressive end-to-end architecture that runs at 15× the speed of Whisper-Large while bundling ASR + LID + SER + AED in one model, plus a GGUF/llama.cpp path that brings the whole pipeline to CPU/edge devices without Python at runtime.

Links:

· · · · · · · · · · · · · ·

FunASR

Description: Industrial-grade end-to-end speech recognition toolkit with SOTA pretrained models, supporting ASR, VAD, punctuation, language modeling, speaker verification, speaker diarization, and multi-talker ASR across 50+ languages. Architectures are updated frequently and the project tracks upstream ModelScope releases.

Release Date: Ongoing (First: 2023)

Feature	Value
Parameters	~220M (Paraformer-large)
Asr	✅
Languages	50+
Vad	✅
Punctuation	✅
Diarization	✅
Timestamps	yes
Emotion	yes
Streaming	✅
License

Links:

· · · · · · · · · · · · · ·

Audio Codecs & Tokenizers

Audio autoencoders, codecs, and latent-space tokenizers that compress waveforms into compact continuous or discrete tokens — the substrate for downstream TTS, music, and a2a generators.

Audio Codecs & Tokenizers Quick Comparison

Model	Type	Sample Rate	Latent Dim	Modalities
ZipCodec	Streaming Neural Speech Codec	16 kHz	64	Audio
DSA-Tokenizer	Disentangled Semantic-Acoustic Speech Tokenizer	24 kHz	—	Audio
KVAE-Audio	Audio VAE	48 kHz	64	Speech, Music, Sound
LinaCodec	Single-Stream Neural Audio Codec	48 kHz	—	Audio
QuarkAudio-HCodec	Dual-Stream Discrete Audio Codec	16/48 kHz	-	Audio

ZipCodec

Description: ZipCodec attacks the frame rate of neural speech coding rather than only the bitrate. It runs at 6.25 Hz — one token every 160 ms — at 0.80 kbit/s, with a theoretical latency of 160 ms, by combining large-scale WavLM distillation with a redesigned transformer-based encoder/decoder, a scalar spherical quantizer (64 codebooks × 4 entries, i.e. 2 bits per latent dimension) and a latency-aware streaming decoder. The project page compares it head-to-head against EnCodec (1.50 kbps), AudioDec (1.60 kbps), HILCodec (1.50 kbps), Mimi (0.83 kbps), PAST (1.00 kbps), FocalCodec-Stream (0.80 kbps) and FocalCodec (0.65 kbps) on speech resynthesis and voice conversion, and the authors report it substantially outperforms existing streaming codecs at comparable bitrates while operating at a far lower frame rate. Despite 842M parameters, it reaches real-time single-stream inference on a consumer-grade CPU; ONNX, OpenVINO, torch.compile and CUDA-graph backends ship as install extras.

Release Date: September 11, 2026

Feature	Value
Parameters	842M (842,154,100 F32 in the released checkpoint)
License
Type	Streaming Neural Speech Codec (single-stream discrete tokenizer)
Sample Rate	16,000 Hz
Latent Dim	64 (scalar quantized; 64 codebooks × 4 entries = 2 bits per dimension)
Modalities	Audio
Frame Rate	6.25 Hz (one frame per 160 ms)
Bitrate	0.80 kbit/s
Theoretical Latency	160 ms
Base Model	microsoft/wavlm-large (large-scale distillation)
Quantizer	scalar spherical quantizer (64 codebooks × 4 entries)
Decoder	latency-aware streaming decoder
Streaming	✅
Cpu Inference	real-time single-stream on a consumer-grade CPU despite 842M parameters
Tasks	speech resynthesis, voice conversion, streaming
Compared Against	EnCodec 1.50 kbps, AudioDec 1.60 kbps, HILCodec 1.50 kbps, Mimi 0.83 kbps, PAST 1.00 kbps, FocalCodec-Stream 0.80 kbps, FocalCodec 0.65 kbps
Related Work	FocalCodec (same author; 0.65 kbps, non-streaming)
Install	`pip install zipcodec` (extras: `zipcodec[onnx]`, `zipcodec[openvino]`, `zipcodec[streaming]`)
Usage	`torch.hub.load("lucadellalib/zipcodec", "zipcodec", config="lucadellalib/zipcodec", trust_repo=True)`
Api	`from zipcodec import ZipCodec` then `ZipCodec.from_pretrained("lucadellalib/zipcodec")`
Api Core	`codec.wav_to_toks(wav)` → `codec.toks_to_codes(toks)` → `codec.toks_to_wav(toks)`
Export Backends	eager, torch.compile, jit, CUDA graphs, ONNX, ONNX I/O binding, OpenVINO
Benchmark Default	five runs of 40.96 s with four CPU threads
Runtime Requirements	Python 3.10+, PyTorch, NumPy, safetensors, huggingface-hub
Training Data	not stated in the repository

Features: The central bet is that frame rate, not bitrate, is the bottleneck for speech generation pipelines. At 6.25 Hz every token has to carry ~25× more information than at a typical 50 Hz codec, which is why prior work treated sub-10 Hz frame rates as a quality cliff. ZipCodec's arithmetic is unusually legible: 64 scalar dimensions × 2 bits × 6.25 frames/s = 800 bit/s, so the 0.80 kbit/s headline is exactly what the quantizer geometry implies — the compression is bought by making each latent dimension nearly binary rather than by pruning the sequence. Two design choices do the heavy lifting against the resulting reconstruction penalty: large-scale WavLM distillation gives the encoder a strong, already-linguistic feature space to compress into, and a latency-aware streaming decoder lets the decoder exploit only causally available context, so the model is trained for the streaming regime it will actually run in instead of being adapted to it. The 842M parameter count is the counterintuitive part: rather than trading size for latency, ZipCodec keeps a large distilled backbone and buys back real-time CPU inference through backend engineering (ONNX / OpenVINO / CUDA-graph exports are first-class in the repo). Like the other entries in this category, ZipCodec is a drop-in tokenizer for TTS / ASR / voice-conversion pipelines, not a synthesis model itself.

Links:

· · · · · · · · · · · · · ·

DSA-Tokenizer

Description: DSA-Tokenizer is a speech tokenizer that explicitly disentangles semantic and acoustic speech information into independent discrete token streams, designed as a building block for fully discrete Speech LLMs. The two token kinds are supervised with distinct optimization constraints: semantic tokens are trained to low WER/CER under ASR supervision (capturing linguistic content only), while acoustic tokens are trained to reconstruct mel-spectrograms (capturing speaker style / prosody / acoustic texture). The decoder is a hierarchical Flow Matching DiT trained with two strategies — self-reconstruction (predict the velocity field of the full mel-spectrogram from complete acoustic + semantic tokens) and recombination / contextual inpainting (predict the masked mel-spectrogram region given the unmasked acoustic context + the full semantic tokens). The DiT is distilled to 4-step inference and fine-tuned with GAN for synthesis quality, while supporting cross-utterance voice clone through the disentangled token streams.

Release Date: July 21, 2026

Feature	Value
Parameters	not stated (hierarchy of encoders + DiT decoder + vocoder)
Type	Disentangled Semantic-Acoustic Speech Tokenizer (dual-stream codec)
Sample Rate	24,000 Hz (vocoder: flow2gan_mel_24k_base_50Hz)
Latent Dim	discrete semantic tokens + discrete acoustic tokens (separate codebooks; vocab.txt ships)
Token Rate	50 Hz (mel2gan vocoder)
Modalities	Audio
License
Architecture	Two-stream encoder (semantic + acoustic branches) → hierarchical Flow Matching DiT decoder → GAN-fine-tuned vocoder (flow2gan_mel_24k_base_50Hz)
Semantic Supervision	ASR loss (low WER/CER)
Acoustic Supervision	mel-spectrogram reconstruction loss
Decoder Optimisation	Flow Matching + 4-step distillation + GAN fine-tuning
Training Strategies	self-reconstruction + recombination / contextual inpainting (joint)
Inference Steps	4 (distilled)
Inference Pattern	`from f5_tts.dsa_api import DSATokenizer; DSATokenizer.from_pretrained('DiscreteSpeech/dsa-tokenizer', device='cuda:0')`
Downstream Tasks Supported	SpeechLLM-TTS (e.g. Llasa 1B/3B/8B, Spark-TTS), SpeechLLM-VC, speech reconstruction, voice clone across utterances
Disentanglement Quality	outperforms WavTokenizer/Mimi/Encodec/SpeechTokenizer/DualCodec/SAC on disentanglement probing (low WER + low speaker similarity on semantic; high WER + high speaker similarity on acoustic)
Bundle Files	`dsa_tokenizer/model_10000.pt`, `dsa_tokenizer/vocab.txt`, `dsa_tokenizer/config.yaml`; vocoder `flow2gan_mel_24k_base_50hz/epoch-20.pt`
Pipeline Tag	text-to-speech (the canonical use is LLM-TTS); tags include voice-cloning, tokenizer, speech
Library	zips under `f5_tts.dsa_api` integration
Demo Url Format	anonymous.4open.science uses `/w/.../index.html` (root URL returns 400 folder_not_supported; index.html works)
Downloads	0 (released July 21 2026)

Features: Two bets together are the technical center of DSA-Tokenizer. The first is explicit semantic-acoustic disentanglement via distinct optimization constraints: rather than training a single representational bottleneck and hoping semantic and acoustic content separate (as prior fusing tokenizers do), DSA trains semantic tokens against ASR supervision and acoustic tokens against mel reconstruction as separate streams — so the disentanglement is engineered into the loss surface, not extracted post-hoc. The second is a hierarchical Flow Matching decoder with joint self-reconstruction + contextual-inpainting training, which lets the same decoder both reconstruct complete utterances and inpaint missing regions using surrounding acoustic context (the characteristic operation needed for voice conversion / segment editing). Coupled with 4-step distillation and GAN fine-tuning for inference speed and audio quality, the result is a single tokenizer that supports high-fidelity speech reconstruction and cross-utterance voice clone and LLM-TTS (paired with Llasa 8B, Spark-TTS, etc.) — and on the project's disentanglement probing beats WavTokenizer / Mimi / Encodec / SpeechTokenizer / DualCodec / SAC at producing a truly separable semantic / acoustic representation.

Links:

· · · · · · · · · · · · · ·

KVAE-Audio

Description: KVAE-Audio is a continuous, full-band (48 kHz) audio autoencoder by Kandinsky Lab / SberBank. It compresses raw waveforms into compact continuous latents (64-dim) and reconstructs them with high fidelity across speech, music, and general sound. The model is intended both for reconstruction and as a latent space for downstream generative models — internal ablations show that swapping the autoencoder under a fixed generator (same DiT, same data, same steps) consistently improves generative quality.

Release Date: June 29, 2026

Feature	Value
Parameters	166.9M
Sample Rate	48 kHz full-band
Latent Dim	64
Modalities	speech, music, sound
Reconstruction Quality	beats MMAudio/DACVAE/SAME-L on AudioCaps FAD
Architecture	VAE (continuous latents)
Related Papers	2412.15322, 2410.13720, 2605.18613
License

Features: Full-band 48 kHz continuous audio VAE with a comparatively tiny 64-dimensional latent space, designed as a drop-in replacement for prior codec/VAEs for generative pipelines — internal benchmarks show lower FAD on AudioCaps and Song Describer than MMAudio, DACVAE (MovieGen), and SAME-L under a fixed generator.

Links:

· · · · · · · · · · · · · ·

LinaCodec

Description: LinaCodec is a highly-compressive neural audio tokenizer designed for speech models. Audio is compressed to just 12.5 tokens per second (171 bps) and decoded back to 48 kHz waveforms — roughly 60× more compressed than DAC (774 tokens/sec, 44.1 kHz), 24× more than EnCodec (300 tokens/sec, 24 kHz), 4× more than Xcodec2 (50 tokens/sec, 16 kHz), and 16× more than Mimi (200 tokens/sec, 24 kHz). The encoder runs at 200× real-time; the decoder at 400× real-time (faster with batching). Although this is a tokenizer rather than a synthesis model, it supports indirect downstream tasks like voice conversion, audio super-resolution, and audio denoising. Designed for direct drop-in use as the audio tokenizer of TTS/ASR pipelines — reported use case is enabling TTS models to run at 800× real-time, ~8× faster than the same author's MiraTTS, and faster training (high-quality TTS in < 1 day).

Release Date: January 2, 2026

Feature	Value
Parameters	not stated (encoder + decoder transformer pair)
License
Type	Neural audio codec (single-stream discrete tokenizer)
Sample Rate	48,000 Hz
Latent Dim	— (token-stream tokenizer; discrete codebook rather than continuous latent)
Modalities	Audio
Tokens Per Second	12.5 (171 bps)
Compression Vs Dac	~60× tighter (vs DAC 774 tokens/s @ 44.1 kHz)
Compression Vs Encodec	~24× tighter (vs EnCodec 300 tokens/s @ 24 kHz)
Compression Vs Xcodec2	~4× tighter (vs Xcodec2 50 tokens/s @ 16 kHz)
Compression Vs Mimi	~16× tighter (vs Mimi 200 tokens/s @ 24 kHz)
Encoder Speed	200× real-time
Decoder Speed	400× real-time (faster with batching)
Downstream Tasks	TTS, ASR, voice conversion (indirect), audio super-resolution (indirect), audio denoising (indirect)
Inference Install	`pip install git+https://github.com/ysharma3501/LinaCodec.git`
Usage Inference	`from linacodec.codec import LinaCodec`
Author Other Work	MiraTTS (https://github.com/ysharma3501)
Inference Speed Claim	enables TTS at 800× real-time (~8× faster than MiraTTS)
Training Speed Claim	high-quality TTS in < 1 day
Variants	single 48 kHz checkpoint at HF

Features: The architectural decision that drives LinaCodec's pitch is the 12.5 tokens-per-second target rate. That number sits substantially below every comparable neural audio codec (Xcodec2 50, Mimi 200, EnCodec 300, DAC 774) — an order-of-magnitude compression jump relative to EnCodec / DAC and ~4× tighter than the previous state-of-the-art at the time of release. The pragmatic consequence for TTS pipelines is that the sequence length the LLM decoder has to traverse is proportionally shorter, so attention cost collapses quadratically, and the same TTS backbone sped up by 8× vs the author's MiraTTS hit "~800× real-time" with the codec swap. The "codec isn't a TTS model" caveat matters: LinaCodec shows up here as a drop-in tokenizer for TTS / ASR pipelines, not a dialogue or voice-synthesis model itself — the comparison-table columns reflect it isn't measuring the same thing as the TTS rows above.

Links:

· · · · · · · · · · · · · ·

QuarkAudio-HCodec

Description: H-Codec is a unified, dual-stream discrete audio tokenizer that quantizes acoustic and semantic features into independent codebooks, preserving both signal fidelity and linguistic content separately. Three versions release alongside the paper:

Release Date: December 23, 2025

Feature	Value
License
Modalities	audio
Zero Shot Voice	no (tokenizer, not a synthesis model)
Architectures	Acoustic quantizer + Semantic quantizer (separate codebooks)
Inference	encoder → 2× quantizers → discrete tokens; decoder reconstructs waveform
Variants	1.0 (16 kHz fixed), 1.5 (16 kHz adaptive), 2.0 (48 kHz fixed)
Frame Rate	Fixed (1.0, 2.0); Adaptive (1.5)
Sample Rate	16 kHz (1.0, 1.5); 48 kHz (2.0)
Ssl Backbone	WavLM (used as encoder for semantic stream)
Downstream	TTS, VC, audio editing, TTA, SE

Features: A dual-stream neural audio codec with separate codebooks for acoustic and semantic quantization — instead of fusing the two before discretizing. This separation lets the acoustic codebook preserve high-frequency detail while the semantic codebook retains linguistic content for downstream LLM-conditioned generation. The adaptive-frame-rate variant (1.5) further reduces token count on temporally simple content, lowering LLM training and inference cost.

Links:

· · · · · · · · · · · · · ·

Audio Transcription Models

Multi-stance audio annotators — specialized models that transcribe not just speech but also musical audio (lyrics, song structure). Often used to label training data for downstream speech or music generation.

Audio Transcription Models Quick Comparison

Model	Use Case	Input	Languages	Base Model	License
ACE-Step Transcriber	Music data labeling	Audio	50+	Qwen2.5 Omni 7B

ACE-Step Transcriber

Description: ACE-Step Transcriber is the annotation model that ACE-Step used to label its v1.5 music-generation training data. It is a multilingual audio annotation model built on Qwen2.5 Omni 7B that transcribes both speech and singing voice with high accuracy, then automatically identifies song structural elements (verse, chorus, bridge, intro, outro, pre/post-chorus, instrumental interludes). Output is structured: # Languages + # Lyrics blocks with [Section Tag - Optional Instrument] markers.

Release Date: January 23, 2026

Feature	Value
Asr	✅
Languages	50+
Streaming	❌
License
Architecture	Qwen2.5 Omni 7B base, multi-modal audio-text-to-text
Pipeline Tag	audio-text-to-text
Target Output	structured `# Languages` + `# Lyrics` with `[Section Tag]` markers (verse, chorus, bridge, intro, outro, pre-chorus, post-chorus, intro/outro, guiter interlude, instrumental, spoken)
Training Role	annotation / labeling model for ACE-Step v1.5 music data
Modalities	speech + singing voice + musical structure
Datasets	multilingual (50+), music + speech corpora

Features: A specialized audio annotator that bridges speech recognition and music-structure understanding: it labels both what is being said and how the song is organized (verse/chorus/bridge and instrumental boundaries) in one pass — built specifically to label the training corpus of the ACE-Step v1.5 music model, but usable as a stand-alone multilingual lyrics / audio-structure transcriber.

Links:

· · · · · · · · · · · · · ·

Additional Resources

Community-maintained leaderboards for tracking and comparing speech models across providers and benchmarks.

Text-to-Speech

Artificial Analysis — Text-to-Speech Leaderboard — Elo-based speech arena ranking TTS models on each provider's native voices, with API pricing and arena voice counts.

Speech-to-Text (ASR)

🤗 Open ASR Leaderboard · Paper (arXiv 2510.06961) — Hugging Face community benchmark ranking open and proprietary ASR models by average WER and RTFx across English and multilingual datasets.
Artificial Analysis — Speech-to-Text (Streaming) Leaderboard — Compares streaming STT models and providers on the AA-WER Streaming index, latency, and pricing.
FFASR Leaderboard (Treble Technologies) — Far-field ASR multi-condition benchmark reporting WER and RTFx across noisy, reverberant, and SNR-stratified scenarios (measured & simulated RIRs).

Voice Cloning & Speaker Privacy

RVCBench · Paper (arXiv 2602.00443) · Dataset · Demo — First large-scale robustness benchmark for voice cloning + speaker privacy: 27 TTS/VC adversary models, 10 dataset configurations, 5 audio protection methods, with standardised speaker-similarity, intelligibility, and perceptual-quality metrics.

Contributing

This list is continuously evolving. If you have any models to add or updates to suggest, please feel free to contribute! See CONTRIBUTING.md for the template-driven workflow.

Last Updated: November 2026

Top AI Repos — open-source AI, indexed and scored

Trend

From the README

Awesome TTS & Voice Generation Models

Table of Contents

Text-to-Speech (TTS) Models

TTS Quick Comparison

Iroha-TTS-V1

OGG TTS

Ming-3.0-tts-0.5B

Fish Audio Drama 3

Resona

AuK

AuK-Flash

rumik-oss 1

Irodori-TTS-v4.1-Anime

ICE-012 Audio

Coqui TTS Anonymizer (rm00cr/coqui-tts)

TontaubeV1

Breeze TTS 2

Sopro v2 Turbo

CuteTTS

Rynsan TTS

Audio8 TTS Preview 0.1B

Kiseki-TTS

FireRedTTS3

Audio8-TTS-Preview-0.6b

NeuTTS-2E

Scylla's Band

sanoTTS

WordVoice-base-0.5B

FreyaTTS

Inflect-Nano-v2

Gepard

Higgs Audio v3 TTS

dots.tts

Confucius4-TTS

WavTTS

MOSS-TTS

VoxFlash-TTS

Miso TTS

Raon-OpenTTS-1B

OronTTS

Supertonic 3

Scenema Audio

Dramabox

Sarashina2.2-TTS

LongCat-AudioDiT

VoXtream2

SILMA TTS

Fish Audio S2 Pro

LongCat-Next

Voxtral-4B-TTS

Blue (Light Blue) TTS

KittenTTS

Ming-omni-tts

SoulX-Singer

SoproTTS

Qwen3-TTS

TADA

Irodori-TTS-500M-v2

KugelAudio

LEMAS-TTS

MioTTS-2.6B

MOSS-TTS-Nano

NeuTTS

OmniVoice

T5Gemma-TTS

TinyTTS

VoxCPM2

Soprano

GLM-TTS

Echo-TTS

VibeVoice-Realtime

Fun-CosyVoice 3.0

LFM2-Audio-1.5B

Marvis-TTS

IndexTTS2

Maya1

Step-Audio-EditX