Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of Voicebox, new SOTA Text-to-speech network from MetaAI, in Pytorch
| Date | Stars |
|---|---|
| 2026-07-24 | 699 |
| 2026-07-25 | 699 |
| 2026-07-28 | 699 |
| 2026-07-30 | 699 |
| 2026-07-31 | 699 |
| 2026-08-06 | 699 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img src="./images/voicebox.png" width="400px"></img>
## Voicebox - Pytorch
Implementation of <a href="https://arxiv.org/abs/2306.15687">Voicebox</a>, new SOTA Text-to-Speech model from MetaAI, in Pytorch. <a href="https://about.fb.com/news/2023/06/introducing-voicebox-ai-for-speech-generation/">Press release</a>
In this work, we will use rotary embeddings. The authors seem unaware that ALiBi cannot be straightforwardly used for bidirectional models.
The paper also addresses the issue with time embedding incorrectly subjected to relative distances (they concat the time embedding along the frame dimension of the audio tokens). This repository will use adaptive normalization, as applied successfully in <a href="https://arxiv.org/abs/2211.07292">Paella</a>
Update: Recommend you just use <a href="https://github.com/lucidrains/e2-tts-pytorch">E2 TTS</a> instead of this work
## Appreciation
- <a href="https://translated.com"><img style="vertical-align: middle;" src="./images/translated.png" height="20px" alt="Translated"><img></a> for awarding me the <a href="https://imminent.translated.com/research-grants-ceremony-innovations-in-language-technology">Imminent Grant</a> to advance the state of open sourced text-to-speech solutions. This project was started and will be completed under this grant.
- <a href="https://stability.ai/">StabilityAI</a> for the generous sponsorship, as well as my other sponsors, for affording me the independence to open source artificial intelligence.
- <a href="https://github.com/b-chiang">Bryan Chiang</a> for the ongoing code review, sharing his expertise on TTS, and pointing me to <a href="https://github.com/atong01/conditional-flow-matching">an open sourced implementation</a> of conditional flow matching
- <a href="https://github.com/manmay-nakhashi">Manmay</a> for getting the repository started with the alignment code
- <a href="https://github.com/chenht2010">@chenht2010</a> for finding a bug with rotary positions, and for validating that the code in the repository converges
- <a href="https://github.com/lucasnewman">Lucas Newman</a> for (yet again) pull requesting all the training code for Spear-TTS conditioned Voicebox training!
- <a href="https://github.com/lucasnewman">Lucas Newman</a> has demonstrated that the whole system works with Spear-TTS conditioning. Training converges even better than <a href="https://github.com/lucidrains/soundstorm-pytorch">Soundstorm</a>
## Install
```bash
$ pip install voicebox-pytorch
```
## Usage
Training and sampling with `TextToSemantic` module from <a href="https://github.com/lucidrains/spear-tts-pytorch">SpearTTS</a>
```python
import torch
from voicebox_pytorch import (
VoiceBox,
EncodecVoco,
ConditionalFlowMatcherWrapper,
HubertWithKmeans,
TextToSemantic
)
# https://github.com/facebookresearch/fairseq/tree/main/examples/hubert
wav2vec = HubertWithKmeans(
checkpoint_path = '/path/to/hubert/checkpoint.pt',
kmeans_path = '/path/to/hubert/kmeans.bin'
)
text_to_semantic = TextToSemantic(
wav2vec = wav2vec,
dim = 512,
source_depth = 1,
target_depth = 1,
use_openai_tokenizer = True
)
text_to_semantic.load('/path/to/trained/spear-tts/model.pt')
model = VoiceBox(
dim = 512,
audio_enc_dec = EncodecVoco(),
num_cond_tokens = 500,
depth = 2,
dim_head = 64,
heads = 16
)
cfm_wrapper = ConditionalFlowMatcherWrapper(
voicebox = model,
text_to_semantic = text_to_semantic
)
# mock data
audio = torch.randn(2, 12000)
# train
loss = cfm_wrapper(audio)
loss.backward()
# after much training
texts = [
'the rain in spain falls mainly in the plains',
'she sells sea shells by the seashore'
]
cond = torch.randn(2, 12000)
sampled = cfm_wrapper.sample(cond = cond, texts = texts) # (2, 1, <audio length>)
```
For unconditional training, `condition_on_text` on `VoiceBox` must be set to `False`
```python
import torch
from voicebox_pytorch import (
VoiceBox,
ConditionalFlExcerpt of 8,707 characters
Read on GitHub
Phil Wang · United States
125
Lucas Newman · United States
10
5
manmay nakhashi · India
4
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:37dae2fd50a4fca3, topic:text-to-speech, desc:text-to-speech, readme:text-to-speech
matched fp:37dae2fd50a4fca3, topic:deep-learning