Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of Natural Speech 2, Zero-shot Speech and Singing Synthesizer, in Pytorch
| Date | Stars |
|---|---|
| 2026-07-24 | 1333 |
| 2026-07-25 | 1333 |
| 2026-07-28 | 1333 |
| 2026-07-30 | 1333 |
| 2026-08-06 | 1333 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img src="./diagram.png" width="450px"></img>
<img src="./diagram2.png" width="450px"></img>
## Natural Speech 2 - Pytorch (wip)
Implementation of <a href="https://arxiv.org/abs/2304.09116">Natural Speech 2</a>, Zero-shot Speech and Singing Synthesizer, in Pytorch
NaturalSpeech 2 is a TTS system that leverages a neural audio codec with continuous latent vectors and a latent diffusion model with non-autoregressive generation to enable natural and zero-shot text-to-speech synthesis
This repository will use denoising diffusion rather than score-based SDE, and may potentially offer elucidated version as well. It will also offer improvements for the attention / transformer components wherever applicable.
## Appreciation
- <a href="https://stability.ai/">Stability</a> and <a href="https://huggingface.co/">🤗 Huggingface</a> for their generous sponsorships to work on and open source cutting edge artificial intelligence research
- <a href="https://huggingface.co/">🤗 Huggingface</a> for the amazing accelerate library
- <a href="https://github.com/manmay-nakhashi">Manmay</a> for submitting the initial code for phoneme, pitch, duration, and speech prompt encoders as well as the multilingual phonemizer and phoneme aligner!
- <a href="https://github.com/manmay-nakhashi">Manmay</a> for wiring up the complete end-to-end conditioning of the diffusion network!
- You? If you are an aspiring ML / AI engineer or work in the TTS field and would like to contribute to open sourcing state-of-the-art, jump right in!
## Install
```bash
$ pip install naturalspeech2-pytorch
```
## Usage
```python
import torch
from naturalspeech2_pytorch import (
EncodecWrapper,
Model,
NaturalSpeech2
)
# use encodec as an example
codec = EncodecWrapper()
model = Model(
dim = 128,
depth = 6
)
# natural speech diffusion model
diffusion = NaturalSpeech2(
model = model,
codec = codec,
timesteps = 1000
).cuda()
# mock raw audio data
raw_audio = torch.randn(4, 327680).cuda()
loss = diffusion(raw_audio)
loss.backward()
# do the above in a loop for a lot of raw audio data...
# then you can sample from your generative model as so
generated_audio = diffusion.sample(length = 1024) # (1, 327680)
```
With conditioning
ex.
```python
import torch
from naturalspeech2_pytorch import (
EncodecWrapper,
Model,
NaturalSpeech2,
SpeechPromptEncoder
)
# use encodec as an example
codec = EncodecWrapper()
model = Model(
dim = 128,
depth = 6,
dim_prompt = 512,
cond_drop_prob = 0.25, # dropout prompt conditioning with this probability, for classifier free guidance
condition_on_prompt = True
)
# natural speech diffusion model
diffusion = NaturalSpeech2(
model = model,
codec = codec,
timesteps = 1000
)
# mock raw audio data
raw_audio = torch.randn(4, 327680)
prompt = torch.randn(4, 32768) # they randomly excised a range on the audio for the prompt during training, eventually will take care of this auto-magically
text = torch.randint(0, 100, (4, 100))
text_lens = torch.tensor([100, 50 , 80, 100])
# forwards and backwards
loss = diffusion(
audio = raw_audio,
text = text,
text_lens = text_lens,
prompt = prompt
)
loss.backward()
# after much training
generated_audio = diffusion.sample(
length = 1024,
text = text,
prompt = prompt
) # (1, 327680)
```
Or if you want a `Trainer` class to take care of the training and sampling loop, just simply do
```python
from naturalspeech2_pytorch import Trainer
trainer = Trainer(
diffusion_model = diffusion, # diffusion model + codec from above
folder = '/path/to/speech',
train_batch_size = 16,
gradient_accumulate_every = 2,
)
trainer.train()
```
## Todo
- [x] complete perceiver then cross attention conditioning on ddpm side
- [x] add classifier free guidance, even if not in paper
- [x] complete duration / pitch prediction during training - thanks to Manmay
- [x] make sExcerpt of 7,042 characters
Read on GitHub
Phil Wang · United States
88
manmay nakhashi · India
37
p0p
2
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:b1abfb7b5679ae76, topic:speech-synthesis, readme:text-to-speech, readme:speech synthesis
matched fp:b1abfb7b5679ae76, topic:deep-learning