Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions
| Date | Stars |
|---|---|
| 2026-07-24 | 270 |
| 2026-07-25 | 270 |
| 2026-07-28 | 270 |
| 2026-07-30 | 270 |
| 2026-07-31 | 270 |
| 2026-08-06 | 270 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions
### Yinghao Aaron Li, Cong Han, Xilin Jiang, Nima Mesgarani
> Large-scale pre-trained language models have been shown to be helpful in improving the naturalness of text-to-speech (TTS) models by enabling them to produce more naturalistic prosodic patterns. However, these models are usually word-level or sup-phoneme-level and jointly trained with phonemes, making them inefficient for the downstream TTS task where only phonemes are needed. In this work, we propose a phoneme-level BERT (PL-BERT) with a pretext task of predicting the corresponding graphemes along with the regular masked phoneme predictions. Subjective evaluations show that our phoneme-level BERT encoder has significantly improved the mean opinion scores (MOS) of rated naturalness of synthesized speech compared with the state-of-the-art (SOTA) StyleTTS baseline on out-of-distribution (OOD) texts.
Paper: [https://arxiv.org/abs/2301.08810](https://arxiv.org/abs/2301.08810)
Audio samples: [https://pl-bert.github.io/](https://pl-bert.github.io/)
## Pre-requisites
1. Python >= 3.7
2. Clone this repository:
```bash
git clone https://github.com/yl4579/PL-BERT.git
cd PL-BERT
```
3. Create a new environment (recommended):
```bash
conda create --name BERT python=3.8
conda activate BERT
python -m ipykernel install --user --name BERT --display-name "BERT"
```
4. Install python requirements:
```bash
pip install pandas singleton-decorator datasets "transformers<4.33.3" accelerate nltk phonemizer sacremoses pebble
```
## Preprocessing
Please refer to the notebook [preprocess.ipynb](https://github.com/yl4579/PL-BERT/blob/main/preprocess.ipynb) for more details. The preprocessing is for English Wikipedia dataset only. I will make a new branch for Japanese if I have extra time to demostrate training on other languages. You may also refer to [#6](https://github.com/yl4579/PL-BERT/issues/6#issuecomment-1797869275) for preprocessing in other languages like Japanese.
## Trianing
Please run each cell in the notebook [train.ipynb](https://github.com/yl4579/PL-BERT/blob/main/train.ipynb). You will need to change the line
`config_path = "Configs/config.yml"` in cell 2 if you wish to use a different config file. The training code is in Jupyter notebook primarily because the initial epxeriment was conducted in Jupyter notebook, but you can easily make it a Python script if you want to.
## Finetuning
Here is an example of how to use it for StyleTTS finetuning. You can use it for other TTS models by replacing the text encoder with the pre-trained PL-BERT.
1. Modify line 683 in [models.py](https://github.com/yl4579/StyleTTS/blob/main/models.py#L683) with the following code to load BERT model in to StyleTTS:
```python
from transformers import AlbertConfig, AlbertModel
log_dir = "YOUR PL-BERT CHECKPOINT PATH"
config_path = os.path.join(log_dir, "config.yml")
plbert_config = yaml.safe_load(open(config_path))
albert_base_configuration = AlbertConfig(**plbert_config['model_params'])
bert = AlbertModel(albert_base_configuration)
files = os.listdir(log_dir)
ckpts = []
for f in os.listdir(log_dir):
if f.startswith("step_"): ckpts.append(f)
iters = [int(f.split('_')[-1].split('.')[0]) for f in ckpts if os.path.isfile(os.path.join(log_dir, f))]
iters = sorted(iters)[-1]
checkpoint = torch.load(log_dir + "/step_" + str(iters) + ".t7", map_location='cpu')
state_dict = checkpoint['net']
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
name = k[7:] # remove `module.`
if name.startswith('encoder.'):
name = name[8:] # remove `encoder.`
new_state_dict[name] = v
bert.load_state_dict(new_state_dict)
nets = Munch(bert=bert,
# linear projection to match the hidden size (BERT 768, StyleTTS 512)
bert_encoder=nn.Linear(plbert_config['model_params']['hidden_size'], args.hidden_dim),
predictor=predictor,
decoder=decoder,
Excerpt of 7,057 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ce5427d0d5ea62f3, topic:text-to-speech, topic:tts, desc:text-to-speech