Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
This is the code for the SpeechTokenizer presented in the SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models. Samples are presented on
| Date | Stars |
|---|---|
| 2026-07-31 | 658 |
| 2026-08-06 | 658 |
| 2026-08-14 | 662 |
| 2026-08-18 | 663 |
| 2026-08-24 | 662 |
| 2026-08-30 | 664 |
| 2026-09-18 | 665 |
| 2026-09-20 | 665 |
Today
— stars today
This week
+1 stars this week
This month
+2 stars this month
Momentum
0.0
growth rate 0.15%/day
# SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models
<a href='https://0nutation.github.io/SpeechTokenizer.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> <a href='https://arxiv.org/abs/2308.16692'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a>
## Introduction
This is the code for the SpeechTokenizer presented in the [SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models](https://arxiv.org/abs/2308.16692). SpeechTokenizer is a unified speech tokenizer for speech language models, which adopts the Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying semantic and acoustic tokens, SpeechTokenizer disentangles different aspects of speech information hierarchically across different RVQ layers. Specifically, the code indices that the first quantizer of RVQ outputs can be considered as semantic tokens and the output of the remaining quantizers mainly contain timbre info, which serve as supplements for the information lost by the first quantizer. We provide our models:
* A model operated at 16khz on monophonic speech trained on Librispeech with average representation across all HuBERT layers as semantic teacher.
* A model with [Snake activation](https://arxiv.org/abs/2306.06546) operated at 16khz on monophonic speech trained on Librispeech and Common Voice with average representation across all HuBERT layers as semantic teacher.
<br>
<p align="center">
<img src="images/overview.png" width="95%"> <br>
Overview
</p>
<p align="center">
<img src="images/speechtokenizer_framework.jpg" width="95%"> <br>
The SpeechTokenizer framework.
</p>
<br>
Welcome to try our [SLMTokBench](https://github.com/0nutation/SLMTokBench)
and we will also open source our [USLM](https://github.com/0nutation/USLM)!
## Qick Link
* [Relase](#release)
* [Samples](#samples)
* [Installation](#installation)
* [Model List](#model-list)
* [Usage](#usage)
* [Train SpeechTokenizer](#train-speechtokenizer)
* [Data Preprocess](#data-preprocess)
* [Train](#train)
* [Quick Start](#quick-start)
* [Citation](#citation)
* [License](#license)
## Release
- [2024/6/9] 🔥 We released the training code of SpeechTokenizer.
- [2024/3] 🔥 We released a checkpoint of SpeechTokenizer with [Snake activation](https://arxiv.org/abs/2306.06546) trained on LibriSpeech and Common Voice.
- [2023/9/11] 🔥 We released code of [soundstorm_speechtokenizer](https://github.com/ZhangXInFD/soundstorm-speechtokenizer).
- [2023/9/10] 🔥 We released code and checkpoints of [USLM](https://github.com/0nutation/USLM).
- [2023/9/1] 🔥 We released code and checkpoints of SpeechTokenizer. Checkout the [paper](https://arxiv.org/abs/2308.16692) and [demo](https://0nutation.github.io/SpeechTokenizer.github.io/).
## Samples
Samples are provided on [our demo page](https://0nutation.github.io/SpeechTokenizer.github.io/).
## Installation
SpeechTokenizer requires Python>=3.8, and a reasonly recent version of PyTorch.
To install SpeechTokenizer, you can run from this repository:
```bash
pip install -U speechtokenizer
# or you can clone the repo and install locally
git clone https://github.com/ZhangXInFD/SpeechTokenizer.git
cd SpeechTokenizer
pip install .
```
## Model List
| Model| Dataset |Discription|
|:----|:----:|:----|
|[speechtokenizer_hubert_avg](https://huggingface.co/fnlp/SpeechTokenizer/tree/main/speechtokenizer_hubert_avg)|LibriSpeech|Adopt average representation across all HuBERT layers as semantic teacher |
|[speechtokenizer_snake](https://huggingface.co/fnlp/AnyGPT-speech-modules/tree/main/speechtokenizer)|LibriSpeech + Common Voice|Snake activation, average representation across all HuBERT layers |
## Usage
### load model
```python
from speechtokenizer import SpeechTokenizer
config_path = '/path/config.json'
ckpt_path = '/path/SpeechTokenizer.pt'
model = SpeechTokenizer.load_from_checkpoint(config_path, ckpt_path)
model.eval()
```
### Extracting discrete representaExcerpt of 9,073 characters
Read on GitHub
25
14
Ikko Eltociear Ashimine · Japan
1
Karthik Ganesan · United States
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:bdd8452f38022693, desc:tokenizer