Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
✨✨Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
| Date | Stars |
|---|---|
| 2026-07-24 | 388 |
| 2026-07-25 | 388 |
| 2026-07-28 | 388 |
| 2026-07-30 | 388 |
| 2026-07-31 | 388 |
| 2026-08-06 | 388 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
<p align="center">
<img src="./assets/logo.png" width="70%" height="70%">
</p>
<font size=7><div align='center' > [[🍎 Project(Demo) Page](https://freeze-omni.github.io)] [[📖 arXiv Paper](https://arxiv.org/abs/2411.00774)] [[🤗 Hugging Face](https://huggingface.co/VITA-MLLM/Freeze-Omni)] [[💬 WeChat (微信)](./assets/wechat_code.jpg)]</div></font></div></font>
---
## 🔥 News
* **`2024.11.26`** 🌟 The inference code, server demo and model weights **have been released**. Long wait!
* **`2024.11.4`** 🌟 We are very proud to launch Freeze-Omni, a speech-to-speech dialogue model
with both low-latency and high intelligence! We have submitted the open-source code, yet it is under review internally. We are moving the process forward as quickly as possible, stay tuned!
## Contents <!-- omit in toc -->
- [Freeze-Omni Overview](#-freeze-omni-overview)
- [Experimental Results](#-experimental-results)
- [Inference](#-inference)
- [Requirements and Installation](#requirements-and-installation)
- [Quick Start](#quick-start)
- [Real-Time Interactive Demo](#real-time-interactive-demo)
## 👀 Freeze-Omni Overview
Freeze-Omni is a speech-to-speech dialogue model, exhibiting the characteristic of being "**smart**" as it is constructed upon a "**frozen**" text-modality LLM. This enables it to keep the original intelligence of the LLM backbone, without being affected by the forgetting problem induced by the fine-tuning process for integration of the speech modality. Specifically, Freeze-Omni contains a speech encoder that supports streaming speech input and a speech decoder that generates streaming output speech. **Three key strategies** are adopted to implement the speech-to-speech dialogue system:
- **Chunk-wise Streaming Input**. Freeze-Omni has a speech encoder supporting chunk-wise streaming input speech features to obtain a fast response to input. A 3-stage training strategy can help it keep strong acoustic robustness.
- **AR-base Speech Output**. Freeze-Omni has an AR speech decoder based on a single codebook, which can achieve low-latency speech output in streaming. A prefix tuning method is used so that training on only a small amount of Q&A data can achieve the ability to produce high-quality speech synthesis.
- **Chunk-level State Prediction**. Freeze-Omni adds a classification layer after the last layer of the backbone LLM to predict different states. These states will determine whether or not the user interrupts the dialogue to achieve a duplex dialogue for the user and the bot.
<p align="center">
<img src="./assets/overview.png" width="88%" height="88%">
</p>
Besides we implement a Model as a Server strategy. We first started several models simultaneously and regarded them as a server. Then, when a user's VAD was triggered, the speech would be sent to the server in the form of chunks, and the server would be responsible for scheduling which idle model should respond to the current chunk. Since we separated all the kv-cache and CNN cache of the speech encoder and LLM during the inference process, the server only needs to save the inference cache for each user. In this way, any model in the server could respond to any chunk of any user, and there was no need to specify which model was used as a monitor or a generator.
## 📈 Experimental Results
- **Evaluation of speech understanding through ASR tasks, using CER(%) and WER(%).**.
<p align="center">
<img src="./assets/asr_res.png" width="70%" height="70%">
</p>
- **Evaluation of output speech quality on different top-k of AR decoder by using CER(%).**
<p align="center">
<img src="./assets/out_cer.png" width="50%" height="50%">
</p>
- **Evaluation of spoken question answering on accuracy(%).**
<p align="center">
<img src="./assets/qa.png" width="70%" height="70%">
</p>
- **Analysis of end-to-end latency for different parts.**
<p align="center">
<img src="./assExcerpt of 7,485 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:31f8024b8c633d9b, topic:speech-recognition, topic:speech-synthesis, readme:speech synthesis
matched fp:31f8024b8c633d9b, topic:large-language-models, readme:model weights