Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Stream-Omni is a GPT-4o-like language-vision-speech chatbot that simultaneously supports interaction across various modality combinations.
| Date | Stars |
|---|---|
| 2026-07-24 | 389 |
| 2026-07-25 | 389 |
| 2026-07-28 | 389 |
| 2026-07-30 | 389 |
| 2026-08-06 | 389 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
[](https://arxiv.org/abs/2506.13642)
[](https://huggingface.co/ICTNLP/stream-omni-8b)
[](https://huggingface.co/datasets/ICTNLP/InstructOmni)
[](https://github.com/ictnlp/Stream-Omni)
> [**Shaolei Zhang**](https://zhangshaolei1998.github.io/), [**Shoutao Guo**](https://scholar.google.com.hk/citations?user=XwHtPyAAAAAJ), [**Qingkai Fang**](https://fangqingkai.github.io/), [**Yan Zhou**](https://zhouyan19.github.io/zhouyan/), [**Yang Feng**](https://people.ucas.edu.cn/~yangfeng?language=en)\*
Stream-Omni is a GPT-4o-like language-vision-speech chatbot that simultaneously supports interaction across various modality combinations, with the following features💡:
- **Omni Interaction**: Support multimodal inputs including text, vision, and speech, and generate both text and speech responses.
- **Seamless "see-while-hear" Experience**: Simultaneously output *intermediate textual results* (e.g., ASR transcriptions and model responses) during speech interactions, like the advanced voice service of GPT-4o.
- **Efficient Training**: Require only a small amount of omni-modal data for training.
<p align="center" width="100%">
<img src="./assets/stream-omni.png" alt="stream-omni" style="width: 90%; min-width: 300px; display: block; margin: auto;">
</p>
## 🖥 Demo
<p align="center" width="100%">
🎧 Vision-grounded Speech Interaction (simultaneously produce intermediate text) 🎧
</p>
https://github.com/user-attachments/assets/25807982-aa95-4633-9e92-10d995900258
https://github.com/user-attachments/assets/df8d79ba-63db-487c-a4a9-f183372168a1
> [!NOTE]
>
> **Stream-Omni can produce intermediate textual results (ASR transcription and text response) during speech interaction, offering users a seamless "see-while-hear" experience.**
- Downlaod Stream-Omni model from [here](https://huggingface.co/ICTNLP/stream-omni-8b), put in `${STREAMOMNI_CKPT}`.
- Downlaod CosyVoice (Tokenizer & Flow Model) from [here](https://modelscope.cn/models/iic/CosyVoice-300M-25Hz/files), put in `COSYVOICE_CKPT=./CosyVoice-300M-25Hz`:
```python
from modelscope import snapshot_download
snapshot_download('iic/CosyVoice-300M-25Hz', local_dir='./CosyVoice-300M-25Hz')
```
- Run these scripts to launch the API and interface, and then interact through the browser (http://localhost:7860):
```bash
# controller
python stream_omni/serve/controller.py --host 0.0.0.0 --port 10000
# CosyVoice worker
COSYVOICE_CKPT=path_to_CosyVoice-300M-25Hz # e.g., ./CosyVoice-300M-25Hz
WAV_DIR=path_to_save_generated_audio
CUDA_VISIBLE_DEVICES=0 PYTHONPATH=CosyVoice/third_party/Matcha-TTS python ./CosyVoice/cosyvoice_worker.py --port 21003 --model ${COSYVOICE_CKPT} --wav_dir ./gen_wavs/
# Stream-Omni worker, add --load-8bit for VRAM lower than 32GB
STREAMOMNI_CKPT=path_to_stream-omni-8b # e.g., ./stream-omni-8b
CUDA_VISIBLE_DEVICES=1 python ./stream_omni/serve/model_worker.py --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ${STREAMOMNI_CKPT} --model-name stream-omni
# Interface
python stream_omni/serve/gradio_web.py --controller http://localhost:10000 --model-list-mode reload --port 7860
```
- You can also refer to [`api.py`](./api.py) for the usage of API.
## 🔥 Quick Start
<p align="center" width="100%">
<img src="./assets/model.png" alt="model" style="width: 100%; min-width: 300px; display: block; margin: auto;">
</p>
> [!Tip]
>
> **Stream-Omni achieves modalityExcerpt of 8,498 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:b72035cf50179a8a, topic:speech-recognition, topic:tts, topic:asr
matched fp:b72035cf50179a8a, topic:large-language-models, topic:llm, topic:llama
matched fp:b72035cf50179a8a, topic:chatbot, topic:chatgpt, desc:chatbot