━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
MiMo Audio: Audio Language Models are Few-Shot Learners
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with only a few examples or simple instructions. GPT-3 has shown that scaling next-token prediction pretraining enables strong generalization capabilities in text, and we believe this paradigm is equally applicable to the audio domain. By scaling MiMo-Audio's pretraining data to over one hundred million of hours, we observe the emergence of few-shot learning capabilities across a diverse set of audio tasks. We develop a systematic evaluation of these capabilities and find that MiMo-Audio-7B-Base achieves SOTA performance on both speech intelligence and audio understanding benchmarks among open-source models. Beyond standard metrics, MiMo-Audio-7B-Base generalizes to tasks absent from its training data, such as voice conversion, style transfer, and speech editing. MiMo-Audio-7B-Base also demonstrates powerful speech continuation capabilities, capable of generating highly realistic talk shows, recitations, livestreaming and debates. At the post-training stage, we curate a diverse instruction-tuning corpus and introduce thinking mechanisms into both audio understanding and generation. MiMo-Audio-7B-Instruct achieves open-source SOTA on audio understanding benchmarks, spoken dialogue benchmarks and instruct-TTS evaluations, approaching or surpassing closed-source models.
MiMo-Audio-Tokenizer is a 1.2B-parameter Transformer operating at 25 Hz. It employs an eight-layer RVQ stack to generate 200 tokens per second. By jointly optimizing semantic and reconstruction objectives, we train MiMo-Audio-Tokenizer from scratch on a 10-million-hour corpus, achieving superior reconstruction quality and facilitating downstream language modeling.
MiMo-Audio couples a patch encoder, an LLM, and a patch decoder to improve modeling efficiency for high-rate sequences and bridge the length mismatch between speech and text. The patch encoder aggregates four consecutive time steps of RVQ tokens into a single patch, downsampling the sequence to a 6.25 Hz representation for the LLM. The patch decoder autoregressively generates the full 25 Hz RVQ token sequence via a delayed-generation scheme.
- 🎧 Try the Hugging Face demo: MiMo-Audio Demo
- 📰 Read the Official Blog: MiMo-Audio Blog
- 📄 Dive into the Technical Report: MiMo-Audio Technical Report
| Models | 🤗 Hugging Face |
|---|---|
| MiMo-Audio-Tokenizer | XiaomiMiMo/MiMo-Audio-Tokenizer |
| MiMo-Audio-7B-Base | XiaomiMiMo/MiMo-Audio-7B-Base |
| MiMo-Audio-7B-Instruct | XiaomiMiMo/MiMo-Audio-7B-Instruct |
pip install huggingface-hub
hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir ./models/MiMo-Audio-Tokenizer
hf download XiaomiMiMo/MiMo-Audio-7B-Base --local-dir ./models/MiMo-Audio-7B-Base
hf download XiaomiMiMo/MiMo-Audio-7B-Instruct --local-dir ./models/MiMo-Audio-7B-InstructSpin up the MiMo-Audio demo in minutes with the built-in Gradio app.
- Python 3.12
- CUDA >= 12.0
git clone https://github.com/XiaomiMiMo/MiMo-Audio.git
cd MiMo-Audio
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1Note
If the compilation of flash-attn takes too long, you can download the precompiled wheel and install it manually:
pip install /path/to/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whlpython run_mimo_audio.pyThis launches a local Gradio interface where you can try MiMo-Audio interactively.
Enter the local paths for MiMo-Audio-Tokenizer and MiMo-Audio-7B-Instruct, then enjoy the full functionality of MiMo-Audio!
We provide an example script to explore the in-context learning capabilities of MiMo-Audio-7B-Base.
See: inference_example_pretrain.py
To try the instruction-tuned model MiMo-Audio-7B-Instruct, use the corresponding inference script.
See: inference_example_sft.py
Full evaluation suite are available at 🌐MiMo-Audio-Eval.
This toolkit is designed to evaluate MiMo-Audio and other recent audio LLMs as mentioned in the paper. It provides a flexible and extensible framework, supporting a wide range of datasets, tasks, and models.
@misc{coreteam2025mimoaudio,
title={MiMo-Audio: Audio Language Models are Few-Shot Learners},
author={LLM-Core-Team Xiaomi},
year={2025},
url={https://github.com/XiaomiMiMo/MiMo-Audio},
}Please contact us at [email protected] or open an issue if you have any questions.
You are solely responsible for your use of XiaomiMiMo/MiMo-Audio and any outputs, actions, or consequences arising therefrom, and you agree not to use the XiaomiMiMo/MiMo-Audio or any derivatives thereof:
- For any unlawful, fraudulent, or malicious purpose, or in any manner that violates any applicable laws or regulations；
- To infringe upon the intellectual property rights, privacy rights, publicity rights, or other lawful rights or interests of any third party;
- To exploit, harm, harass, defame, unlawfully discriminate against, or otherwise adversely affect any individual or group, including minors or vulnerable persons;
- For any military purpose or application;
- To engage in or facilitate malicious cyber activities, including but not limited to malware, ransomware, phishing, credential theft, or unauthorized access, or any activity intended to compromise or unlawfully control computer systems, networks, or encryption systems;
- To access, collect, process, disclose, or disseminate personal data or other protected information without appropriate authorization or lawful basis; or to access, interact with, scrape, or automate any third-party platform or service in violation of any applicable laws, policies, or authorization requirements;
- To use the XiaomiMiMo/MiMo-Audio in a manner that autonomously executes high-risk actions without appropriate human oversight or authorization.
This notice is an informational reminder only and does not amend the applicable model licence or impose additional licence conditions. The permissions granted under that licence remain unaffected. Nothing in this notice restricts uses permitted by applicable law, including applicable exceptions and limitations.
[email protected]. For information on submitting a complaint, please visit the https://mimo.xiaomi.com.
