Fully Open Framework for Democratized Multimodal Training
🌟 The core work on this project is led by Glint Lab.
🤗 2.5 Models · Technical Report (coming soon) · Xiaohongshu Group
🤗 2.0 Models · Datasets · Technical Report · HomePage · Codec Playground · vLLM · Discord
🤗 1.5 Models · Datasets · Technical Report · NeMo · Discord
- 2026-11-18: Released LLaVA-OneVision-2.5-4B-Instruct — one 4B model for offline and streaming video, with its training code.
- 2026-11-05: Thanks to ResearchBoost-Agent for making LLaVA-OneVision-2-4B SFT 2.2–3.0× faster! See Training Acceleration for details.
- 2026-04-30: Released LLaVA-OneVision-2 — next-generation multimodal model, with new LLaVA-OneVision-2-VideoCaption and LLaVA-OneVision-2-Spatial datasets.
- 2026-02-10: Released OneVision-Encoder — codec-aligned vision encoders, with Technical Report.
- 2025-12-11: Released RL recipe for LLaVA-OneVision-1.5, with Project, Code, Data, and Model.
- 2025-09-30: Released the LLaVA-OneVision-1.5 Technical Report.
- Introduction
- Results
- Method
- Models
- Datasets
- Training Acceleration
- Citation
- Acknowledgement
- LLaVA-OneVision-1.5
LLaVA-OneVision-2.5 is the latest release of the family. LLaVA-OneVision-2.5-4B-Instruct is a 4B model for single-image, multi-image and video input that is also trained for streaming video: frames arrive one at a time, and at every step the model decides whether to stay silent or speak. The same weights and the same chat template serve both modes.
LLaVA-OneVision-2 is the next-generation release of the LLaVA-OneVision family — a fully open 8B multimodal model that unifies image, long-form video, and spatial understanding under a single architecture, with the entire pipeline (data, encoders, training, checkpoints, logs) released end-to-end.
Beyond uniform patchification. OneVision-Encoder and OneVision-Encoder-Lang are HEVC-style vision transformers that add a codec-stream input mode alongside image and uniform-frame video — selecting only motion- and residual-rich patches and sampling dense frames sparsely instead of sparse frames densely. The result is dramatically longer temporal coverage under the same token budget, where prior ViT backbones simply run out of context.
Most open multimodal models still live in a 2D, single-image world. LLaVA-OneVision-2.5-4B-Instruct breaks out of it — one model, native resolution, no task-specific adapters, no hidden tricks.
- Long video — multi-frame reasoning with efficient codec-aligned inference
- Streaming video — frames arrive one at a time, and the model decides at every step whether to stay silent or speak
- 3D-aware spatial reasoning — depth, layout, object relations
Four datasets ship with the LLaVA-OneVision family — two new for 2, two carried forward from 1.5:
- LLaVA-OneVision-2-VideoCaption — extremely dense video captions
- LLaVA-OneVision-2-Spatial — 3D-aware spatial reasoning
- LLaVA-OneVision-1.5-Mid-Training-85M — 85M concept-balanced mid-training corpus
- LLaVA-OneVision-1.5-Instruct — full instruction-tuning mixture
And unlike most "open" releases, everything ships alongside them: encoder weights, training code, configs, and full training logs. Reproducible end to end.
Compared with Qwen3.5-4B, thinking disabled; -- means not available. VideoMME is the mean over the short, medium and long splits. Each side reports its best score over the settings tried, from single runs; the setting behind each video score is listed in the model card. The image block shows selected benchmarks; full results, settings and protocol will be given in the technical report.
Standard video pipelines uniformly sample a handful of frames and process every patch — most of it static background. We borrow from HEVC: keep I-frames dense, keep only motion- and residual-rich patches from P-frames. Same 54-token budget, 18 frames instead of 6 — 3× the temporal range, no extra LLM context, no input-type adapters.
Most multimodal stacks ship a different tokenizer per input type — one path for images, another for video, a third for multi-image. We don't. Image, uniform frames, and codec-aligned tokens all flow into the same OneVision-Encoder under a shared (t, h, w) position scheme. No task-specific tokenizers, no per-modality routing.
We train LLaVA-OneVision-2 in four compact stages:
- Bootstrap video ability from LLaVA-OneVision-1.5 with short 30s video captions.
- Instruction tune with large multimodal instruction data and 30–180s video captions.
- Extend to long videos with 10–15 min captions and public video instruction data.
- Refine codec, spatial, and tracking skills with denser long-video sampling, point tracking, and 4M spatial samples.
The curriculum mixes LLaVA-OneVision-1.5 data, FineVision, and new in-house video caption/spatial datasets; we do not synthesize any video instruction data.
| Description | Link | Status |
|---|---|---|
| LLaVA-OneVision-2-VideoCaption | ||
| LLaVA-OneVision-2-Spatial | ||
| LLaVA-OneVision-1.5-Mid-Training-85M | ||
| LLaVA-OneVision-1.5-Instruct |
Training infrastructure optimizations by ResearchBoost-Agent make the 4B quick-start SFT 2.2–3.0× faster. Pick the preset that matches your GPUs:
| Preset | GPU Architecture | Peak memory per GPU | Speed-up |
|---|---|---|---|
blackwell |
Blackwell | 157.6 GiB | 3.0× |
hopper_80g |
Hopper | 74.8 GiB | 2.2× |
OV2_PRESET=blackwell bash examples/llava_onevision2/quick_start_4b/quick_start.shWithout a preset, training runs exactly as before. See docs/faster_training.md for how to set up the environment for accelerated training and for details on each speed-up switch.
Thanks so much to all of our amazing contributors!
If you find LLaVA-OneVision-2 useful in your research, please cite the following paper:
@misc{an2026llavaonevision2nextgenerationperceptualintelligence,
title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence},
author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
year = {2026},
eprint = {2605.25979},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2605.25979}
}
@misc{an2025llavaonevision15fullyopenframework,
title = {LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
author = {Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Didi Zhu and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng},
year = {2025},
eprint = {2509.23661},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2509.23661}
}
@misc{tang2026onevisionencodercodecalignedsparsityfoundational,
title = {OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence},
author = {Feilong Tang and Xiang An and Yunyao Yan and Yin Xie and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Chunyuan Li and Shikun Feng and Changrui Chen and Huajie Tan and Ming Hu and Manyuan Zhang and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
year = {2026},
eprint = {2602.08683},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2602.08683}
}
@article{li2025llavaonevisioneasyvisualtasktransfer,
title = {LLaVA-OneVision: Easy Visual Task Transfer},
author = {Bo Li and Yuanhan Zhang and Dong Guo and Renrui Zhang and Feng Li and Hao Zhang and Kaichen Zhang and Peiyuan Zhang and Yanwei Li and Ziwei Liu and Chunyuan Li},
journal = {Transactions on Machine Learning Research},
year = {2025},
url = {https://openreview.net/forum?id=zKv8qULV6n}
}We extend our sincere gratitude to the LoongForge team from Baidu AI Cloud for providing the exceptional AIAK-based training framework. The outstanding capabilities of AIAK-Training-LLM and AIAK-Megatron have significantly accelerated our training process with remarkable efficiency. We are especially grateful for their citation of LLaVA-OneVision-2 and deeply appreciate their recognition and support.
We acknowledge the support of Synvo AI for contributing to the partial data annotation in this work, and also thank the maintainers and contributors of the following open-source projects, whose work greatly inspired and supported our research:
- LLaVA: Large Language-and-Vision Assistant — LLaVA
- LLaVA-NeXT: Next-generation multi-modal assistant — LLaVA-NeXT
- lmms-eval: A standardized evaluation framework for Large Multimodal Models — lmms-eval
- Megatron-LM: Efficient, scalable training for large language models — Megatron-LM
- Qwen2.5-VL: Strong vision-language foundation model — Qwen2.5-VL
- InternVL: Open-source large-scale vision-language foundation model — InternVL
- Qwen3: Next-generation Qwen LLM — Qwen
- MetaCLIP: Scalable contrastive pretraining — MetaCLIP
- FineVision: Open Data Is All You Need — FineVision