Awesome VLM Architectures is a citation-first visual catalog of 200 Vision-Language Model (VLM/MLLM) architectures, spanning contrastive encoders, multimodal LLMs, native multimodal models, unified understanding and generation, video, OCR, GUI agents, and embodied AI. Each entry links to primary sources and summarizes the model architecture, modality alignment or fusion, training stages, datasets, and distinctive design choices, with an architectural figure when available.
Use this repository to compare multimodal model families, trace architectural ideas over time, or retrieve grounded references for research and AI-agent workflows. The catalog includes a source-linked release timeline reviewed through October 27, 2026 and covers foundational systems such as CLIP and Flamingo alongside current multimodal reasoning and agentic models. Expand any model panel for its detailed architecture summary.
Last reviewed: October 27, 2026 (Europe/Istanbul).
October 2026 review: additions since the August 1 review, earlier omissions, family updates, date evidence, and availability caveats. This is a curated architecture catalog, not an exhaustive index of every fine-tune or API version.
If this repository is useful in your work, you may cite it below. Please also cite the original paper for claims about any individual model; this catalog is a guide to the literature, not a substitute for it.
Created and maintained by Gökay Aydoğan at fal.ai (ORCID; [email protected]).
Architecture images are credited individually in the figure credits and remain subject to the rights described in the figure notice.
📚 BibTeX
@misc{aydogan2024awesomevlmarchitectures,
author = {Gökay Aydoğan},
title = {Awesome VLM Architectures},
year = {2024},
howpublished = {\url{https://github.com/gokayfem/awesome-vlm-architectures}},
note = {GitHub repository, fal.ai},
url = {https://github.com/gokayfem/awesome-vlm-architectures}
}All architecture panels are ordered by release date, newest first. Models released on the same day retain editorial catalog order.
🧭 Chronological Model Index (200 architectures, newest first)
2026: Nemotron 3.5 Parse | Nemotron 3.5 Super VL | Muse Spark Open | Step 5 Preview | Liquid d1 | GroundAnything | Holo4 | MiMo-V2.6 | Qwen3.8-Omni-Flash | Taichu 5.0 | DeepSeek-V4.1-Flash | LLaDA-UI | Ling-3.0-flash-VL | Jina-OCR-v1 | NeoMME | Qwen-Drive-1.0 | GLM-5.3-Flash | Qwen3.8-Flash-Next | dots3-note-preview | North Micro Vision | Muse Glimmer | Shieldstral | MODUS | Argus-Unified | Kimi K3 | Mage-VL | Apertus 1.5 | Audio-Visual Flamingo | Inkling | Hy-Embodied-VLM | MonkeyOCRv2 | DiffusionGemma | InternVideo3 | Keye-VL 2.0 | Zamba2-VL | MiniMax M3 | Cosmos 3 | Fara1.5 | Lance | ZAYA1-VL | LensVLM | Falcon Perception | Granite Vision 4.1 | GLM-5V-Turbo | Nemotron 3 Nano Omni | SenseNova-U1 and U1.5 | PLaMo 2.1-VL | Qwen3.5-Omni | EXAONE 4.5 | BidirLM and BidirLM-Omni | Gemma 4 | Penguin-VL | Phi-4-Reasoning-Vision | V-SONAR and V-LCM | Qwen3.5 | Youtu-VL | Kimi K2.5 and K2.6 | LightOnOCR-2 and LightOnOCR-3 | Step3-VL-10B
2025: Molmo2 | Mistral Large 3 | Ministral 3 | ERNIE 5.0 | Emu3.5 | Nemotron Nano V2 VL | DeepSeek-OCR | OmniVinci | PaddleOCR-VL | Qwen3-Omni | Qwen3-VL | Moondream 3 | Intern-S1 | LFM2-VL and LFM2.5-VL | Step3 | GLM-4.1V-Thinking | ERNIE 4.5-VL | Show-o2 | MiMo-VL | BAGEL | Seed1.5-VL | PerceptionLM | Perception Encoder | InternVL3 and InternVL3.5 | Kimi-VL | Llama 4 Scout and Maverick | Qwen2.5-Omni | Gemma 3 | Aya Vision | Phi-4-multimodal | SigLIP 2 | EVEv2 | Qwen2.5-VL | VideoLLaMA 3 | UI-TARS | MiniMax-01 | MiniCPM-o-2.6 | Eagle 2 | Sa2VA
2024: VideoChat-Flash | OmniVLM | Apollo | DeepSeek-VL2 | Maya | InternVL 2.5 | PaliGemma 2 | ShowUI | SmolVLM | AIMv2 | LLaVA-CoT | LLM2CLIP | Tarsier2 | Janus and Janus-Pro | ARIA | Emu3 | Molmo and PixMo | Llama 3.2-Vision | NVLM | Pixtral 12B | VILA-U | Qwen2-VL | EAGLE | Show-o | Idefics3-8B | Transfusion | mPLUG-Owl3 | VITA | LLaVA-OneVision | VILA² | INF-LLaVA | SlowFast-LLaVA | EVLM | InternLM-XComposer-2.5 | OMG-LLaVA | Cambrian-1 | EVE | Ovis | Parrot | ConvLLaVA | Phi-3-Vision and Phi-3.5-Vision | CogVLM2 | Chameleon | PaliGemma | xGen-MM (BLIP-3) | MANTIS | Moondream-next | Idefics2 | InternLM-XComposer2-4KHD | MM1 | DeepSeek-VL | AnyGPT | SPHINX-X | LLaVA 1.6 | MiniCPM-V | MouSi | InternLM-XComposer2 | MoE-LLaVA | moondream1 and moondream2 | FireLLaVA | COSMO
2023: TinyGPT-V | MobileVLM | Alpha-CLIP | Nous-Hermes-2-Vision - Mistral 7B | SPHINX | Florence-2 | u-LLaVA | LLaVA-Plus | OtterHD | CoVLM | GLaMM | Fuyu-8B | PaLI-3 Vision Language Models | MiniGPT-v2 | BakLLaVA | Ferret | LLaVA 1.5 | CogVLM | MetaCLIP | Qwen-VL | IDEFICS | BLIVA | KOSMOS-2 | LaVIN | InstructBLIP | ImageBind | LLaVA | MiniGPT-4 | SigLIP | OpenFlamingo | PaLM-E | KOSMOS-1 | BLIP-2
2022: MULTIINSTRUCT | PaLI | Flamingo | BLIP
2020: ViT
Dates use the first documented official model release; when none is available, they use the paper's arXiv v1 submission or first technical report. Family point releases are folded into their first architecture release, and same-day entries retain catalog order.
🗓️ Release Timeline (200 architectures, newest first)
| Date | Architecture | Distinctive contribution |
|---|---|---|
| 2026-10-17 | Nemotron 3.5 Parse | Compact visual-prefix decoding for structured documents |
| 2026-10-15 | Nemotron 3.5 Super VL | Hybrid state-space MoE for long-context visual agents |
| 2026-10-15 | Muse Spark Open | Sparse multimodal experts with a shared perception encoder |
| 2026-10-15 | Step 5 Preview | Deep sparse experts with gated sliding-window and sparse GQA |
| 2026-10-07 | Liquid d1 | Single-pass multimodal decisions without text decoding |
| 2026-09-30 | GroundAnything | Autoregressive and block-diffusion spatial grounding |
| 2026-09-28 | Holo4 | Visual computer-use agents trained across GUI and tool environments |
| 2026-09-22 | MiMo-V2.6 | Hybrid sliding-window omni encoders and multi-teacher distillation |
| 2026-09-18 | Qwen3.8-Omni-Flash | Native omni agents with spatial audio and sparse long-context reasoning |
| 2026-09-15 | Taichu 5.0 | Entropy-gated recurrent latent visual reasoning |
| 2026-09-10 | DeepSeek-V4.1-Flash | Causal encoder-decoder multimodality with shared sparse KV memory |
| 2026-09-09 | LLaDA-UI | Block-diffusion reasoning and actions for GUI agents |
| 2026-09-04 | Ling-3.0-flash-VL | Kimi Delta Attention and gated latent attention for vision |
| 2026-09-02 | Jina-OCR-v1 | Compressed visual parsing with recursive multi-token drafting |
| 2026-08-31 | NeoMME | Single-tower bidirectional multimodal embeddings from masked diffusion |
| 2026-08-31 | Qwen-Drive-1.0 | Shared visual reasoning with external 3D perception and planning |
| 2026-08-26 | GLM-5.3-Flash | Native multimodal sparse and linear attention with mHC |
| 2026-08-26 | Qwen3.8-Flash-Next | Sparse hybrid attention and off-accelerator n-gram memory |
| 2026-08-14 | dots3-note-preview | Omnimodal MoE with a sparse vision expert encoder |
| 2026-08-12 | North Micro Vision | Compact native-resolution vision with multilayer feature injection |
| 2026-08-10 | Muse Glimmer | Dense visual agents with local-global attention and diffusion drafting |
| 2026-08-04 | Shieldstral | Policy-conditioned visual moderation in one forward pass |
| 2026-07-28 | MODUS | Shared decoder context with autoregressive and flow-matching modality experts |
| 2026-07-28 | Argus-Unified | Hybrid continuous and discrete visual tokens for economical understanding and generation |
| 2026-07-27 | Kimi K3 | Kimi Delta Attention, Attention Residuals, and extremely sparse LatentMoE routing |
| 2026-07-27 | Mage-VL | Codec-native selective video tokenization with a proactive event gate |
| 2026-07-24 | Apertus 1.5 | Early fusion of discrete visual and audio tokens |
| 2026-07-17 | Audio-Visual Flamingo | Temporally aligned audio-video reasoning over long clips |
| 2026-07-15 | Inkling | Relative-position million-context multimodal MoE trained from scratch |
| 2026-07-15 | Hy-Embodied-VLM | Action-centric sparse-MoE reasoning for physical-world agents |
| 2026-07-11 | MonkeyOCRv2 | Joint image-to-text and pixel-reconstruction pretraining for document vision |
| 2026-06-10 | DiffusionGemma | Cached causal context with a block-diffusion text decoder |
| 2026-06-10 | InternVideo3 | Token-preserving latent KV compression and closed-loop video reasoning |
| 2026-06-09 | Keye-VL 2.0 | DeepSeek Sparse Attention adapted to GQA-based long-video multimodality |
| 2026-06-02 | Zamba2-VL | Hybrid Mamba-2 and shared-attention blocks for efficient VLM inference |
| 2026-06-01 | MiniMax M3 | Native multimodality with block-sparse grouped-query attention at million-token context |
| 2026-05-31 | Cosmos 3 | Coupled autoregressive reasoner and diffusion generator for physical AI |
| 2026-05-21 | Fara1.5 | Verified synthetic environments for screenshot-based computer use |
| 2026-05-18 | Lance | Shared-sequence understanding, generation, and editing with modality experts |
| 2026-05-08 | ZAYA1-VL | Vision-conditional LoRA and compressed convolutional attention in an open-data MoE |
| 2026-05-07 | LensVLM | Selective expansion of visually compressed text context |
| 2026-05-03 | Falcon Perception | Early fusion with hybrid attention and continuous mask heads |
| 2026-04-29 | Granite Vision 4.1 | Window Q-Formers and dual DeepStack feature injection |
| 2026-04-29 | GLM-5V-Turbo | Perception integrated into reasoning, planning, tools, and execution |
| 2026-04-28 | Nemotron 3 Nano Omni | Hybrid MoE with dynamic visual resolution and native audio |
| 2026-04-27 | SenseNova-U1 and U1.5 | Native pixel-space unified understanding and generation |
| 2026-04-21 | PLaMo 2.1-VL | Compact Japanese VQA and grounding for edge deployment |
| 2026-04-17 | Qwen3.5-Omni | Hybrid-attention Thinker-Talker and adaptive speech alignment |
| 2026-04-09 | EXAONE 4.5 | Native multimodal pretraining with document-focused data and 256K context |
| 2026-04-02 | BidirLM and BidirLM-Omni | Converting causal decoders into bidirectional multimodal encoders |
| 2026-03-31 | Gemma 4 | Dense and MoE native multimodality, including an encoder-free 12B design |
| 2026-03-06 | Penguin-VL | Text-LLM-initialized vision encoder and priority-aware token compression |
| 2026-03-04 | Phi-4-Reasoning-Vision | Mid-fusion compact VLM with explicit reasoning and direct-answer modes |
| 2026-03-01 | V-SONAR and V-LCM | Vision-language alignment and prediction in multilingual concept space |
| 2026-02-16 | Qwen3.5 | Native early fusion with hybrid linear/full attention and sparse MoE variants |
| 2026-01-27 | Youtu-VL | Unified autoregressive visual tokens that emit dense vision outputs without task heads |
| 2026-01-27 | Kimi K2.5 and K2.6 | Trillion-parameter native multimodal MoE for agents and computer use |
| 2026-01-20 | LightOnOCR-2 and LightOnOCR-3 | Compact end-to-end OCR with document grounding |
| 2026-01-14 | Step3-VL-10B | Language-aligned perception encoder with 16-fold visual-token compression |
| 2025-12-11 | Molmo2 | Open video understanding with pointing and object tracking |
| 2025-12-02 | Mistral Large 3 | Large sparse multimodal experts with an integrated visual encoder |
| 2025-12-02 | Ministral 3 | Dense multimodal backbones with compute-efficient training |
| 2025-11-13 | ERNIE 5.0 | One autoregressive sparse MoE for text, images, video, audio, and generation |
| 2025-10-30 | Emu3.5 | Native multimodal next-token learning with diffusion adaptation |
| 2025-10-28 | Nemotron Nano V2 VL | Hybrid state-space visual reasoning with efficient video sampling |
| 2025-10-20 | DeepSeek-OCR | DeepEncoder compresses high-resolution documents into very short visual contexts |
| 2025-10-17 | OmniVinci | Joint audio-visual alignment with grouped temporal embeddings |
| 2025-10-16 | PaddleOCR-VL | NaViT-style dynamic resolution with a compact ERNIE decoder for document parsing |
| 2025-09-22 | Qwen3-Omni | Thinker-Talker MoE with streaming multicodebook speech |
| 2025-09-22 | Qwen3-VL | DeepStack multi-level ViT fusion and explicit video timestamp alignment |
| 2025-09-18 | Moondream 3 | Compact sparse visual experts with learned attention temperature |
| 2025-08-21 | Intern-S1 | Scientific multimodality with dynamic tokenization and expert reasoning |
| 2025-08-12 | LFM2-VL and LFM2.5-VL | Short-convolution language backbones with native-resolution vision |
| 2025-07-25 | Step3 | Model-system co-design for communication-efficient sparse-MoE multimodality |
| 2025-07-01 | GLM-4.1V-Thinking | Curriculum-sampled reinforcement learning for multimodal reasoning |
| 2025-06-30 | ERNIE 4.5-VL | Heterogeneous shared and modality-specific experts with isolated routing |
| 2025-06-19 | Show-o2 | Unified text prediction and continuous visual flow matching |
| 2025-06-04 | MiMo-VL | Four-stage multimodal pretraining followed by mixed on-policy RL |
| 2025-05-20 | BAGEL | Mixture-of-Transformer-Experts for understanding and generation |
| 2025-05-11 | Seed1.5-VL | Compact vision encoder with a 20B-active MoE for reasoning and agents |
| 2025-04-17 | PerceptionLM | Open data and training for detailed image-video understanding |
| 2025-04-17 | Perception Encoder | Contrastive visual pretraining with aligned language and spatial features |
| 2025-04-11 | InternVL3 and InternVL3.5 | Native multimodal pretraining, later extended with adaptive resolution and cascade RL |
| 2025-04-10 | Kimi-VL | MoonViT native-resolution packing with a sparse MoE decoder |
| 2025-04-05 | Llama 4 Scout and Maverick | Early-fusion native multimodality in sparse-MoE Scout and Maverick models |
| 2025-03-26 | Qwen2.5-Omni | Streaming Thinker-Talker architecture for multimodal input and speech output |
| 2025-03-12 | Gemma 3 | Efficient local/global attention with long-context image understanding |
| 2025-03-04 | Aya Vision | Cross-modal model merging for multilingual multimodality without language forgetting |
| 2025-03-03 | Phi-4-multimodal | Mixture-of-LoRAs for text, vision, and speech |
| 2025-02-20 | SigLIP 2 | Multilingual, localization-aware, native-aspect-ratio vision-language encoding |
| 2025-02-08 | EVEv2 | Improved Baselines for Encoder-Free Vision-Language Models |
| 2025-01-26 | Qwen2.5-VL | Enhanced Vision-Language Capabilities in the Qwen Series |
| 2025-01-21 | VideoLLaMA 3 | Frontier Multimodal Foundation Models for Image and Video Understanding |
| 2025-01-20 | UI-TARS | Pioneering Automated GUI Interaction with Native Agents |
| 2025-01-14 | MiniMax-01 | Scaling Foundation Models with Lightning Attention |
| 2025-01-12 | MiniCPM-o-2.6 | A GPT-4o Level MLLM for Vision, Speech and Multimodal Live Streaming |
| 2025-01-10 | Eagle 2 | Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models |
| 2025-01-07 | Sa2VA | Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos |
| 2024-12-31 | VideoChat-Flash | Hierarchical Compression for Long-Context Video Modeling |
| 2024-12-16 | OmniVLM | A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference |
| 2024-12-13 | Apollo | An Exploration of Video Understanding in Large Multimodal Models |
| 2024-12-13 | DeepSeek-VL2 | Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding |
| 2024-12-10 | Maya | An Instruction Finetuned Multilingual Multimodal Model |
| 2024-12-05 | InternVL 2.5 | Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling |
| 2024-12-04 | PaliGemma 2 | A Family of Versatile VLMs for Transfer |
| 2024-11-26 | ShowUI | UI-guided visual-token selection and interleaved action histories |
| 2024-11-26 | SmolVLM | A Small, Efficient, and Open-Source Vision-Language Model |
| 2024-11-21 | AIMv2 | Multimodal Autoregressive Pre-training of Large Vision Encoders |
| 2024-11-15 | LLaVA-CoT | Let Vision Language Models Reason Step-by-Step |
| 2024-11-06 | LLM2CLIP | Powerful Language Model Unlocks Richer Visual Representation |
| 2024-11-05 | Tarsier2 | Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding |
| 2024-10-17 | Janus and Janus-Pro | Decoupled visual encoders for understanding and generation with one transformer |
| 2024-10-08 | ARIA | An Open Multimodal Native Mixture-of-Experts Model |
| 2024-09-27 | Emu3 | One next-token objective over discrete text, image, and video tokens |
| 2024-09-25 | Molmo and PixMo | Open data pipeline with human captions and grounded pointing supervision |
| 2024-09-25 | Llama 3.2-Vision | Enhanced Multimodal Capabilities Built on Llama 3 |
| 2024-09-17 | NVLM | Open Frontier-Class Multimodal LLMs |
| 2024-09-11 | Pixtral 12B | A Cutting-Edge Open Multimodal Language Model |
| 2024-09-06 | VILA-U | Shared discrete visual tokens for autoregressive understanding and generation |
| 2024-08-29 | Qwen2-VL | A Powerful Open-Source Vision-Language Model for Image and Video Understanding |
| 2024-08-28 | EAGLE | Exploring The Design Space for Multimodal LLMs with Mixture of Encoders |
| 2024-08-22 | Show-o | Autoregressive language and discrete-diffusion image generation in one transformer |
| 2024-08-22 | Idefics3-8B | Building and Better Understanding Vision-Language Models |
| 2024-08-20 | Transfusion | Autoregressive text and continuous image diffusion in one transformer |
| 2024-08-09 | mPLUG-Owl3 | Hyper-attention for long image sequences and video |
| 2024-08-09 | VITA | Towards Open-Source Interactive Omni Multimodal LLM |
| 2024-08-05 | LLaVA-OneVision | Easy Visual Task Transfer |
| 2024-07-24 | VILA² | VILA Augmented VILA |
| 2024-07-23 | INF-LLaVA | High-Resolution Image Perception for Multimodal Large Language Models |
| 2024-07-22 | SlowFast-LLaVA | A Strong Training-Free Baseline for Video Large Language Models |
| 2024-07-19 | EVLM | An Efficient Vision-Language Model for Visual Understanding |
| 2024-07-03 | InternLM-XComposer-2.5 | A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output |
| 2024-06-27 | OMG-LLaVA | Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding |
| 2024-06-24 | Cambrian-1 | Spatial Vision Aggregator and systematic multi-encoder study |
| 2024-06-17 | EVE | Unveiling Encoder-Free Vision-Language Models |
| 2024-06-14 | Ovis | Learnable visual vocabulary for structural visual-text embedding alignment |
| 2024-06-04 | Parrot | Multilingual Visual Instruction Tuning |
| 2024-05-24 | ConvLLaVA | Hierarchical Backbones as Visual Encoder for Large Multimodal Models |
| 2024-05-21 | Phi-3-Vision and Phi-3.5-Vision | Compact dynamic-resolution VLM with 128K context |
| 2024-05-20 | CogVLM2 | Enhanced Vision-Language Models for Image and Video Understanding |
| 2024-05-16 | Chameleon | Mixed-modal early fusion over a shared token sequence |
| 2024-05-14 | PaliGemma | A Versatile and Transferable 3B Vision-Language Model |
| 2024-05-06 | xGen-MM (BLIP-3) | An Open-Source Framework for Building Powerful and Responsible Large Multimodal Models |
| 2024-05-02 | MANTIS | Mastering Multi-Image Understanding Through Interleaved Instruction Tuning |
| 2024-04-19 | Moondream-next | Compact Vision-Language Model with Enhanced Capabilities |
| 2024-04-15 | Idefics2 | Open 8B VLM with native-resolution inputs and strong OCR and document understanding |
| 2024-04-09 | InternLM-XComposer2-4KHD | A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD |
| 2024-03-14 | MM1 | Controlled study of encoders, connectors, token counts, and data mixtures |
| 2024-03-08 | DeepSeek-VL | Towards Real-World Vision-Language Understanding |
| 2024-02-19 | AnyGPT | Any-to-any autoregression over discrete text, image, speech, and music tokens |
| 2024-02-08 | SPHINX-X | Scaling Data and Parameters for a Family of Multi-modal Large Language Models |
| 2024-01-30 | LLaVA 1.6 | LLaVA-NeXT Improved reasoning, OCR, and world knowledge |
| 2024-01-30 | MiniCPM-V | A GPT-4V Level MLLM on Your Phone |
| 2024-01-30 | MouSi | Poly-Visual-Expert Vision-Language Models |
| 2024-01-29 | InternLM-XComposer2 | Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model |
| 2024-01-29 | MoE-LLaVA | Mixture of Experts for Large Vision-Language Models |
| 2024-01-20 | moondream1 and moondream2 | Compact SigLIP–Phi VLMs optimized for efficient edge inference |
| 2024-01-05 | FireLLaVA | LLaVA derivative trained rapidly on a curated multimodal instruction mixture |
| 2024-01-01 | COSMO | COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training |
| 2023-12-28 | TinyGPT-V | Efficient Multimodal Large Language Model via Small Backbones |
| 2023-12-28 | MobileVLM | A Fast, Strong and Open Vision Language Assistant for Mobile Devices |
| 2023-12-06 | Alpha-CLIP | A CLIP Model Focusing on Wherever You Want |
| 2023-11-28 | Nous-Hermes-2-Vision - Mistral 7B | SigLIP-equipped Mistral VLM with OCR and function-calling data |
| 2023-11-13 | SPHINX | The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models |
| 2023-11-10 | Florence-2 | A Deep Dive into its Unified Architecture and Multi-Task Capabilities |
| 2023-11-09 | u-LLaVA | Unifying Multi-Modal Tasks via Large Language Model |
| 2023-11-09 | LLaVA-Plus | Learning to Use Tools for Creating Multimodal Agents |
| 2023-11-07 | OtterHD | A High-Resolution Multi-modality Model |
| 2023-11-06 | CoVLM | Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding |
| 2023-11-06 | GLaMM | Pixel Grounding Large Multimodal Model |
| 2023-10-17 | Fuyu-8B | A Multimodal Architecture for AI Agents |
| 2023-10-13 | PaLI-3 Vision Language Models | Smaller, Faster, Stronger |
| 2023-10-13 | MiniGPT-v2 | large language model as a unified interface for vision-language multi-task learning |
| 2023-10-12 | BakLLaVA | Mistral-based LLaVA variant with a CLIP vision encoder and projection adapter |
| 2023-10-11 | Ferret | Refer and Ground Anything Anywhere at Any Granularity |
| 2023-10-05 | LLaVA 1.5 | Improved Baselines with Visual Instruction Tuning |
| 2023-10-05 | CogVLM | Visual Expert for Pretrained Language Models |
| 2023-09-28 | MetaCLIP | Demystifying CLIP Data |
| 2023-08-24 | Qwen-VL | A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond |
| 2023-08-22 | IDEFICS | Open Flamingo-style model for interleaved image-text generation |
| 2023-08-19 | BLIVA | A Simple Multimodal LLM for Better Handling of Text-rich Visual Questions |
| 2023-06-26 | KOSMOS-2 | Grounding Multimodal Large Language Models to the World |
| 2023-05-24 | LaVIN | Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models |
| 2023-05-11 | InstructBLIP | Towards General-purpose Vision-Language Models with Instruction Tuning |
| 2023-05-09 | ImageBind | One Embedding Space To Bind Them All |
| 2023-04-17 | LLaVA | Large Language and Vision Assistant - Visual Instruction Tuning |
| 2023-04-16 | MiniGPT-4 | Enhancing Vision-Language Understanding with Advanced Large Language Models |
| 2023-03-27 | SigLIP | Sigmoid Loss for Language Image Pre-Training |
| 2023-03-14 | OpenFlamingo | An Open-Source Framework for Training Large Autoregressive Vision-Language Models |
| 2023-03-06 | PaLM-E | An Embodied Multimodal Language Model |
| 2023-02-27 | KOSMOS-1 | Language Is Not All You Need: Aligning Perception with Language Models |
| 2023-01-30 | BLIP-2 | Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models |
| 2022-12-21 | MULTIINSTRUCT | Improving Multi-Modal Zero-Shot Learning via Instruction Tuning |
| 2022-09-14 | PaLI | A Jointly-Scaled Multilingual Language-Image Model |
| 2022-04-28 | Flamingo | a Visual Language Model for Few-Shot Learning |
| 2022-01-28 | BLIP | Bootstrapping Language-Image Pre-training |
| 2021-12-07 | GLIP | Grounded Language-Image Pre-training |
| 2021-06-25 | FROZEN | Multimodal Few-Shot Learning with Frozen Language Models |
| 2021-01-05 | CLIP | Contrastive Language-Image Pre-training |
| 2020-10-22 | ViT | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale |
A compact C-RADIO and mBART system generates document text, element classes, and bounding boxes with a causal visual prefix.
NVIDIA
Released: 2026-10-17
Architecture figure: The October 17 model card describes C-RADIO, convolutional compression, and mBART in text; no model-specific architecture diagram or paper was found in the cited release.
ℹ️ More Information
C-RADIOv4-SO400M processes rectangular NaViT inputs. Convolutional adapters compress its features before a ten-layer mBART causal decoder with grouped-query attention. The approximately 0.631B parameters include a one-step prediction head. Task tokens select text, layout classes, and box extraction.
Supervision pairs document images with OCR text and layout annotations. The released decoder supports ordinary generation; the accompanying serving plugin can use the bundled prediction head for speculative decoding. The card does not disclose a complete stage-by-stage training recipe.
Training combines rendered documents, scientific PDFs, tables, and synthetic word/character pages with automated and human labels. Outputs can be converted into Markdown, HTML, or structured records. Weights use OpenMDW 1.1; tokenizer and benchmark components have separately documented terms.
A Mamba-Transformer mixture-of-experts backbone combines a C-RADIO visual encoder with long-context image and video reasoning.
NVIDIA
Released: 2026-10-15
Figure 4. Visual-language post-training pipeline: teachers, text-trunk reinforcement learning, and multi-teacher on-policy distillation. Source paper, PDF p. 13. Figure notice.
ℹ️ More Information
The 120B/A12B model combines Mamba-2, attention, and LatentMoE blocks. A C-RADIO encoder, projection layers, and temporal video embedding connect visual inputs to the language backbone. Multi-token prediction supports speculative decoding; the advertised context is 1M tokens.
Training starts from Nemotron 3 Super, proceeds through continued pretraining and multimodal supervised tuning, then combines verifiable-reward and preference optimization with multi-teacher on-policy distillation. A final stage improves the prediction head.
Inputs are text, images, and video; output is text. The recipe discloses data mixtures and stages, but some supervised/RL datasets remain pending release. Weights use OpenMDW 1.1. Encoder-version labels differ across official artifacts, so the family name C-RADIO is used here.
A large sparse multimodal transformer integrates perceptual inputs with agentic language reasoning over a million-token context.
Meta Superintelligence Labs
Released: 2026-10-15
Architecture figure: The Muse Spark Open model card provides architecture specifications but no model-specific diagram. The cited evaluation report is not an architecture paper.
ℹ️ More Information
The 816B/42B-active model has 62 decoder layers, gated grouped-query attention, and 256 experts with eight selected per token. A 1.8B perception encoder combines learned absolute and 2D rotary positions with sparse/full attention. Supported inputs are text, images, video, and audio; output is text.
The downloadable version includes additional safety post-training relative to the hosted system. The card details the architecture and evaluations but does not publish a complete reproducible pretraining or alignment recipe.
Data sources include public web material, third-party collections, Meta services, and synthetic multimodal examples, with filtering and deduplication. BF16 and NVFP4 checkpoints are offered through gated access under the Muse Model License. October 15 is explicitly recorded as the release date.
A narrow, deep multimodal MoE interleaves gated local attention with sparse grouped-query attention for long agent trajectories.
StepFun
Released: 2026-10-15
Timeline basis: Dated primary architecture/training article; earlier API release day remains unresolved.
Official architecture diagram. Sparse GQA indexing and selected-token attention in Step 5 Preview. Primary source. Figure notice.
ℹ️ More Information
The 600B/27B-active model uses 92 transformer blocks and a 3:1 ratio of gated sliding-window to sparse full-attention layers. Sparse GQA adds blockwise token merging to reduce indexer cost. The public checkpoint supports text and image input with text output and up to 1M context.
The technical article describes precision and gradient-handling changes for deep-model stability. An agent-native synthesis system constructs tasks under expert-defined controls, followed by agentic reinforcement learning.
The release targets coding and knowledge-work trajectories. Full training data are not published. The card’s introduction also advertises video, while its input table lists text/image; this discrepancy is left explicit. October 15 is the dated architecture disclosure used here, not a claim about the earliest hosted API availability.
A decision head returns typed, calibrated answers in one forward pass instead of generating an autoregressive response.
Liquid AI
Released: 2026-10-07
Official architecture diagram. d1-3B visual decision model: a generative backbone adapted to single-pass decisions. Primary source. Figure notice.
Official architecture diagram. d1-omni-600M decision model with visual and audio encoders; accepts text with either image or audio input. Primary source. Figure notice.
ℹ️ More Information
d1-3B adapts LFM2.5-VL-3B to score yes/no questions, named options, and ratings. The experimental d1-omni-600M instead combines a bidirectional LFM encoder with SigLIP2 vision and FastConformer audio pathways. Its supported combinations are text-plus-image or text-plus-audio, not simultaneous image-plus-audio.
Decision-focused post-training teaches calibrated typed outputs from a shared state and multiple questions. The interface reads scores directly from decision positions, eliminating output-token decoding.
The release provides inference examples and evaluations across text, visual, and audio decisions. A complete training-data inventory is not published in the cards. Both checkpoints use the custom LFM license. These are discriminative decision models, distinct from the chat-generating LFM2.5-VL base.
A shared visual-language backbone supports separate autoregressive and parallel diffusion checkpoints for spatial predictions.
Qize Yu et al., GroundingPI and collaborators
Released: 2026-09-30
Timeline basis: First paper publication; code and weights followed in October.
Figure 2. A shared visual-language backbone supports separate autoregressive and parallel diffusion checkpoints for spatial predictions. Source paper, PDF p. 6. Figure notice.
ℹ️ More Information
MoonViT-V2/Kimi-K3 perception feeds Qwen3-4B through 2×2 spatial aggregation and a two-layer MLP. A vocabulary of 1,000 coordinate tokens represents spatial outputs. The diffusion variant adds a mask token and bidirectional response-block denoising; GroundAnything-VLM is the autoregressive checkpoint.
The training recipe develops grounded perception before converting the decoder to diffusion. Entropy-guided decoding refines spatial hypotheses in parallel. An optional self-speculative mode offers a separate speed/accuracy operating point.
Tasks span object localization, referring expressions, pointing, OCR, document layout, and GUI grounding. The paper appeared September 30, code October 3, and ModelScope weights October 14. Original contributions use Apache 2.0, while upstream Kimi-derived material retains its license conditions.
A family of visual agents combines screenshot-based interaction with code, API, and MCP tools through task-factory training.
H Company
Released: 2026-09-28
Official architecture diagram. Holo4 task-factory training pipeline. Primary source. Figure notice.
ℹ️ More Information
Holo4-27B builds on dense Qwen3.8, while Holo4-35B-A3B uses Qwen3.6 MoE. Both retain their base visual-language pathways and learn action policies spanning browser, desktop, and mobile environments. The contribution is the agent training system rather than a new shared vision tower.
A task factory creates environments and trajectories for supervised learning. Separate LoRA reinforcement-learning experts train on desktop/web tasks and terminal/MCP/API tasks, then merge with equal weighting. This connects GUI actions with structured tools within one agent.
The release documents task construction and provides evaluation trajectories; those traces are not the complete training corpus. Availability differs by checkpoint: 35B-A3B uses Apache 2.0, whereas 27B carries research/noncommercial terms. The family release date does not imply identical licenses.
Visual and audio encoders feed a hybrid sliding-window MoE, with specialist reinforcement learning consolidated through multi-teacher on-policy distillation.
Xiaomi MiMo
Released: 2026-09-22
Figure 2. Visual and audio encoders feed a hybrid sliding-window MoE, with specialist reinforcement learning consolidated through multi-teacher on-policy distillation. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
Flash has 309B total/15B active parameters and Pro 1.02T/42B. A 681M vision transformer uses local/global attention and spatial patch merging; audio tokenization and a patch encoder provide another input path. The language backbone alternates sliding-window and global attention and includes multi-token prediction.
Multimodal midtraining precedes large-scale agentic reinforcement learning. MOPD2 combines specialist teachers with rollout, teacher-prefix, and supervised-prefix training to consolidate capabilities and reduce failures such as repeated tool calls.
The report details coding and general-agent task construction, verifiers, and multimodal mixtures. Text, image, audio, and video inputs produce text with up to 1M context. RL and distilled checkpoints are separate releases under MIT; they should not be conflated in comparisons.
A sparse hybrid Thinker combines vision, general audio, and spatial audio encoders for long-context multimodal agent work.
Qwen Team
Released: 2026-09-18
Figure 2. AuT audio encoder and its local attention layout; this is a component diagram, not the full multimodal system. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
The backbone inherits Qwen3.8-Next’s Gated DeltaNet and Qwen Sparse Attention. Alongside the vision encoder, AuT encodes general audio while Spatial AuT processes multichannel spatial cues. Explicit timestamps preserve cross-modal correspondence; post-training extends native 256K context to 1M.
Early mixed-modality co-training is followed by long-context reasoning and interactive agent training, then multi-teacher distillation. The report includes realtime text/speech interaction and selective retrieval of audio-video segments.
Training spans pure text and paired/interleaved audio, image, and video data. Qwen-MM-Plugins and Qwen-Live-Harness are released tooling; they are not downloadable model weights. The report documents hosted-model architecture and agent integration.
A compact visual reasoner adds adaptive recurrent latent computation to a Qwen-based model with C-RADIO perception.
Taichu AI
Released: 2026-09-15
Timeline basis: Explicit Initial release commit.
Architecture figure: The Taichu 5.0 project page and model card describe adaptive latent recurrence but provide examples and benchmarks rather than a published architecture diagram.
ℹ️ More Information
ZDTaichu5.0-9B combines Qwen3.5-9B with C-RADIOv4-H and adaptive image resolution. Entropy-gated adaptive recurrence allocates additional latent reasoning computation when needed. The supported context is 128K.
The release targets general visual understanding alongside spatial, multi-view, and embodied reasoning. Public architecture descriptions explain adaptive recurrence, but do not disclose a complete training corpus or sufficiently detailed stage schedule for independent reproduction.
Inputs include text and visual observations; outputs are textual answers and reasoning. Official code and a model checkpoint are linked. The timeline follows the explicitly titled Initial release commit on September 15, not the earlier model-repository creation timestamp.
A causal encoder-decoder MoE shares global KV memory across decoder layers, combining native vision with compressed sparse attention.
DeepSeek-AI
Released: 2026-09-10
Figure 3. A causal encoder-decoder MoE shares global KV memory across decoder layers, combining native vision with compressed sparse attention. Source paper, PDF p. 7. Figure notice.
ℹ️ More Information
The 40-layer backbone splits into 20 causal encoder and 20 decoder layers. Decoder layers reuse memory derived from the final encoder states. CSA2 combines full, reindex, and reuse modes with hierarchical sparse selection. DeepSeekViT uses 2D rotary positions, 3×3 patch unshuffle, and an MLP connector. The 552B backbone has a separate 196B Engram store; active computation differs between prefill and decoding.
Early multimodal pretraining spans 45T tokens, followed by instruction tuning, reinforcement learning, and on-policy distillation. Context training extends to 1M tokens; sparse attention and low-precision caching control memory costs.
Text and image inputs produce text. The report describes corpus categories and training stages rather than releasing the complete training corpus. Weights use the MIT license.
Flash-1017 is a subsequent checkpoint update in the same architecture. Its official card describes reasoning, visual, and agentic improvements and links the same technical report; it is not counted as another architecture. Updated checkpoint
A native-resolution visual encoder feeds a block-diffusion MoE that jointly predicts GUI reasoning and action tokens.
inclusionAI
Released: 2026-09-09
Timeline basis: First arXiv submission.
Figure 2. A native-resolution visual encoder feeds a block-diffusion MoE that jointly predicts GUI reasoning and action tokens. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
The approximately 16.7B system adapts LLaDA2.0-mini to screenshots with a vision encoder and multimodal connector. Bidirectional denoising operates within response blocks while earlier context remains available. Outputs combine reasoning with executable actions and normalized coordinates.
Training first aligns vision and language, then applies GUI instruction tuning across web, desktop, and mobile environments. The task-generation pipeline decomposes and composes capabilities, executes candidate trajectories, and verifies outcomes.
The report describes multimodal alignment data and synthesized GUI trajectories. Public model and inference code are available. Reported parallel-decoding gains depend on block size and decoding settings; this is a diffusion agent, not an autoregressive model with only a speculative head.
A visual MoE alternates Kimi Delta Attention with gated multi-head latent attention for efficient image and video reasoning.
inclusionAI
Released: 2026-09-04
Timeline basis: First model/configuration upload.
Architecture figure: The official Ling-3.0-flash-VL card provides architecture specifications but no architecture diagram. Its training-content disclosure is not an architecture paper.
ℹ️ More Information
The card specifies 124B total and 5.5B active parameters across 42 layers, using a 5:1 KDA-to-gated-MLA ratio. A vision transformer and two-layer MLP connect visual tokens to the decoder; VideoRoPE carries spatial-temporal position information.
The released instruction model supports reasoning and tool workflows with up to 256K context. The model card supplies architecture and serving details, while a separate training-content disclosure summarizes data provenance. It does not provide a complete stage-by-stage reproducible recipe.
Text, images, and video produce text. Weights use MIT. The timeline records the first model/configuration upload on September 4, rather than interpreting a later README edit as a new architecture release.
A DeepSeek-OCR-derived parser adds shared recursive speculative decoding and dense verifiable rewards for document structure.
Alejandro Barón García et al., Jina AI
Released: 2026-09-02
Timeline basis: First arXiv submission.
Figure 2. A DeepSeek-OCR-derived parser adds shared recursive speculative decoding and dense verifiable rewards for document structure. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
DeepEncoder provides a global compressed view and local tiles to a 3B MoE decoder with roughly 570M active parameters. FastMTP reuses one draft block over three prediction steps. Greedy verification preserves the verifier’s greedy output rather than accepting approximate drafts.
Post-training combines instruction alignment, difficult-document fine-tuning, and GRPO. Formula, table, and structure checks award partial credit through dense deterministic rewards. Iterative error analysis guides subsequent data and checkpoint refinement.
Training mixes cleaned public document corpora with targeted synthetic pages. The model outputs structured document text and is distributed under CC BY-NC 4.0. Performance and speed depend on resolution and decoding settings; the paper reports the corresponding operating points.
A single bidirectional transformer learns joint text-patch embeddings without a separate vision tower or causal language decoder.
Aurélien Lac and Tony Wu, H Company
Released: 2026-08-31
Timeline basis: First arXiv submission, despite the September identifier.
Figure 1. A single bidirectional transformer learns joint text-patch embeddings without a separate vision tower or causal language decoder. Source paper, PDF p. 1. Figure notice.
ℹ️ More Information
NeoMME-260M and 800M ingest raw image patches and text tokens in one encoder. Dynamic-resolution processing and alternating local/global attention support long multimodal sequences. Retrieval can use either pooled dense vectors or token-level late interaction.
Pretraining reconstructs masked discrete text while conditioning on visible image patches; it does not add a pixel-reconstruction loss. Subsequent contrastive retrieval training supports both dense and late-interaction representations.
The report covers web, code, mathematics, documents, image captions, and synthetic OCR data. Compression experiments combine pooling and quantization; their ratios are configuration-specific. Apache-licensed weights target retrieval and representation learning rather than autoregressive chat.
A shared VLM supports visual questions, explicit 3D perception, and flow-matching trajectory planning through external heads.
Xin Zhou et al., Qwen Team
Released: 2026-08-31
Timeline basis: First arXiv submission.
Figure 2. A shared VLM supports visual questions, explicit 3D perception, and flow-matching trajectory planning through external heads. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
The Qwen3.5-4B vision-language architecture remains unchanged. An external bird’s-eye-view head predicts objects, occupancy, and maps from shared representations. A separate Planning Expert uses flow matching to generate future ego trajectories; the VLM continues answering free-form visual questions.
A staged recipe initializes and trains perception, adapts the multimodal pathway, and develops planning. Joint driving/general-vision supervision reduces forgetting. Separate supervised and reward-optimized planners are released.
Training unifies labels and trajectories across driving datasets, with consistency filtering and general vision-language data. The RL planner uses NAVSIM PDMS, WOD-E2E RFS, and displacement rewards. Public code and Apache-licensed weights accompany the first paper dated August 31.
A newly pretrained multimodal MoE combines sparse and linear attention with manifold-constrained residual connections.
Z.ai
Released: 2026-08-26
Timeline basis: First substantive model-card and weight commits on August 26; repository initialized August 25.
Architecture figure: The cited GLM-5.3-Flash card publishes architecture text and benchmark plots, but no model-specific architecture figure. Its linked GLM-5 paper predates this architecture.
ℹ️ More Information
The official card specifies 320B total and 18B active parameters, hybrid sparse/linear attention, and manifold-constrained hyper-connections (mHC). This is a new multimodal base, rather than simply a post-training update of GLM-5.2.
The model uses a 30T-token multimodal pretraining corpus and targets coding, visual reasoning, and long-running tool workflows. The card does not disclose a complete visual-encoder specification or a reproducible post-training mixture.
The public checkpoint accepts visual and textual inputs and generates text under the MIT license. The linked February GLM-5 paper describes an earlier model: its architecture and figure should not be presented as a GLM-5.3-Flash-specific report.
Gated DeltaNet and micro-block sparse attention combine with n-gram memory and widened residual streams for efficient multimodal inference.
Qwen Team
Released: 2026-08-26
Figure 1. Gated DeltaNet and micro-block sparse attention combine with n-gram memory and widened residual streams for efficient multimodal inference. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
The model has 125B backbone parameters with 6B active, plus separate 51B n-gram embedding tables and a prediction module. Three Gated DeltaNet layers alternate with one Qwen Sparse Attention layer. Gated residual read/write operations and a causal vision encoder complement sparse MoE routing.
The technical report studies attention, residual, and optimization choices, including Muon/AdamW training. Native context is 262,144 tokens, with extension to roughly 1M; these are distinct operating settings.
Text, image, and video inputs produce text. Public weights and inference integrations are available under Qwen Community License 1.0. The paper provides architectural ablations, not a release of the entire pretraining dataset.
A sparse vision encoder and audio pathway feed a hybrid sparse-attention and sliding-window MoE language model.
dots.studio
Released: 2026-08-14
Timeline basis: Explicit preview-release commit dated August 14.
Architecture figure: The official dots3 preview card and technical blog provide configuration details but no architecture diagram; the full technical report is still pending.
ℹ️ More Information
The preview specifies a 280B/16B-active language backbone with 46 layers, combining 13 sparse-attention and 33 sliding-window layers. Its vision MoE contains about 7B parameters with 1.2B active, alongside an 800M audio encoder. A multi-token prediction module supports efficient decoding.
The official technical blog and card describe architectural configuration and deployment. A complete training report is still pending, so precise stage sizes, alignment objectives, and post-training datasets are not inferred from other models.
Text, images, video, and audio are accepted within a 512K context; output is text. Apache 2.0 weights are available as a preview. The August 14 release commit, rather than the earlier repository creation timestamp, anchors the timeline.
A compact vision-language model injects multiple visual feature levels into a hybrid local/global language decoder.
Cohere Labs
Released: 2026-08-12
Official architecture diagram. Native-resolution vision encoder, multilayer projection, and local/global language decoder. Primary source. Figure notice.
ℹ️ More Information
A custom 400M vision encoder and in-house 2B language model form the approximately 2.4B system. Vision uses native resolution, 2D rotary positions, and learned 1D embeddings. DeepStack-style projectors inject intermediate visual features into early decoder layers. The decoder repeats three sliding-window layers with rotary positions and one global layer without positional embeddings.
The instruction checkpoint targets OCR, documents, charts, visual reasoning, and grounding. The public card explains the architecture and usage but does not provide a complete reproducible training recipe or corpus inventory.
Text and image inputs produce text or normalized grounding coordinates. Although the underlying language model supports 128K context, the released multimodal model is validated at 8K. Weights are Apache 2.0.
A compact dense agent combines a Perception Encoder with local-global attention and a separate block-diffusion speculative drafter.
Meta Superintelligence Lab
Released: 2026-08-10
Timeline basis: Model publication commit; card specifies August 2026 only.
Table 1. Architecture specifications from the paper; no system schematic is published. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
The roughly 29.6B model includes a 1.8B ViT-G/14 perception encoder. Its 52-layer language stack repeats three local layers and one global layer, with gated grouped-query attention and rotary positions confined to local layers. An optional five-layer DFlash drafter proposes sixteen-token blocks for verification.
Distillation from Muse Spark and agent-focused post-training target tool use, multi-step tasks, and recovery from failed actions. Quantized checkpoints and drafter variants support local inference.
The card lists public, third-party, and Meta-service multimodal data across more than 100 languages, without releasing the full mixture. Inputs are text and images; outputs are text. Weights use Apache 2.0. The card dates the release only to August; August 10 is the model publication commit used here.
A small visual-language model scores compliance with a supplied natural-language policy without lengthy response generation.
Mistral AI
Released: 2026-08-04
Figure 1. A small visual-language model scores compliance with a supplied natural-language policy without lengthy response generation. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
Shieldstral-1.0-3B adapts Ministral 3 with Pixtral visual perception. A policy and text/image content condition a binary decision and confidence score. Classification uses one forward pass, separating moderation decisions from general assistant generation.
The report studies policy-conditioned training and evaluation across varied moderation policies. The released model was trained at 32K context; the larger architectural context limit is not equivalent to validated training coverage.
The public checkpoint supports twelve languages and both textual and visual inputs under Apache 2.0. The August 4 announcement is the release date; the preceding technical report documents the method. Dataset composition and evaluation protocols are described in the report rather than inferred from the base model.
MODUS shares a decoder context across modalities, with discrete autoregressive and continuous flow-matching experts enabling chained generation and cross-modal self-verification.
Mingqiao Ye et al., EPFL
Released: 2026-07-28
Figure 2. Decoder-only any-to-any modeling across tokenized 1D and 2D modalities. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
MODUS builds on BAGEL's mixture-of-transformer design, sharing a causal context across modalities while retaining two experts. Discrete 1D representations use autoregressive next-token prediction; continuous 2D representations use latent flow matching. Attention is causal within 1D sequences, bidirectional within 2D modalities, and causal between modalities. Vision uses both semantic ViT features and VAE latents.
The experts are trained with different objectives: cross-entropy for discrete tokens and mean-squared error for the continuous velocity field. Thus, the paper's symmetric modality formulation should not be read as eliminating separate experts or loss functions. The training mixture connects text, RGB, geometry, segmentation, edges, and visual feature representations, allowing intermediate-modality chains and self-verification within one checkpoint.
Argus-Unified combines continuous tokens for understanding with learned discrete tokens for generation, reusing a frozen unified visual encoder and pretrained VLM to lower the cost of unified modeling.
Weiming Zhuang et al.
Released: 2026-07-28
Figure 3. Two-stage hybrid-token training for unified image understanding and generation. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Argus-Unified resolves the conflicting visual representations required by comprehension and synthesis through hybrid visual tokens. Continuous encoder features preserve semantic information for understanding, while a learned quantizer produces discrete tokens for image generation from the same frozen visual encoder.
Training first learns the quantizer and image decoder, then initializes the language component from a pretrained VLM and trains unified multimodal prediction. The reported recipe uses 15.6 million examples and roughly $2,000 of compute, positioning the model as a reproducible baseline for understanding-and-generation research rather than a scale demonstration.
Kimi K3 is a native multimodal sparse MoE that combines Kimi Delta Attention, gated latent attention, Attention Residuals, and 896 routed experts for million-token reasoning and agency.
Kimi Team, Moonshot AI
Released: 2026-07-27
Figure 2. Kimi Delta Attention, Stable LatentMoE, Attention Residuals, and MoonViT-V2. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Kimi K3 has 2.8T total and 104B active parameters across 93 layers. Sixty-nine layers use gated delta-rule linear attention and 24 use gated Multi-head Latent Attention; learned Attention Residuals create structured information paths across depth. Stable LatentMoE activates 16 of 896 routed experts plus two shared experts, making the model unusually sparse for its scale.
A 401M-parameter MoonViT-V2 encoder supplies native image and video input. The model supports a 1,048,576-token context and receives multimodal, long-context, agentic, reasoning, and tool-use training. Quantization-aware pretraining targets MXFP4 weights and MXFP8 activations rather than treating low-precision deployment as an afterthought.
Mage-VL uses video-codec signals to encode only dynamic, information-rich regions and pairs a lightweight event gate with a causal decoder for proactive streaming perception.
Senqiao Yang et al., Microsoft Research
Released: 2026-07-27
Figure 3. Codec-native streaming perception with an event gate and causal language decoder. Source paper, PDF p. 8. Figure notice.
ℹ️ More Information
Mage-ViT replaces uniform frame sampling with selective 16×16-patch encoding guided by motion vectors and residual energy across anchor and predicted frames. This codec-native tokenization reduces visual-token use by more than 75% while retaining spatiotemporal context. The encoder is trained from scratch on roughly 560M unlabeled images and 100M unlabeled video frames.
The complete VLM uses a dual-system design: a small event gate decides when a stream warrants deeper processing, and a causal decoder performs contextual reasoning. This turns streaming perception into an event-driven process and yields up to a reported 3.5× wall-clock speedup while retaining static-image and spatial reasoning ability.
An unchanged language decoder learns image and audio understanding through early fusion of discrete modality tokens.
Swiss AI Initiative
Released: 2026-07-24
Timeline basis: July 24 weight release explicitly recorded in October 15 release notes.
Official architecture diagram. Apertus 1.5 continued-pretraining and post-training stages. Primary source. Figure notice.
ℹ️ More Information
Apertus 1.5 extends 8B and 70B decoders by expanding their vocabulary with Emu3.5 visual tokens and WavTokenizer audio tokens. A 512-pixel image maps to 1,024 discrete tokens; audio uses a 40Hz token stream. No separate continuous vision tower is attached to the decoder.
Continued pretraining mixes text with unpaired and paired multimodal data, followed by cooldown and long-context stages. Supervised tuning and preference/reward-based alignment develop instruction following and tool behavior.
The release notes document 4T/2T continued-pretraining budgets for the respective scales and link data-transparency records. Context extends to 256K. Inputs are text, images, and audio; output is text. Apache-licensed weights were released July 24; October 15 is the release-notes publication date.
Interleaved audio and visual features with constrained rotary time embeddings support grounded reasoning over long videos.
Sreyan Ghosh et al., NVIDIA and collaborators
Released: 2026-07-17
Timeline basis: First arXiv submission; June 29 card label is a checkpoint date.
Figure 2. Paper architecture, including an optional speech-generation path; the released checkpoint produces text only. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
SigLIP with Dynamic-S2 processing and an AF-Whisper audio encoder connect through separate MLP projectors to Qwen2.5-7B. Time-aligned chunks are interleaved, and Constrained Rotary Time Embeddings encode their correspondence. The released checkpoint generates text, not speech.
A three-stage curriculum progresses from short clips to long-video supervised training and then chain-of-thought tuning with GRPO. Temporal interleaved reasoning explicitly references evidence timestamps.
Audio-Visual-Skills contains about 7M caption/QA instances; AVF-Instruct and AVF-Think support instruction and reasoning stages. The final stage supports roughly fifteen-minute clips at 32K tokens. Weights use NVIDIA’s noncommercial research terms. The timeline uses the paper date, not the June 29 internal checkpoint label.
Inkling is an open-weight multimodal MoE trained from scratch on text, images, audio, and video, using relative positions and alternating local/global attention for million-token contexts.
Thinking Machines Lab
Released: 2026-07-15
Architecture figure: The official Inkling model card contains no architecture figure.
ℹ️ More Information
Inkling contains 975B total and 41B active parameters. Its MoE layers activate six of 256 routed experts plus two shared experts using sigmoid routing and auxiliary-loss-free balancing. Attention alternates five local sliding-window layers with one global layer and uses learned relative-position representations rather than RoPE; short convolutions further refine key, value, attention, and MLP pathways.
Pretraining spans 45T text, image, audio, and video tokens. Post-training begins with a small synthetic supervised bootstrap and scales asynchronous reinforcement learning across multimodal understanding, tools, coding, mathematics, conversation, and safety. The released weights support controllable reasoning effort and contexts up to one million tokens.
Hy-Embodied-VLM-1.0 joins a native-aspect-ratio vision encoder to a compact sparse MoE and trains an action-centric reasoning hierarchy for perception, planning, reflection, and recovery.
Tencent Robotics X, Hy Vision Team and Futian Laboratory
Released: 2026-07-15
Figure 4. Self-evolving supervised fine-tuning, rejection sampling, and specialized reinforcement learning. Source paper, PDF p. 11. Figure notice.
ℹ️ More Information
Hy-Embodied-VLM-1.0 connects Hy-ViT2 to a roughly 30B-total, 3B-active language model. Each sparse-MoE layer selects eight of 128 routed experts plus one shared expert. It accepts as many as 128 images in a 32K context and exposes direct-response and explicit-thinking modes through the same checkpoint.
Its curriculum moves from action-relevant state understanding to transition reasoning and then sequential, adaptive reasoning. A self-evolving post-training loop alternates reinforcement learning with rejection-sampling fine-tuning, separately optimizing geometric precision and higher-level planning before policy fusion.
MonkeyOCRv2 learns document vision jointly through image-to-text generation and pixel reconstruction, preserving both semantic content and character-level strokes across 17 languages.
Yuliang Liu et al.
Released: 2026-07-11
Figure 1. Document-native pretraining through text generation and pixel reconstruction. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
MonkeyOCRv2 is a visual-text foundation encoder specialized for the properties that natural-image pretraining often misses: dense text, small glyphs, fine strokes, and layout. Its two complementary objectives align images with their transcribed content while forcing the encoder to retain pixel-level document structure.
The MonkeyDoc v2 corpus contains 113M document images in 17 languages. The frozen encoder improves recognition, detection, tamper analysis, segmentation, parsing, and understanding; paired with a lightweight language model, it forms a 0.7B document parser. Public weights were announced July 11, two days before the arXiv report, so the timeline follows the model release rather than the paper date.
A causal context encoder supplies cached memory to a bidirectional diffusion decoder that generates text in blocks.
Google DeepMind
Released: 2026-06-10
Figure 4. Context encoding, iterative canvas denoising, and cached context extension. Source paper, PDF p. 7. Figure notice.
ℹ️ More Information
The 26B-A4B Gemma-derived MoE uses a causal encoder and cross-attending diffusion decoder. Each 256-token canvas is iteratively denoised, then appended to cached context before the next block. This preserves long-context reuse while parallelizing generation within a canvas.
The two-stage conversion uses supervised denoising fine-tuning, followed by reinforcement learning and sampler distillation. Uniform discrete diffusion permits revising earlier choices within a canvas; completed canvases remain fixed. The conversion consumes less than one-tenth of the base model’s training-token budget.
The model accepts text, images, and video frames and produces text, with a 256K context window. Training data includes web text, code, mathematics, and visual material. Apache 2.0 weights are available; the model name rounds the approximately 25.2B parameter accounting to 26B.
InternVideo3 reframes long-video understanding as an evolving loop of observation, reasoning, tools, and memory, while compressing KV state without dropping the underlying token stream.
Ziang Yan et al.
Released: 2026-06-10
Figure 2. InternVideo3 with multimodal multi-head latent attention across long contexts. Source paper, PDF p. 7. Figure notice.
ℹ️ More Information
Multimodal Contextual Reasoning maintains one evolving context containing observations, instructions, intermediate reasoning, tool actions, and memory. This closed loop lets the model accumulate and verify evidence across long videos rather than treating video QA as a single static prompt.
Multimodal Multi-head Latent Attention reparameterizes and compresses KV-cache states while preserving the complete visual-token sequence. Training combines continued pretraining, short-to-long supervised tuning, rule-based reinforcement learning, and on-policy distillation; a retrieval-equipped video-agent implementation demonstrates the intended tool-using setting.
Keye-VL 2.0 adapts DeepSeek Sparse Attention to a GQA-based multimodal MoE, targeting lossless 256K contexts, hour-scale video, and self-correcting tool-using agents.
Kwai Keye Team
Released: 2026-06-09
Figure 2. Four-stage curriculum extending Keye-VL from alignment to 256K context. Source paper, PDF p. 8. Figure notice.
ℹ️ More Information
Keye-VL-2.0-30B-A3B is the first reported adaptation of DeepSeek Sparse Attention to grouped-query multimodal attention. Only 3B of 30B parameters activate per token, while the sparse retrieval mechanism keeps critical frames and long-range dependencies accessible across a 256K context. Heterogeneous ViT-language-model parallelism and custom sparse-attention kernels address the systems cost of hour-long video.
Cross-Modal Multi-Teacher On-Policy Distillation feeds dense token-level guidance from specialist teachers back into on-policy trajectories. Context-RL and Video-RL then target long-context reasoning, temporal localization, code, search, tools, and multimodal self-correction without collapsing the multi-task mixture.
Zamba2-VL combines efficient Mamba-2 state-space layers with a small number of shared Transformer blocks, reducing long-context prefill and recurrent-state costs across compact VLM scales.
Zyphra
Released: 2026-06-02
Figure 1. Zamba2 hybrid state-space language backbone connected to a vision encoder. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Zamba2-VL extends the Zamba hybrid backbone to vision-language inputs in 1.2B, 2.7B, and 7B variants. Most sequence processing occurs in Mamba-2 state-space layers, with a small set of shared attention blocks supplying global token interaction. This preserves transformer-like multimodal reasoning while moving more computation toward near-linear prefill and bounded recurrent state.
The architecture is particularly relevant for edge and long-context deployment: visual sequences can be large even when the language model is small, so reducing quadratic attention changes time-to-first-token and memory behavior more materially than ordinary parameter pruning.
MiniMax M3 combines native multimodal pretraining with learned block-sparse grouped-query attention, making million-token contexts practical in a 109B-parameter model.
MiniMax
Released: 2026-06-01
Figure 1. MiniMax Sparse Attention index and exact-attention branches. Source paper, PDF p. 1. Figure notice.
ℹ️ More Information
MiniMax Sparse Attention adds a lightweight index branch that scores KV blocks independently for each grouped-query-attention group. The main branch performs exact attention only over selected blocks, while a KV-outer kernel groups queries retrieving the same block to improve memory locality and reuse.
The released 109B-parameter model is trained natively across modalities and evaluated at contexts up to one million tokens. The sparse-attention report focuses on architecture, kernel design, and scaling behavior; it does not publish a complete source-level training-data inventory.
MiniMax M3.1 Flash launched on October 19, 2026. Its card describes a 428B/23B-active native multimodal MoE using MSA throughout, quantization-aware training, and a DSpark draft head. It accepts text, images, and video and emits text with up to 1M context. Weights are available under MiniMax Community terms at the review cutoff; the October 19 launch initially announced hosted availability and deferred weights. The linked MSA report and figure explain the attention mechanism, not the full M3.1 training recipe or corpus. Launch · Model card
Cosmos 3 couples an autoregressive reasoner with a diffusion generator through a shared representation for language, vision, audio, actions, simulation, and robot control.
NVIDIA
Released: 2026-05-31
Figure 5. Mixture-of-Transformers reasoner and generator with shared attention. Source paper, PDF p. 11. Figure notice.
ℹ️ More Information
Cosmos 3 uses a Mixture-of-Transformers with two communicating towers. An autoregressive transformer reasons over language, vision, and action sequences; a diffusion transformer generates continuous images, video, and world trajectories. Their shared representation lets the family serve as a VLM, forward or inverse dynamics model, simulator, generator, or policy without separate task pipelines.
Nano configurations combine 8B-class components and Super configurations combine 32B-class components. Training spans text, image, video, audio, simulated physical interaction, action trajectories, robot demonstrations, driving, and synthetic worlds. The July 20 Cosmos 3 Edge release preserves the same two-tower design in a smaller real-time deployment branch and is folded into this family.
A visual-agent family learns computer interaction from verified trajectories generated in live and synthetic environments.
Microsoft Research
Released: 2026-05-21
Figure 2. A visual-agent family learns computer interaction from verified trajectories generated in live and synthetic environments. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Fara1.5 uses Qwen3.5 at 4B, 9B, and 27B scales. Screenshots and interaction history condition structured mouse/keyboard actions without requiring a DOM representation. The distinctive contribution is the FaraGen1.5 environment, solver, and verifier pipeline.
Supervised tuning balances broad task coverage, high-value tasks, and weaknesses exposed by the current model. Trajectories are checked for task correctness, efficiency, and adherence to critical points; a user simulator supports multi-turn tasks.
Data combines live websites with synthetic environments for otherwise difficult-to-collect domains. The paper explains generation and filtering rather than claiming that benchmark traces are training data. MIT model weights and code are available. May 21 is the card’s explicit release date; the paper followed June 18.
Lance is a compact model trained from scratch to understand, generate, and edit images and video inside one interleaved sequence while preserving specialized semantic and synthesis capacity.
Lance Team
Released: 2026-05-18
Figure 6. Dual-expert sequence modeling for understanding and visual generation. Source paper, PDF p. 9. Figure notice.
ℹ️ More Information
Lance interleaves text, image, and video in a shared 3B-parameter sequence model while routing semantic understanding and visual synthesis through dedicated experts. Semantic ViT tokens coexist with clean or noised VAE latents, and generalized 3D causal attention plus modality-aware positional encoding preserve spatial and temporal relationships.
One checkpoint supports visual understanding, text-to-image and text-to-video generation, image-to-video generation, and editing. Its training-from-scratch recipe uses no more than 128 A100 GPUs, making the contribution as much about an attainable unified architecture as headline generation quality.
ZAYA1-VL adds visual capacity to a sparse MoE through vision-only LoRA and compressed-convolutional-attention parameters, trained on an entirely open multimodal data mixture.
Zyphra
Released: 2026-05-08
Figure 2. Visual routing, compressed convolutional attention, and a hybrid language backbone. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
ZAYA1-VL-8B joins a Qwen2.5-VL visual encoder to ZAYA1's MoE decoder. Vision-only LoRA and Compressed Convolutional Attention parameters activate for visual tokens without duplicating the language backbone, while bidirectional attention within image-token spans improves spatial integration.
The model is trained on roughly 140B vision-language tokens assembled from open data and released under Apache 2.0. Its importance is the clean separation of visual specialization from shared language capacity in a compact sparse model.
A VLM reads rendered text pages and selectively expands relevant regions into higher-fidelity context through learned tool use.
Roy Xie et al., Apple
Released: 2026-05-07
Timeline basis: First arXiv submission; paper revised October 9.
Figure 2. A VLM reads rendered text pages and selectively expands relevant regions into higher-fidelity context through learned tool use. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
LensVLM uses a Qwen3.5-9B backbone. Long text is rendered into compact images; the model alternates reasoning with an Expand tool that retrieves original text or higher-resolution visual evidence. Compression and selective retrieval jointly determine effective context use.
Evidence-grounded synthetic tool trajectories provide supervised fine-tuning, followed by on-policy reinforcement learning. The vision encoder and projector remain frozen while language parameters learn answer and expansion behavior. Tool responses enter context but are excluded from the supervised loss.
The method is evaluated across long-context question-answering datasets with held-out task transfer. Model weights and code are published with Apple-specific licenses, rather than a blanket permissive-weight claim. The timeline uses the first paper date; later checkpoint publication and paper revisions are separate events.
Falcon Perception uses early image-text fusion, hybrid attention, instance tokens, and continuous mask heads to unify open-vocabulary grounding, segmentation, and OCR in compact models.
Technology Innovation Institute
Released: 2026-05-03
Figure 1. Early-fusion perception Transformer with grounding, geometry, and segmentation pathways. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Falcon Perception is a dense early-fusion Transformer in which image patches and text interact from the first layer. Its attention mask is bidirectional over image tokens and causal over prediction tokens. Variable-length instance tokens identify multiple objects, while lightweight continuous heads produce masks without turning segmentation into an unwieldy text sequence.
The family includes a 600M perception model and a 300M OCR model. The paper was submitted March 28, but the timeline uses the documented May 3 public model launch, following this repository's release-date policy.
A document-focused VLM compresses local visual windows and injects both depth and spatial feature groups into its decoder.
IBM Research
Released: 2026-04-29
Architecture figure: The Granite Vision 4.1 card describes Window Q-Former and dual DeepStack connections in text. The linked ChartNet dataset paper is not a model-specific architecture diagram.
ℹ️ More Information
SigLIP2-SO400M encodes 384-pixel tiles and a global view. Window Q-Formers reduce each 4×4 patch group to 2×2 tokens. LayerDeepstack reverses the depth mapping from four encoder levels, while SpatialDeepstack injects four complementary spatial groups, yielding eight injection points into Granite 4.1.
Training uses LoRA in the vision encoder and language model; adapters are merged in the released checkpoint. Extraction-focused fine-tuning covers charts, complex tables, key-value pairs, and general visual instructions.
ChartNet contributes million-scale chart data with rendering code, images, source values, and summaries, including real and synthetic examples. General Granite Vision instruction data supplements extraction tasks. The roughly 4B model accepts English instructions and images and is available under Apache 2.0.
GLM-5V-Turbo integrates visual perception into reasoning, planning, tool use, execution, and verification instead of treating images and video as an auxiliary language-model interface.
GLM-V Team
Released: 2026-04-29
Figure 2. Multimodal multi-token prediction with image placeholders and shared Transformer blocks. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
GLM-5V-Turbo is organized around end-to-end multimodal agency across images, video, webpages, documents, GUIs, code, and external tools. Its contribution is the integration of perception with the agent loop: observations remain available during planning and execution, and outcomes can be visually verified rather than handed off to a detached captioning stage.
The report describes coordinated changes to model design, multimodal training, hierarchical optimization, reinforcement learning, toolchains, and agent-framework integration. Because this is a new native agent foundation model rather than a latency-only checkpoint, it is kept separate from the GLM-4.1V family.
A hybrid Mamba-Transformer MoE adds native audio and temporal visual compression to efficient multimodal reasoning.
NVIDIA
Released: 2026-04-28
Figure 1. A hybrid Mamba-Transformer MoE adds native audio and temporal visual compression to efficient multimodal reasoning. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
The Nemotron 3 Nano 30B-A3B backbone connects to C-RADIOv4-H vision and Parakeet-TDT audio encoders through MLP projectors. Native-aspect-ratio image processing replaces tiling, while Conv3D compresses video tokens temporally. Context expands to 256K.
Staged training begins with vision supervised tuning, adds joint omni-modal supervision while extending context, and develops reasoning through further post-training. Modality balancing and text-preservation stages reduce interference with the original language model.
Text, images, video, and audio produce text. The release includes BF16/FP8/NVFP4 checkpoints, selected training pipelines, and the approximately 6.9M-sample Nemotron-Image-Training-v3 subset. Weights use the NVIDIA Open Model Agreement; a partial data release should not be described as the entire training corpus.
A native mixture-of-transformers model unifies visual understanding and generation through lightweight pixel interfaces without a separate pretrained vision encoder or VAE.
SenseNova Team
Released: 2026-04-27
Figure 3. A native mixture-of-transformers model unifies visual understanding and generation through lightweight pixel interfaces without a separate pretrained vision encoder or VAE. Source paper, PDF p. 7. Figure notice.
ℹ️ More Information
SenseNova-U1 follows NEO-unify with modality-specialized transformer computation and lightweight visual interfaces. U1.5 refines spatially coherent patch reconstruction and the encoding/decoding interfaces for native 4K generation, editing, and multimodal understanding.
Pretraining and supervised tuning are followed by specialist reinforcement learning and multi-expert on-policy distillation. Specialists cover aesthetics, text rendering, infographics, and editing. Distilled eight-step adapters provide a separate efficiency option.
The reports describe understanding, generation, editing, and interleaved data mixtures. U1 launched April 27; U1.5 preview appeared July 31 and the final 8B checkpoint August 20. The September 24 adapter revision is a follow-up within this family. Code and weights are linked; individual model/adaptor terms apply.
PLaMo 2.1-VL provides compact 2B and 8B Japanese-English VLMs that combine visual question answering and grounding for local, edge, and autonomous-device deployment.
Tommi Kerola et al., Preferred Networks
Released: 2026-04-21
Architecture figure: The PLaMo 2.1-VL paper contains application and data figures, but no model architecture diagram.
ℹ️ More Information
PLaMo 2.1-VL is designed around a constrained deployment envelope rather than frontier scale. Both sizes support VQA and bounding-box grounding, with evaluations emphasizing Japanese language operation and applications such as factory tool recognition and infrastructure anomaly detection.
A large synthetic-data pipeline supplies Japanese multimodal instructions, grounding examples, and domain-oriented supervision. The entry broadens the catalog's language and edge coverage; its contribution is a deployable bilingual VQA-and-grounding family rather than a new attention primitive.
A hybrid-attention multimodal MoE couples long-context perception with adaptive text-speech alignment for streaming conversation.
Qwen Team
Released: 2026-04-17
Timeline basis: First arXiv submission.
Figure 2. A hybrid-attention multimodal MoE couples long-context perception with adaptive text-speech alignment for streaming conversation. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Both Thinker and Talker use hybrid attention and mixture-of-experts computation. The system scales to hundreds of billions of parameters and 256K context. ARIA dynamically aligns text and speech units to stabilize streaming synthesis and prosody.
Training combines heterogeneous text-vision pairs with more than 100 million hours of audio-visual content. Multimodal pretraining and post-training develop perception, reasoning, speech interaction, and temporally grounded descriptions.
The report covers long audio, video, multilingual speech, and structured scene captions. Its hosted Plus model should not be confused with the openly downloadable Qwen3-Omni-30B weights. The timeline uses the first report date; detailed model size and full training corpus are not publicly released.
EXAONE 4.5 is LG AI Research's first open-weight VLM, integrating a dedicated visual encoder into EXAONE and emphasizing document-centric native multimodal pretraining and Korean context.
Eunbi Choi et al., LG AI Research
Released: 2026-04-09
Figure 1. Native-resolution vision encoding, projection, language decoding, and multi-token prediction. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
EXAONE 4.5 adds a dedicated vision encoder to the EXAONE 4.0 framework and trains the resulting stack natively over text and visual inputs. Its data mixture is deliberately document-heavy, targeting enterprise documents, OCR-rich layouts, and Korean contextual reasoning while retaining general language capability.
Context extends to 256K tokens for long documents and enterprise workflows. The combination of open weights, native multimodal training, long context, and focused document/Korean coverage makes it a distinct public VLM release rather than a post-training adapter.
BidirLM converts pretrained causal language decoders into bidirectional representation encoders; BidirLM-Omni extends the method to one contrastively aligned text, image, and audio embedding model.
BidirLM Team
Released: 2026-04-02
Figure 10. Specialist-backbone merging with frozen modality projection heads. Source paper, PDF p. 25. Figure notice.
ℹ️ More Information
BidirLM removes the causal mask and adapts a decoder with masked next-token prediction and contrastive learning. Weight merging and mixed-domain training preserve generative knowledge while recovering the full-context interaction needed by retrieval and embedding tasks.
BidirLM-Omni combines Qwen3 text, Qwen3-VL, and Qwen3-ASR specialist backbones with frozen visual and audio projectors, aligning all three modalities in one contrastive space. The entry belongs beside CLIP, SigLIP, and ImageBind as an encoder architecture, despite originating from decoder checkpoints.
Gemma 4 is Google DeepMind's open-weight multimodal family spanning mobile-efficient, dense, sparse-MoE, and encoder-free designs, with configurable reasoning and deployment targets ranging from phones to servers.
Gemma Team, Google DeepMind
Released: 2026-03-31
Figure 2. Aspect-preserving image resizing, patch pooling, and soft-token production. Source paper, PDF p. 16. Figure notice.
ℹ️ More Information
Gemma 4 is a suite of related architectures rather than one uniform network. E2B and E4B target mobile and edge deployment; 31B is a dense server-class decoder; and 26B-A4B is a sparse Mixture-of-Experts model with roughly 4B active parameters. These branches accept images through improved visual processing, while the mobile models additionally support audio. The distinctive Gemma 4 12B Unified removes separate vision and audio encoders: raw image patches and audio features enter through direct linear projections, reducing duplicated encoder memory and latency. Context windows reach 128K or 256K depending on the variant.
Pretraining jointly develops language, code, reasoning, and multimodal understanding, followed by instruction tuning, safety alignment, and reasoning post-training. A configurable thinking mode lets the instruction-tuned models emit a reasoning trace before answering. The family also ships dedicated multi-token-prediction draft models for speculative decoding, while per-layer embeddings and shared key-value caching improve inference efficiency in applicable variants.
Google describes a large multilingual mixture containing web text, documents, code, mathematics, image-text pairs, OCR and document examples, audio, and video-derived supervision. The complete record-level training corpus is not released, so benchmark collections should not be mistaken for an exhaustive dataset list. Image support includes variable resolutions and aspect ratios; native audio and video coverage depends on the specific branch.
Penguin-VL challenges the assumption that compact VLMs require contrastively pretrained vision backbones, adapting a text-only language model into a fine-grained visual encoder for efficient image and video understanding.
Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, Tianyuan Qu, Rossell Chen, Dong Yu, Leoweiliang
Released: 2026-03-06
Figure 3. An LLM-initialized vision encoder with priority-aware video-token compression. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Penguin-VL builds its Penguin-Encoder by initializing a vision backbone from a text-only Qwen3-0.6B model. Causal attention is converted to bidirectional attention, and 2D rotary position embeddings permit variable-resolution visual tokenization. The encoder is connected to compact language decoders to form 2B- and 8B-class VLMs. For video, Temporal Redundancy-Aware compression assigns larger token budgets to informative key frames and compresses redundant intermediate frames, allowing longer sequences within a fixed visual-token budget.
Training begins with mixed-supervision encoder adaptation. A teacher vision encoder supplies amplitude, direction, and relational distillation targets while image reconstruction-style supervision stabilizes the LLM-to-vision conversion. A low-to-high-resolution curriculum then aligns the encoder for fine-grained perception, followed by unified image and video multimodal pretraining and instruction tuning. This recipe is designed to preserve OCR, spatial, temporal, and dense-captioning signals that category-oriented contrastive objectives can suppress.
The project releases Penguin-Recap-I, a reconstructed high-quality image training set, and Penguin-Recap-V, video annotations at dense timestamp, paragraph, and whole-video granularities. These resources complement broader image-text, document, OCR, mathematical reasoning, and video instruction mixtures described in the report. The authors' comparisons attribute the compact models' gains primarily to visual representation quality and token allocation rather than parameter scaling alone.
Phi-4-Reasoning-Vision-15B is a compact open-weight model that combines high-resolution perception with selectable direct and chain-of-thought modes, focusing on mathematical, scientific, document, and GUI reasoning.
Jyoti Aneja, Michael Harrison, Neel Joshi, Tyler LaBonte, John Langford, Eduardo Salinas
Released: 2026-03-04
Figure 3. SigLIP2 vision encoding, cross-modal projection, mid-fusion, and language reasoning. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Phi-4-Reasoning-Vision-15B uses a mid-fusion architecture built from the Phi-4-Reasoning language backbone and a SigLIP 2 vision encoder. Visual features are projected into the language embedding space and inserted into the decoder sequence. Dynamic-resolution processing produces up to 3,600 visual tokens, supporting fine-grained documents and interface screenshots. Bidirectional attention is restricted to tokens within each image, improving spatial interaction while avoiding the overfitting Microsoft observed with broader bidirectional attention.
The model is trained through supervised fine-tuning on a deliberately mixed curriculum of reasoning and non-reasoning examples. Explicit <think> and <nothink> modes allow one checkpoint to use extended reasoning for mathematics and science or answer directly for perception-heavy tasks such as captioning, OCR, detection, and grounding. Microsoft reports four days of training on 240 NVIDIA B200 GPUs, followed by safety SFT and automated red-team evaluation.
Training data primarily comprises heavily filtered and corrected open-source vision-language datasets, augmented with synthetic examples, internally produced domain data, and targeted acquisitions. Microsoft emphasizes filtering, error correction, and synthetic augmentation as larger contributors than raw scale, but does not publish a complete source-level dataset inventory. Public evaluations span scientific diagrams, charts, mathematical reasoning, OCR, and GUI grounding; those benchmarks are evaluation references, not a claimed exhaustive training list.
V-SONAR aligns image and video representations with SONAR's multilingual concept space, enabling text-trained concept models to process vision. V-LCM then instruction-tunes a latent-diffusion Large Concept Model directly over unified visual and linguistic embeddings.
Yifu Qiu, Paul-Ambroise Duquenne, Holger Schwenk
Released: 2026-03-01
Figure 1. Visual-semantic alignment and concept-space prediction. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
V-SONAR starts with Meta's Perception Encoder for image and video features and learns a lightweight projector into the pre-existing SONAR embedding space. Positional embeddings and a temporal-attention layer preserve frame order, while attention pooling produces a modality-level representation compatible with SONAR text concepts. Because the target space is shared with OMNISONAR, the representation can be decoded into text across SONAR's multilingual coverage. A text-only Large Concept Model can also consume these aligned visual embeddings without vision-specific pretraining.
V-SONAR is trained with a coarse-to-fine post-hoc alignment curriculum rather than retraining the foundation encoders from scratch. The alignment uses paired visual-caption supervision and SONAR text embeddings, with staged optimization and asynchronous learning rates for the newly initialized projector and pretrained encoder. V-LCM then concatenates V-SONAR visual concepts with SONAR language concepts and retains LCM's two-tower contextualizer and denoiser architecture. It predicts the next continuous concept embedding with the same latent-diffusion objective used during text-only LCM pretraining.
Vision-language instruction tuning uses M3IT, which spans image and video inputs, eight task categories, and 80 languages. Retrieval and captioning experiments additionally use VATEX, DREAM-1K, PE-Video, and VideoXum. The resulting system separates perception, multilingual semantic alignment, and generative concept modeling instead of converting every modality into a single discrete token vocabulary.
Qwen3.5 unifies text, image, and video understanding in one model family while combining hybrid linear attention with sparse expert routing. Qwen3.6 continues the same native-multimodal direction with stronger agentic coding and updated dense and MoE checkpoints.
Qwen Team
Released: 2026-02-16
Architecture figure: Qwen3.5 has no public technical paper containing an architecture figure.
ℹ️ More Information
Qwen3.5 is a natively multimodal decoder family rather than a separate text model with a later VL edition. Its flagship 397B-A17B checkpoint uses sparse Mixture-of-Experts routing, activating about 17B parameters per token, and combines Gated DeltaNet linear-attention layers with standard softmax-attention layers. Visual inputs are handled inside the same model interface as text, supporting image and video reasoning, tool use, coding, and long-context agent workflows. Smaller dense and MoE variants extend the design across deployment scales.
The training pipeline combines large-scale multimodal pretraining with supervised fine-tuning and reinforcement learning. Qwen describes a scalable asynchronous RL system spanning text, multimodal, and multi-turn interaction, enabling a single checkpoint to switch between thinking and non-thinking behavior. The data mixture covers natural language, code, mathematics, images, video, documents, GUI interactions, tool calls, and multi-turn agent trajectories; the full source-level corpus is not publicly enumerated.
Qwen3.6 is best treated as a continuation of this architecture rather than a separate VLM family. Its open dense and sparse checkpoints retain native multimodality and unified thinking and non-thinking modes while prioritizing stability and agentic coding. Folding these releases into the Qwen3.5 entry keeps the catalog focused on architectural changes rather than model-version churn.
Qwen3.8 Max and 27B continue this native multimodal family; the official repository records weight releases on August 12 and August 14, 2026. Max scales the sparse model to 2.4T total and 95B active parameters. These releases remain grouped here; Qwen3.8-Flash-Next, with its separately documented sparse-attention and n-gram-memory design, has its own entry. The existing Qwen3.5 figure is a family reference, not a Qwen3.8-specific diagram. Official releases
Youtu-VL extends the language vocabulary with learned visual codes, making visual tokens prediction targets so one autoregressive model can emit segmentation, depth, pose, and detection without task-specific heads.
Tencent Youtu Lab
Released: 2026-01-27
Figure 3. Unified visual-text autoregressive supervision and dense-output decoding. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Vision-Language Unified Autoregressive Supervision learns a visual codebook whose symbols expand the text vocabulary into one multimodal vocabulary. Visual tokens are not passive conditioning embeddings: the decoder predicts them alongside text through joint visual-token and text reconstruction.
Because dense outputs are serialized in the same vocabulary, a standard autoregressive VLM can produce segmentation, monocular depth, human pose, and object detections without separate heads. This turns dense perception into native generation while retaining ordinary visual question answering and instruction following.
Kimi K2.5 is a trillion-parameter native multimodal MoE for reasoning, coding, and computer use; K2.6 keeps the topology while extending context and agentic post-training.
Kimi Team, Moonshot AI
Released: 2026-01-27
Figure 10. Agentic reinforcement-learning environments, rollout management, and training services. Source paper, PDF p. 23. Figure notice.
ℹ️ More Information
Kimi K2.5 connects MoonViT to a 1T-total, 32B-active decoder with 61 layers, Multi-head Latent Attention, and 384 routed experts. Eight routed experts plus shared capacity activate per token. Native multimodal training supports images, video, reasoning, code, tools, and GUI interaction within the same model.
Released April 20, Kimi K2.6 retains the K2.5 architecture while extending context to 256K and strengthening coding, research, and multimodal-agent post-training. It is folded here rather than receiving a duplicate timeline row; Kimi K3 is separate because its attention, residual, sparsity, scale, and vision stack change materially.
A compact page-to-text architecture adds localized document elements, chart extraction, and verification-based training.
Said Taghadouini, Adrien Cavaillès, and Baptiste Aubertin, LightOn
Released: 2026-01-20
Timeline basis: First technical report for LightOnOCR-2; LightOnOCR-3 released October 8.
Figure 1. A compact page-to-text architecture adds localized document elements, chart extraction, and verification-based training. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
The 1B lineage joins a 400M NaViT encoder from Mistral Small 3.1, a 2×2 spatial-merging MLP projector, and a 600M Qwen3 decoder. LightOnOCR-3 retains this at 1B but adopts Qwen3.5 at 0.8B/4B. Grounding mode adds labels, normalized boxes, image descriptions, and chart values.
Large-scale distillation precedes fine-tuning and verifiable-reward optimization. The second-generation report studies localization rewards, checkpoint averaging, and task arithmetic. The October 8 third-generation release extends supervised grounding and document-specific reinforcement learning.
Training covers scans, scientific PDFs, multilingual documents, synthetic/transcribed pages, and verified grounding examples. Weights use Apache 2.0; datasets retain separate terms. The captured paper figure describes the 1B lineage, not the different Qwen3.5-based 0.8B/4B variants.
Step3-VL-10B combines a 1.8B language-aligned perception encoder with a Qwen3-8B decoder and aggressively compresses high-resolution visual features before language reasoning.
StepFun
Released: 2026-01-14
Architecture figure: The Step3-VL-10B report contains performance and RL figures, but no architecture diagram.
ℹ️ More Information
Step3-VL-10B is architecturally distinct from the much larger 2025 Step3 sparse-MoE system. Its 1.8B perception encoder is aligned to language during pretraining, then a projector with two stride-2 stages reduces the spatial sequence by 16× before feeding a Qwen3-8B decoder.
Images combine one 728×728 global view with 504×504 local crops, treated as independent batch items. Explicit newline tokens mark patch rows while standard one-dimensional RoPE is retained. The PaCoRe parallel proposer-and-synthesizer reasoning method is a post-training and inference addition rather than the defining visual architecture.
A grounded image-video model extends open multimodal training to temporal pointing, tracking, and multi-image reasoning.
Christopher Clark et al., Allen Institute for AI and University of Washington
Released: 2025-12-11
Figure 2. A grounded image-video model extends open multimodal training to temporal pointing, tracking, and multi-image reasoning. Source paper, PDF p. 6. Figure notice.
ℹ️ More Information
A SigLIP-family encoder feeds Qwen3 4B/8B or Olmo 7B through a connector pooling features from two encoder depths. Images use 2×2 pooling, videos 3×3; frame timestamps and packed attention preserve temporal organization. Outputs include answers, points, and tracks.
The released training stack includes pretraining, supervised fine-tuning, and long-context fine-tuning. Grounding-focused data construction teaches precise localization alongside captions and question answering.
Published datasets include Molmo2-Cap, VideoCapQA, VideoSubtitleQA, AskModelAnything, VideoPoint, VideoTrack, and multi-image QA/pointing collections. Weights use Apache 2.0; datasets retain their own terms. The December 11 announcement precedes the March 2026 full code release, so code availability and model release should be distinguished.
A sparse multimodal decoder scales visual and language reasoning through fine-grained expert routing.
Mistral AI
Released: 2025-12-02
Architecture figure: The Mistral Large 3 release and model card disclose model configuration but no model-specific architecture figure; the Ministral 3 training figure belongs to a different family.
ℹ️ More Information
Mistral Large 3 has approximately 675B total and 41B active parameters including a 2.5B vision encoder. The language component accounts for approximately 673B total and 39B active parameters. Visual inputs join the language sequence, with a supported context of 256K.
The model was trained from scratch and instruction-tuned for multilingual reasoning and tool use. The official release describes large-scale distributed training but does not publish a complete pretraining corpus or fully reproducible alignment recipe.
Inputs are text and images; outputs are text. Apache 2.0 weights and model configuration are available. Parameter totals should specify whether the visual encoder is included. The subsequently announced Mistral Large 4 preview is tracked separately in the review notes, not represented as released Large 3 weights.
A family of compact dense decoders integrates visual perception across base, instruction, and reasoning variants.
Mistral AI
Released: 2025-12-02
Figure 1. A family of compact dense decoders integrates visual perception across base, instruction, and reasoning variants. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
Ministral 3 supplies 3B, 8B, and 14B language-model scales with a visual encoder and projector. It preserves dense inference and long-context image-text processing rather than adopting the sparse expert routing of Mistral Large 3.
Cascade Distillation prunes Mistral Small 3.1 into a 14B child, up-trains it with logit distillation, then repeats the process for 8B and 3B. Long-context distillation follows. Instruction variants use SFT and online DPO; reasoning variants add chain-of-thought SFT and GRPO.
Public models support text and image inputs with textual output and are released under Apache 2.0. The report discusses training data curation and alignment, while the full underlying corpus is not released. The December announcement is the family release; the technical report appeared in January 2026.
ERNIE 5.0 unifies text, image, video, and audio understanding and generation in a 2.4T ultra-sparse MoE trained through modality-specific forms of grouped next-token prediction.
Baidu ERNIE Team
Released: 2025-11-13
Figure 2. Unified image understanding, image generation, and video generation objectives. Source paper, PDF p. 6. Figure notice.
ℹ️ More Information
ERNIE 5.0 maps language, images, video, and audio into one autoregressive framework using Next-Group-of-Tokens Prediction. Text uses next-token and multi-token prediction; visual generation predicts the next frame and scale; audio generation predicts codec tokens depth-wise. Modality-agnostic expert routing activates less than 3% of the 2.4T-parameter network.
Elastic depth, width, and top-k sparsity train a super-network from which smaller deployment configurations can be extracted. Baidu first unveiled ERNIE 5.0 at Baidu World on November 13, 2025, formally released it in January 2026, and published the technical report on February 4; the timeline therefore uses the earliest documented public disclosure.
A native discrete-token model learns interleaved visual-language sequences and explores parallel decoding through diffusion adaptation.
Yufeng Cui et al., BAAI
Released: 2025-10-30
Timeline basis: First arXiv submission.
Figure 3. A native discrete-token model learns interleaved visual-language sequences and explores parallel decoding through diffusion adaptation. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
Emu3.5 predicts text and discrete visual tokens in one next-token sequence. The paper’s Discrete Diffusion Adaptation converts autoregressive visual generation to bidirectional parallel prediction while retaining the trained representation.
End-to-end pretraining uses more than 10T tokens, largely from sequential video frames and transcripts. Large-scale reinforcement learning improves multimodal reasoning and generation; diffusion adaptation is an additional inference-oriented stage.
The model supports interleaved understanding and generation, image editing, and long visual-language continuations. Public Emu3.5 and Emu3.5-Image checkpoints use next-token decoding; the repository still lists DiDA inference/weights as forthcoming. The paper’s accelerated results therefore should not be attributed to those downloadable checkpoints.
A dense hybrid Mamba-Transformer language backbone combines C-RADIO perception with token-efficient video handling.
NVIDIA
Released: 2025-10-28
Figure 1. A dense hybrid Mamba-Transformer language backbone combines C-RADIO perception with token-efficient video handling. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
C-RADIOv2-H features pass through an MLP connector into Nemotron Nano 12B V2, yielding about 12.6B parameters overall. Images use dynamic tiling; video uses frame sampling and Efficient Video Sampling to remove redundant visual tokens.
Training warms up the connector, then unfreezes the model for multimodal instruction tuning, video/context extension, text-recovery tuning, and further long-context training. The paper distinguishes training sequence lengths from deployed context settings.
The card inventories approximately 39.5M samples from 270 text, image-text, and video-text datasets, mixing public and internally annotated material. Published data subsets and recipes accompany model weights. Inputs are text, images, and video; output is text. Checkpoint and data licenses are documented separately.
DeepSeek-OCR treats document vision as a context-compression mechanism, representing thousands of text tokens with a much smaller sequence of visual tokens before decoding them with a sparse language model.
DeepSeek-AI
Released: 2025-10-20
Figure 3. A SAM-CLIP DeepEncoder connected to a sparse language decoder. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
DeepSeek-OCR consists of DeepEncoder and a DeepSeek-3B-MoE decoder with roughly 570M active parameters. DeepEncoder combines a SAM-derived local-perception pathway, a CLIP-derived semantic pathway, convolutional compression, and windowed and global attention to turn high-resolution pages into as few as 64 to 800 visual tokens. Multiple resolution modes trade recognition fidelity for compression.
Training first develops the encoder's visual-text representation and then jointly optimizes document decoding with the MoE language model. Data covers multilingual text pages, natural images, formulas, tables, charts, diagrams, and structured document conversion, supplemented with synthetic renderings. The work frames OCR as an experiment in optical compression for future long-context memory, rather than only a document benchmark model.
DeepSeek-OCR 2, released January 27, 2026, replaces the fixed raster-scan assumption with DeepEncoder V2 and Visual Causal Flow. The encoder dynamically reorders visual tokens according to document semantics before language decoding, exploring whether cascaded one-dimensional causal reasoning can better represent complex two-dimensional layouts. Paper · Repository
Joint embedding alignment and explicit relative/absolute time structure combine vision and audio within one language model.
Hanrong Ye et al., NVIDIA
Released: 2025-10-17
Timeline basis: First arXiv submission.
Figure 2. Joint audio-video perception and text reasoning; the illustrated frozen TTS component is external speech synthesis. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
OmniAlignNet strengthens interaction between audio and visual representations before language decoding. Temporal Embedding Grouping preserves relative alignment, while Constrained Rotary Time Embeddings encode absolute timing. Separate modality encoders feed a common omni-modal representation pathway.
The training study combines single-modality and cross-modal supervision and evaluates transfer between perception and reasoning. Its staged recipe uses roughly 0.2T training tokens; downstream reinforcement-learning experiments explore further reasoning gains.
A curation and synthesis pipeline produces 24M conversations from public audio, image, and video sources. The model outputs text from text/image/audio/video inputs. Apache-licensed weights, code, and data-processing descriptions are available; the paper provides the detailed data composition and ablations.
PaddleOCR-VL combines a NaViT-style dynamic-resolution encoder with ERNIE-4.5-0.3B to parse multilingual documents using only 0.9B parameters.
PaddleOCR Team, Baidu
Released: 2025-10-16
Figure 2. Document layout analysis, compact VLM inference, instructions, and structured output. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
PaddleOCR-VL-0.9B connects a NaViT-style dynamic-resolution visual encoder to the compact ERNIE-4.5-0.3B language model. Variable-resolution packing preserves page detail and aspect ratio without forcing every document through a fixed square canvas. The decoder emits text and structured representations for paragraphs, tables, formulas, charts, and other document elements across 109 languages.
Its curriculum combines element-level recognition with page-level document parsing, followed by instruction tuning on complex layouts and structured outputs. Training data includes multilingual OCR, synthetic and scanned documents, tables, mathematical expressions, charts, reading-order annotations, and layout-rich pages. PaddleOCR-VL-1.5 (January 2026) adds robust physical-document distortions, seal recognition, and text spotting; 1.6 (June 2026) adds region-aware data optimization and progressive reinforcement-learning post-training without changing the core architecture.
A Thinker-Talker mixture-of-experts system performs multimodal perception and generates streaming text and speech.
Qwen Team
Released: 2025-09-22
Figure 2. A Thinker-Talker mixture-of-experts system performs multimodal perception and generates streaming text and speech. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
The 30B-A3B family separates multimodal reasoning in the Thinker from speech production in the Talker. Multicodebook speech prediction and a causal convolutional decoder replace the earlier block-diffusion speech decoder, enabling streaming output. Audio and video retain temporal alignment.
Training progresses from text capability to mixed multimodal learning and instruction/reasoning post-training. Instruct checkpoints include Thinker and Talker; Thinking checkpoints retain the reasoning component and return text only.
The release covers text, images, audio, and video. Public code and Apache-licensed checkpoints are linked. The later Qwen3.5-Omni report adds hybrid-attention Thinker/Talker scaling, 256K context, and ARIA text-speech alignment; those hosted-model specifications should not be attributed to the 30B checkpoint.
Qwen3-VL expands the Qwen vision-language family with dense and sparse-MoE checkpoints, native long multimodal context, and stronger spatial, video, OCR, visual-agent, and visual-coding capabilities.
Qwen Team
Released: 2025-09-22
Figure 1. Vision encoding, DeepStack injection, and dense or mixture-of-experts decoding. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Qwen3-VL combines a native-dynamic-resolution Vision Transformer with Qwen3 language backbones in dense 2B, 4B, 8B, and 32B configurations and sparse 30B-A3B and 235B-A22B Mixture-of-Experts configurations. Its DeepStack mechanism injects features from multiple ViT depths into corresponding language-model layers instead of relying only on the visual encoder's final layer. Interleaved-MRoPE distributes temporal, height, and width position information across rotary-embedding frequencies, while explicit text-timestamp alignment improves event localization in video. The family supports interleaved text, images, and video in a native 256K-token context, with documented extrapolation for longer contexts.
Training proceeds through large-scale multimodal pretraining followed by supervised instruction tuning and reasoning-oriented post-training. The released Instruct checkpoints target direct response and agent interaction, whereas Thinking checkpoints generate extended reasoning for visual mathematics, spatial analysis, and other difficult multimodal tasks. The curriculum emphasizes recognition, multilingual OCR, document structure, 2D and 3D grounding, long-video understanding, GUI operation, and code generation from visual inputs.
The technical report describes a broad mixture of text, image-text, document, OCR, grounding, chart, multi-image, video, and agent-interaction data. Qwen does not publish a reproducible itemized list of every pretraining source, so individual benchmark datasets should not be presented as the complete training corpus.
A small active-parameter MoE combines token-efficient multi-crop vision with learned attention-temperature scaling.
Moondream / M87 Labs
Released: 2025-09-18
Architecture figure: The Moondream 3 release notes and model card describe the sparse decoder and visual pathway but provide no model-specific architecture diagram or technical paper.
ℹ️ More Information
The 9B/2B-active decoder contains 24 layers: four dense layers followed by twenty MoE layers with 64 experts and eight selected per token. GeGLU expert networks, a SuperBPE tokenizer, and position/data-dependent attention temperature support the 32K context. SigLIP-based multi-crop features are concatenated along channels to control visual-token cost.
The preview release emphasizes improved training dynamics and reinforcement-learning suitability for visual reasoning. Its public notes do not disclose a full training-stage schedule or dataset inventory, so neither is reconstructed from older Moondream models.
The model accepts images and text for visual questions and specialized perception tasks. Weights, custom inference code, and a demo are public under the checkpoint’s own license. September 18 is the official preview announcement; the earlier repository creation and later deployment updates are not separate launches.
A scientific MoE connects visual and time-series encoders with domain-adaptive tokenization for symbolic scientific inputs.
Lei Bai et al., Shanghai AI Laboratory and collaborators
Released: 2025-08-21
Timeline basis: First arXiv submission; earlier model launch is separately documented.
Figure 3. A scientific MoE connects visual and time-series encoders with domain-adaptive tokenization for symbolic scientific inputs. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Intern-S1 combines a 235B Qwen3 MoE with a 6B InternViT, a time-series pathway, and a dynamic tokenizer for scientific sequences such as molecular and protein representations. It preserves general image-language capabilities alongside specialist scientific reasoning.
Continued pretraining uses 5T tokens, over half from scientific domains, followed by supervised tuning and reinforcement learning with a mixture of rewards. Domain-specific tokenization and multimodal alignment connect otherwise heterogeneous scientific representations.
The report documents scientific literature and domain data processing. Later Intern-S2 learns directly from rendered scientific pages and expands multi-domain and long-horizon agent RL; its 397B label should not be substituted for S1’s architecture. Both releases provide Apache-licensed weights.
A hybrid short-convolution and attention language model connects to SigLIP2 NaFlex for efficient on-device visual reasoning.
Liquid AI
Released: 2025-08-12
Official architecture diagram. LFM2-VL visual encoding, projection, and language backbone; the original family architecture. Primary source. Figure notice.
ℹ️ More Information
The family uses native-resolution SigLIP2 encoders, a multimodal projector, and an LFM language backbone. Larger images are split into tiles with a thumbnail. The August 2026 LFM2.5-VL-3B pairs a 2.6B language backbone with a 400M vision encoder; smaller 450M and 1.6B variants cover different budgets.
Training aligns image features with the pretrained language model, then develops instruction following through multimodal training. LFM2.5 improves OCR, grounding, multi-image understanding, and structured tool use.
The initial LFM2-VL release was August 12, 2025; the 3B LFM2.5 update arrived August 12, 2026. The latter supports 32K context and sixteen languages. Weights use the custom LFM License 1.0. Public cards describe task mixtures without publishing the full corpus.
Step3 is a 321B-total, 38B-active multimodal MoE whose architecture and serving system are co-designed to reduce the communication cost of sparse expert decoding.
StepFun
Released: 2025-07-25
Figure 6. Attention-FFN disaggregation across attention and expert instances. Source paper, PDF p. 11. Figure notice.
ℹ️ More Information
Step3 uses a 321B-parameter sparse-MoE decoder with approximately 38B parameters active per token. The language stack contains dense lower layers followed by routed expert layers, while its vision pathway supports images and video for multimodal reasoning. The model is designed together with an Attention-FFN Disaggregation serving architecture, separating attention and expert computation so hardware placement and communication topology match their different workloads.
Training combines large-scale text and multimodal pretraining with instruction tuning and reasoning-oriented reinforcement learning. The data mixture spans language, code, mathematics, image-text, OCR, documents, charts, multi-image, video, and multimodal reasoning. The system report discloses architecture and deployment mechanisms more clearly than the source-level training corpus, which remains categorically described.
GLM-4.1V-9B-Thinking combines a native-resolution AIMv2-based vision stack with long-chain-of-thought post-training and Reinforcement Learning with Curriculum Sampling to strengthen reasoning across visual, document, video, grounding, and agent tasks.
GLM-V Team, Zhipu AI and Tsinghua University
Released: 2025-07-01
Figure 2. Native-resolution vision encoding, projection, decoding, and timestamped video tokens. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
GLM-4.1V-9B-Thinking joins an AIMv2-Huge-initialized vision encoder, an MLP projector, and the GLM-4-9B-0414 language decoder. The vision tower replaces two-dimensional patch convolutions with 3D convolutions, temporally downsampling video while duplicating single images for consistent processing. Interpolated absolute position embeddings and 2D RoPE support native resolutions and extreme aspect ratios; the language decoder applies 3D RoPE to multimodal tokens. Explicit timestamp tokens after video frames supply temporal position and distance cues.
Training begins with broad multimodal pretraining, followed by continual training on video, higher-resolution imagery, and sequences up to 32K tokens. Supervised fine-tuning then teaches a standardized long-reasoning format with separate <think> and <answer> spans. The final stage introduces Reinforcement Learning with Curriculum Sampling (RLCS). Rather than sampling a fixed task mixture, RLCS estimates changing model competence and prioritizes informative problems of suitable difficulty. Domain-specific rule, model, and hybrid rewards cover both verifiable and open-ended tasks.
Pretraining draws on curated caption pairs, knowledge-rich web and academic-book interleaving, 220M OCR images, natural-image and GUI grounding, instructional video, and text replay. SFT and RL span STEM reasoning, charts, documents, video, grounding, coding, GUI agents, and general visual instruction following. The later GLM-4.5V and GLM-4.6V families scale and extend this framework; they are successors, not separate architectures in this catalog.
GLM-4.6V adds native visual function calling: tool arguments can contain images and the model can interpret visual tool results. Its 106B and 9B Flash releases train for 128K context. Interleaved image-text responses can incorporate retrieved images; this should not be confused with a native image-generation decoder. The existing family paper and figure remain the architectural reference. 4.6V model card
ERNIE 4.5-VL combines shared and modality-specific experts with isolated routing and balancing losses so visual learning can reinforce rather than degrade language capability.
Baidu ERNIE Team
Released: 2025-06-30
Architecture figure: ERNIE 4.5-VL has no public paper containing an extractable architecture figure.
ℹ️ More Information
ERNIE 4.5-VL includes 424B-total/47B-active and 28B-total/3B-active multimodal MoE models, plus dense members of the broader ERNIE 4.5 family. Its heterogeneous modality MoE shares some experts across text and visual tokens while reserving other experts for individual modalities. Modality-isolated routing, router orthogonality loss, and multimodal token-balancing loss reduce competition between modalities.
The models are jointly pretrained on text, images, and video, then post-trained in both direct-response and thinking modes. PaddlePaddle infrastructure uses heterogeneous hybrid parallelism, hierarchical load balancing, FP8 mixed precision, and fine-grained recomputation to train and serve the sparse models efficiently. Baidu describes broad text, visual-knowledge, OCR, document, chart, video, and reasoning mixtures but does not provide a complete dataset manifest.
A dual-path visual representation combines autoregressive text modeling with flow matching in a shared multimodal transformer.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou
Released: 2025-06-19
Figure 1. A dual-path visual representation combines autoregressive text modeling with flow matching in a shared multimodal transformer. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Text tokens and image/video latents from a 3D causal VAE enter a unified model. Spatial-temporal fusion accommodates the different dependencies of understanding and generation. Separate output heads apply autoregressive text loss and continuous visual flow matching.
The training recipe develops visual representations and unified multimodal generation/understanding, then supports task-specific fine-tuning. The 1.5B and 7B language-backbone variants were released June 19; video-understanding checkpoints followed September 5.
The report describes image-text, interleaved, and video supervision. Released checkpoints and code support the documented tasks; planned video-generation releases should not be inferred from architecture capability alone. This differs materially from Show-o’s earlier discrete visual diffusion formulation.
MiMo-VL combines a four-stage, 2.4T-token multimodal curriculum with mixed on-policy reinforcement learning across perception, reasoning, and GUI grounding.
Xiaomi MiMo Team
Released: 2025-06-04
Figure 2. Native-resolution vision encoding, projection, and language decoding. Source paper, PDF p. 6. Figure notice.
ℹ️ More Information
MiMo-VL-7B joins a high-resolution vision encoder and projector to a 7B language decoder and is released as supervised and reinforcement-learned checkpoints. Its primary contribution is the training system rather than an exotic connector: long-chain-of-thought examples are introduced during pretraining, not only during post-training, so multimodal reasoning is developed alongside perception and language.
Four pretraining stages consume approximately 2.4T tokens and progressively align vision, general multimodal knowledge, high-quality reasoning, and long-context capabilities. Mixed On-Policy Reinforcement Learning then trains one checkpoint across general QA, mathematics, grounding, and GUI tasks using domain-appropriate rule and model rewards. The mixture includes text replay, image-text, OCR, document, chart, video, spatial-grounding, GUI, and reasoning data; a complete source-level inventory is not published.
BAGEL unifies visual understanding, autoregressive language modeling, and rectified-flow image generation in a decoder-only Mixture-of-Transformer-Experts whose modality-specific parameters communicate through shared self-attention.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, Haoqi Fan
Released: 2025-05-20
Figure 2. Shared self-attention with understanding and generation Transformer experts. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
BAGEL is a unified decoder-only model with 14B total and 7B active parameters. Its Mixture-of-Transformer-Experts architecture duplicates the Qwen2.5-initialized transformer into understanding and generation experts: text and SigLIP 2 ViT tokens use the understanding parameters, while FLUX-VAE latent tokens use the generation parameters. Both experts operate on one interleaved sequence through shared self-attention at every layer, avoiding a small connector bottleneck between comprehension and synthesis. Text is learned with next-token prediction; images are generated with rectified flow.
A generalized causal-attention scheme lets later text or images attend to earlier clean visual representations while preventing access to noised generation targets. Visual understanding uses a native-aspect-ratio-adapted SigLIP2-So400m encoder and MLP connector; generation uses a frozen FLUX VAE. Training progresses through alignment, large-scale pretraining, continued training, and supervised fine-tuning, jointly optimizing cross-entropy and flow-matching objectives.
BAGEL is pretrained on trillions of tokens assembled from text-only, paired image-text, and interleaved multimodal sources. The reported inventory includes 400M text samples, 500M understanding pairs, 1.6B generation pairs, 100M interleaved understanding samples, 45M video-derived sequences, and 20M web documents. OCR, charts, grounding, editing sets, and 500K reasoning-augmented generation and manipulation examples support document reading, spatial control, image editing, and long-context multimodal reasoning.
Seed1.5-VL couples a 532M-parameter vision encoder with a 20B-active sparse-MoE decoder for image, video, 2D and 3D grounding, GUI interaction, and long-chain-of-thought reasoning.
ByteDance Seed Team
Released: 2025-05-11
Figure 1. Native-resolution vision encoding, adaptation, sparse MoE decoding, and timestamped video. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
Seed1.5-VL uses a compact vision tower and a large sparse-MoE language model, maintaining 20B active decoder parameters while scaling total capacity substantially higher. Its visual pathway supports native-aspect-ratio images, multiple images, and video, with explicit support for spatial grounding and GUI control. The same model can provide short direct answers or extended multimodal reasoning.
Training progresses through vision-language pretraining, instruction tuning, long-chain-of-thought cold start, and reinforcement learning. The report emphasizes careful balancing of general text, visual knowledge, OCR and documents, charts, grounding, video, 3D understanding, GUI trajectories, games, and verifiable multimodal-reasoning problems. Source-level corpus provenance and all mixture proportions are not publicly reproducible. The arXiv v1 date is used because an earlier generally accessible official launch date is not unambiguously documented.
A reproducible vision-language family combines a perception encoder with compact decoders and detailed human-annotated video supervision.
Jang Hyun Cho et al., Meta FAIR
Released: 2025-04-17
Timeline basis: First arXiv submission.
Figure 1. PerceptionLM model and data overview, linking the visual encoder and language model to annotation and video-data pipelines. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
PLM connects Perception Encoder features through a visual projector to Llama 3-family decoders at 1B, 3B, and 8B scales. Images, multiple images, and videos support detailed descriptions and spatial-temporal question answering. Its central contribution is transparent data and training design.
The study measures the limits of synthetic-data scaling, then adds human annotations for fine-grained activities and grounded temporal descriptions. Training combines synthetic captions, public visual-language data, and newly collected video supervision without relying on proprietary-model distillation.
PLM-FGQA, PLM-STC, and smart-glasses QA provide detailed video supervision; PLM-VideoBench evaluates corresponding capabilities. Weights, data, and recipes are released with component-specific terms, including FAIR research restrictions for PLM weights. The figure below depicts the data/method overview rather than an undisclosed decoder schematic.
A contrastively pretrained encoder family adapts shared visual representations for image-video retrieval, language models, and dense spatial tasks.
Bolya et al., Meta FAIR
Released: 2025-04-17
Timeline basis: First arXiv submission.
Figure 1. A contrastively pretrained encoder family adapts shared visual representations for image-video retrieval, language models, and dense spatial tasks. Source paper, PDF p. 2. Figure notice.
Figure 2. PE-AV audio, frame, video, fusion, and text encoders with cross-modal contrastive objectives. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
PE uses vision transformers trained on image and video data. PE-core serves contrastive retrieval, PE-lang aligns intermediate visual representations to a language decoder, and PE-spatial adapts features for dense prediction. These are related encoder variants, not interchangeable chat models.
The paper studies large-scale contrastive learning and the mismatch between final contrastive embeddings and representations needed by downstream tasks. Task-specific alignment extracts language-friendly and spatially useful features from the pretrained encoder.
Training combines image-text and video-text supervision. The public repository supplies model variants, configurations, and evaluation recipes; licensing varies by artifact. PE-lang provides the perception pathway used by PerceptionLM and later visual agents.
PE-AV (December 16, 2025) extends the family to joint audio-video-text embeddings. Audio is compressed with DAC-VAE; separate audio, frame, video, fusion, and text encoders learn shared correspondences. Pretraining uses eight contrastive pair types, with two more during fine-tuning. A synthetic captioning engine supplies roughly hundreds of millions of audio-video pairs spanning speech, music, and environmental sounds. PE-A-Frame adds frame-level audio/text alignment. These are encoders for retrieval and perception, not chat decoders. Audiovisual report · PE-AV weights
InternVL3 moves the InternVL family to native multimodal pretraining, while InternVL3.5 adds coarse-to-fine reinforcement learning and dynamically routed visual resolution.
InternVL Team, OpenGVLab
Released: 2025-04-11
Architecture figure: The InternVL3 paper contains evaluation figures, but no definitive model architecture diagram.
ℹ️ More Information
InternVL3 retains the InternViT-MLP-LLM structure and dynamic image tiling of earlier InternVL releases, but jointly learns language and multimodal knowledge during native pretraining instead of treating vision alignment primarily as a later adaptation stage. The family spans dense and sparse-MoE language backbones. Its training recipe combines multimodal pretraining, supervised fine-tuning, Mixed Preference Optimization, and test-time scaling for visual reasoning.
Released on August 26, 2025, InternVL3.5 is part of the same evolving family. It introduces Cascade RL, using offline RL for stable coarse alignment followed by online RL for refinement. A Visual Resolution Router selects visual-token resolution according to input complexity, while Decoupled Vision-Language Deployment places the vision encoder and language model on different devices to balance inference load. Training covers image, multi-image, document, chart, video, GUI, grounding, multilingual, and reasoning data; full pretraining provenance is not enumerated.
Kimi-VL couples the native-resolution MoonViT encoder to a sparse Mixture-of-Experts language decoder, activating 2.8B of 16B decoder parameters while supporting images, video, agent interaction, and 128K-token contexts.
Kimi Team
Released: 2025-04-10
Figure 3. MoonViT, multimodal projection, and a sparse mixture-of-experts decoder. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Kimi-VL-A3B consists of MoonViT, a two-layer MLP projector, and a Moonlight-derived MoE language model with 16B total and 2.8B activated decoder parameters. MoonViT packs variable-sized patch sequences without tiling images into separately processed crops. It combines interpolated SigLIP-SO400M position embeddings with two-dimensional RoPE, preserving pretrained visual knowledge while representing fine spatial positions. A 2×2 pixel-shuffle operation compresses its output before projection into the language model.
After the language checkpoint's 5.2T-token text-only phase, Kimi-VL undergoes standalone vision training, vision-language alignment, joint pretraining, a high-quality cooldown, and long-context activation. These stages consume 4.4T additional tokens, including 2T for MoonViT training, 0.1T for alignment, and 2.3T across joint stages. Context is progressively extended from 8K to 128K. The instruct model receives joint supervised fine-tuning. Kimi-VL-Thinking adds long-chain-of-thought SFT and reinforcement learning; the later Thinking-2506 checkpoint is folded into this family.
The multimodal mixture is organized into six categories: caption, interleaved image-text, OCR, knowledge, video, and agent data. MoonViT learns from alt text, synthetic captions, grounding boxes, and OCR targets using both SigLIP-style contrastive and captioning losses. Long-video, long-document, academic visual QA, GUI-grounding, and text replay data provide long-context and agent capabilities while preserving language performance.
Llama 4 introduces early-fusion multimodality to the Llama family through sparse Mixture-of-Experts decoders with 17B active parameters and contexts extending from one million to ten million tokens.
Meta
Released: 2025-04-05
Architecture figure: The official Llama 4 model card describes the architecture in text and tables only.
ℹ️ More Information
Llama 4 Scout and Llama 4 Maverick are natively multimodal, autoregressive MoE models rather than text decoders retrofitted only during instruction tuning. Scout has 109B total parameters across 16 experts and Maverick approximately 400B across 128 experts; both activate about 17B parameters per token. Early fusion places image and multilingual text tokens in a shared model sequence. Scout supports a 10M-token context, whereas Maverick supports 1M tokens.
Scout was trained on roughly 40T tokens and Maverick on roughly 22T. Meta describes a mixture of public data, licensed data, information from its products and services, and interactions with Meta AI, without releasing a complete corpus manifest. Post-training combines supervised fine-tuning, online reinforcement learning, and direct preference optimization. Maverick was co-distilled from the unreleased Llama 4 Behemoth teacher; Scout also uses distillation during training.
Qwen2.5-Omni understands text, images, audio, and video while generating text and natural speech through a streaming Thinker-Talker architecture.
Qwen Team
Released: 2025-03-26
Figure 2. Thinker-Talker architecture for multimodal perception and streaming speech. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Qwen2.5-Omni is an end-to-end any-input model whose block-wise audio and vision encoders accept streaming speech, images, and video alongside text. Time-aligned Multimodal RoPE (TMRoPE) interleaves audio and video representations on a synchronized timeline. Its Thinker-Talker design separates semantic reasoning from speech realization: the Thinker produces text and hidden representations, while a dual-track autoregressive Talker conditions on those states to generate speech tokens concurrently.
A sliding-window diffusion transformer converts audio tokens into waveform output without waiting for a complete response, reducing first-packet latency. Training jointly aligns text, vision, video, audio, and speech-generation objectives, followed by multimodal instruction tuning. The report describes caption, OCR, video, audio-transcription, speech, and general text mixtures but does not publish a reproducible source-level inventory.
Gemma 3 combines a frozen SigLIP vision encoder with a decoder-only language model whose five-local-to-one-global attention pattern reduces long-context KV-cache cost while retaining 128K-token multimodal context.
Gemma Team
Released: 2025-03-12
Architecture figure: The Gemma 3 report contains examples and analysis charts, but no architecture diagram.
ℹ️ More Information
Gemma 3 is a family of 1B, 4B, 12B, and 27B open-weight models; the 4B, 12B, and 27B variants accept images as well as text. A shared, frozen 400M-parameter SigLIP vision encoder processes images at 896×896 resolution. Its embeddings are average-pooled into a fixed sequence of 256 soft visual tokens, while a Pan-and-Scan preprocessing method handles non-square and high-resolution inputs through crops. The 1B model remains text-only.
For efficient long-context inference, Gemma 3 changes the decoder from Gemma 2's balanced attention schedule to five local sliding-window layers for every global-attention layer. Local attention uses a 1,024-token window, substantially limiting KV-cache growth; global layers retain access to the complete context. The larger models are pretrained at 32K context and extended to 128K near the end of pretraining through RoPE rescaling.
Training relies heavily on knowledge distillation from a larger teacher, updating the language model while keeping the vision encoder frozen. Post-training combines supervised fine-tuning with preference-based reinforcement learning. The pretraining mixture contains text, code, mathematics, multilingual material, and multimodal examples; the report states token totals of 2T, 4T, 12T, and 14T for the four model sizes but does not publish a complete dataset inventory.
Aya Vision uses synthetic multilingual visual instruction data and cross-modal model merging to add image understanding while preserving the language capabilities of Aya Expanse.
Cohere Labs
Released: 2025-03-04
Architecture figure: The Aya Vision paper contains data and evaluation figures, but no model architecture diagram.
ℹ️ More Information
Aya Vision is released in 8B and 32B configurations supporting 23 languages. It connects a vision encoder to Aya Expanse-derived language models and addresses the loss of text-only capability that often occurs when vision is introduced. Its principal modeling contribution is cross-modal model merging, which combines separately learned text and multimodal strengths rather than relying only on sequential fine-tuning.
Training uses a synthetic annotation pipeline that generates and filters culturally and linguistically diverse multimodal instructions, avoiding dependence on literal machine translation alone. The mixture covers multilingual visual question answering, captions, OCR, image-grounded generation, and text-and-image translation. The full pretraining corpus is not released; the paper's reproducible contributions center on its annotation, merging, and evaluation recipes.
Phi-4-multimodal extends a compact language backbone to images and audio using modality encoders, projectors, LoRA adapters, and modality-specific routing, allowing several input combinations without destructive interference between learned capabilities.
Microsoft Phi Team
Released: 2025-03-03
Figure 1. Vision and audio encoders, projectors, and modality-specific LoRA routes. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Phi-4-multimodal is a 5.6-billion-parameter multimodal model built around the Phi-4-Mini language model. Separate vision and speech encoders transform non-text inputs, while modality projectors map their representations into the language model's embedding space. Its defining mechanism is a Mixture-of-LoRAs: modality-specific LoRA parameters and routers adapt the shared decoder for vision-language, speech-language, and joint vision-speech inference. The design isolates modality extensions more effectively than fully merging every adaptation into the base weights; the speech and audio LoRA component itself contains 460 million parameters.
The underlying language model uses grouped-query attention, a 200K-token vocabulary, and supports a 128K-token context. Multimodal training first aligns the modality encoders and projectors, then jointly optimizes modality-specific adaptations. The released instruct checkpoint receives supervised fine-tuning followed by direct preference optimization and reinforcement learning from human feedback for instruction following and safety.
Its data recipe builds on the language, vision, and speech research used for Phi-3.5 and Phi-4. Vision supervision covers captioning, OCR, charts, diagrams, document understanding, visual question answering, and multi-image reasoning. Speech data spans automatic speech recognition, speech translation, and summarization across multiple languages and acoustic conditions. The model card provides a data summary rather than a complete sample-level corpus inventory, so undisclosed sources should not be inferred from benchmark coverage.
SigLIP 2 extends sigmoid-loss image-text pretraining into a multilingual, localization-aware representation learner, with NaFlex checkpoints that preserve native aspect ratios and support multiple token budgets from one model.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, et al.
Released: 2025-02-20
Figure 1. Combined contrastive, captioning, masked-prediction, and self-distillation objectives. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
SigLIP 2 retains the dual-encoder Vision Transformer design and pairwise sigmoid image-text objective of SigLIP, but broadens the representation learned by the vision tower. Its unified recipe combines contrastive image-text learning with captioning-based pretraining, masked prediction, self-distillation, and online data curation. These additions improve not only zero-shot classification and retrieval, but also localization and dense tasks such as segmentation, depth estimation, and surface-normal prediction. Models are released at ViT-B, ViT-L, So400m, and ViT-g scales.
Alongside conventional fixed-resolution checkpoints, the NaFlex variants use patch-and-pack processing to retain an image's native aspect ratio and accept different sequence lengths with a single checkpoint. This avoids distortive square resizing and lets users trade visual-token count for inference cost, which is especially useful for documents, screenshots, and OCR-heavy inputs. Standard fixed-resolution models additionally receive a self-distillation stage.
Training uses WebLI, comprising 10 billion images and 12 billion alt-texts across 109 languages. The reported mixture draws 90% of image-text pairs from English pages and 10% from non-English pages, while debiasing filters target representation and association biases involving sensitive attributes. SigLIP 2 is positioned both as a standalone multilingual image-text encoder and as a stronger frozen vision tower for downstream VLMs.
EVEv2 represents a significant advancement in encoder-free vision-language models (VLMs), addressing limitations of previous approaches by introducing a "Divide-and-Conquer" architecture that maximizes scaling efficiency, reduces inter-modality interference, and achieves strong performance with superior data efficiency.
Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, Xinlong Wang
Figure 3. EVEv2.0 architecture: lossless patch embeddings and text tokens enter a unified decoder-only VLM whose attention, feed-forward, and normalization layers use modality-specific weights. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
EVEv2 departs from the traditional encoder-based VLM approach. Instead of relying on a pre-trained vision encoder (like CLIP), it builds visual perception directly within a decoder-only Large Language Model (LLM). Key architectural features include: Divide-and-Conquer: This is the core innovation. Instead of mixing visual and textual information throughout the entire LLM, EVEv2 introduces modality-specific components. This means separate attention matrices (query, key, value), Layer Normalization layers, and Feed-Forward Networks for visual and textual tokens. This reduces interference and allows for more efficient learning. It's a fully sparse, decoder-only architecture. Patch Embedding Layer: A minimalist patch embedding layer is learned from scratch. This avoids the inductive biases of pre-trained vision encoders. It uses two convolutional layers (Conv1 and Conv2) to process image patches. Lossless Encoding: Unlike some encoder-free models that use discrete tokenization (which can lose information), EVEv2 aims for lossless encoding of visual information. LLM Adaptation: The architecture is designed for seamless adaptation to existing LLMs. The paper experiments with Vicuna-7B and Qwen2-7B. Multi-Stage Training: A four-stage training process is used: LLM-guided Pre-aligning: Only the patch embedding layer is trained, using re-captioned web data (EVE-recap-10M). The LLM is frozen. This establishes a basic alignment between visual and textual representations. Vision Perception Learning: Vision layers within the LLM are trained, using progressively larger datasets and image resolutions. The LLM weights are still frozen. Vision-Text Fully alligning: The entire network is update. Supervised Fine-tuning (SFT): The entire model is fine-tuned on question-answering and instruction-following datasets. DenseFusion++: A new, efficient captioning engine is introduced to generate high-quality image-text pairs for training. This is crucial for building strong visual perception from scratch. It leverages multiple vision experts. Data Efficiency: A key focus of the research is demonstrating that EVEv2 can achieve strong performance with less data than comparable encoder-based models, thanks to its efficient architecture.
Qwen2.5-VL represents a significant advancement in the Qwen series of vision-language models, offering improved image recognition, precise object grounding, enhanced text recognition, document parsing, and video comprehension, while also functioning as a visual agent capable of computer and phone use.
Qwen Team
Figure 1. Qwen2.5-VL framework: a vision encoder processes native-resolution images and dynamic-FPS video into variable-length tokens for a Qwen2.5 language-model decoder with multimodal rotary position encoding. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Qwen2.5-VL builds upon its predecessor, Qwen2-VL, with substantial improvements in perception of temporal and spatial scales, as well as a simplified network structure for increased efficiency. World-wide Image Recognition: Expanded recognition capabilities covering a vast array of categories, including landmarks, objects, and even film/TV IPs. Precise Object Grounding: Uses bounding boxes and point-based representations for object localization, with standardized JSON output for coordinates and attributes, enabling hierarchical positioning. Enhanced Text Recognition (OCR): Improved multi-scenario, multi-language, and multi-orientation text recognition and localization, with enhanced information extraction for applications like document processing. Powerful Document Parsing: Introduces "QwenVL HTML" format, leveraging HTML for layout information extraction from documents, magazines, research papers, web pages, and mobile screenshots. Enhanced Video Comprehension: Supports understanding of ultra-long videos (hourly scale) with dynamic frame rate (FPS) training and absolute time encoding. Enables second-level event localization and key point summarization. Visual Agent Capabilities: Can function as a visual agent for computer and phone use, capable of reasoning and dynamically directing tools. Capable of tasks like booking flights. Time and Image Size Perception In the spatial dimension, the model is capable of adapting varying image sizes into tokens and directly represents coordinates by detection boxes. In the temporal dimension, the model can comprehend the pace of time through temporal dimension. Visual Encoder A native dynamic resolution ViT is trained from scratch. Window Attention is used to minimize computational load. The model comes in three sizes (3B, 7B, and 72B parameters), with both base and instruct-tuned versions available. The 72B-Instruct model achieves competitive performance on various benchmarks, excelling in document and diagram understanding. Smaller models also demonstrate strong performance, with the 7B-Instruct model outperforming GPT-4o-mini in several tasks and the 3B model exceeding the performance of the previous Qwen2-VL 7B model. The models is trained with 18 trillion tokens. Future developments aim to further enhance problem-solving, reasoning, and multi-modality integration.
VideoLLaMA3 is a vision-centric multimodal foundation model designed for both image and video understanding, emphasizing a training paradigm and framework that prioritize high-quality image-text data, alongside an adaptable vision encoder and video token compression, to achieve state-of-the-art performance.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, Deli Zhao
Figure 3. VideoLLaMA 3 pipeline with any-resolution vision tokenization and a differential frame pruner that compresses redundant video tokens before language-model processing. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
VideoLLaMA3 introduces a vision-centric approach in both its training paradigm and framework design, focusing on enhancing image and video understanding capabilities. The core architecture incorporates a pre-trained vision encoder (SigLIP), a video compressor (DiffFP), a projector, and a large language model (LLM - Qwen2.5). The model employs a four-stage training process: 1) Vision Encoder Adaptation, where the vision encoder is adapted to accept images of variable resolutions using scene images, document data, and scene text images; 2) Vision-Language Alignment, which jointly tunes the vision encoder, projector, and LLM using large-scale image-text data (including detailed captions, documents, and charts) and a small amount of text-only data; 3) Multi-task Fine-tuning, incorporating image-text data for downstream tasks and general video caption data; and 4) Video-centric Fine-tuning, using general videos, streaming videos, temporally grounded videos, image-only, and text-only data. A key innovation is Any-resolution Vision Tokenization (AVT), which allows the vision encoder to process images and videos of any resolution by replacing fixed positional embeddings with Rotary Position Embedding (RoPE). This enables handling images with variable shapes and minimal information loss. For video inputs, Differential Frame Pruner (DiffFP) acts as a video compressor, reducing the number of vision tokens by comparing the 1-norm distance between temporally consecutive patches in pixel space and pruning redundant patches. This makes video representations more compact and precise. The training data mixture is carefully curated for each stage, emphasizing high-quality image-text data. The Vision Encoder Adaptation stage uses datasets like VL3-Syn7M-short, LLaVA-Pretrain-558k, and document datasets. The Vision-Language Alignment stage expands on this with detailed captions, OCR data, and fine-grained data with bounding boxes. The Multi-task Fine-tuning stage adds question-answering data and general video caption data. Finally, the Video-centric Fine-tuning stage includes general videos, streaming videos, and temporal grounding data. This "vision-centric" approach, prioritizing image understanding as a foundation for video understanding, along with AVT and DiffFP, allows VideoLLaMA3 to achieve strong performance on both image and video benchmarks.
UI-TARS is a native GUI agent model that operates solely on screenshots, performing human-like interactions (keyboard and mouse operations). Unlike frameworks relying on wrapped commercial models (e.g., GPT-4o), UI-TARS is an end-to-end model achieving state-of-the-art (SOTA) performance on 10+ GUI agent benchmarks in perception, grounding, and task execution, significantly outperforming sophisticated frameworks.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, Guang Shi
Figure 4. UI-TARS architecture and capability overview, connecting visual GUI observations and interaction histories to perception, action, system-level reasoning, and learning from prior experience. Source paper, PDF p. 14. Figure notice.
ℹ️ More Information
UI-TARS leverages several key innovations: (1) Enhanced Perception, utilizing a large-scale GUI screenshot dataset for context-aware understanding and precise captioning of UI elements; (2) Unified Action Modeling, standardizing actions into a unified space across platforms and achieving precise grounding through large-scale action traces; (3) System-2 Reasoning, incorporating deliberate reasoning for multi-step decision-making, including task decomposition, reflection, and milestone recognition; and (4) Iterative Training with Reflective Online Traces, addressing the data bottleneck by automatically collecting, filtering, and refining interaction traces on hundreds of virtual machines. The model is trained iteratively and tuned via reflection, continuously learning from mistakes and adapting to new situations with minimal human intervention. The architecture takes screenshots as input and uses a Vision-Language Model (VLM), specifically Qwen-2-VL 7B and 72B, to process visual information and generate actions. The action space is unified across platforms (mobile, desktop, web) and includes actions like click, type, scroll, and drag. Reasoning is infused by generating explicit "thoughts" before each action, inspired by reasoning-and-acting frameworks. These thoughts are generated through a combination of curated GUI tutorials and augmented action traces, incorporating patterns like task decomposition, long-term consistency, milestone recognition, trial and error, and reflection. The training process involves multiple stages, starting with perception enhancement using a curated dataset of GUI screenshots and associated metadata. This dataset supports tasks like element description, dense captioning, state transition captioning, question answering, and set-of-mark prompting. Action modeling is improved by creating a large-scale dataset of action traces and using grounding data to pair element descriptions with spatial coordinates. The model is trained using a combination of supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) with reflection tuning to learn from errors.
MiniMax-01 is a series of large foundation models, including MiniMax-Text-01 and MiniMax-VL-01, that achieve performance comparable to top-tier models (like GPT-4o and Claude-3.5-Sonnet) while offering significantly longer context windows (up to 4 million tokens). It achieves this through a novel architecture incorporating lightning attention (a highly efficient linear attention variant), Mixture of Experts (MoE), and optimized training/inference frameworks.
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, Zijia Wu
Figure 3. MiniMax-Text-01 backbone architecture, interleaving Lightning Attention and softmax-attention transformer blocks with routed mixture-of-experts feed-forward layers. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Hybrid Attention: The core innovation is the hybrid attention mechanism. It primarily uses "lightning attention" (an I/O-aware implementation of TransNormer linear attention) for efficiency. However, to maintain strong retrieval capabilities, it strategically inserts a standard transformer block with softmax attention after every seven transnormer blocks (with lightning attention). This is a key differentiator from purely linear attention models, which often struggle with retrieval tasks. Mixture of Experts (MoE): To scale the model efficiently, MiniMax-01 employs a Mixture of Experts (MoE) architecture in the feed-forward layers. It has a massive 456 billion total parameters, but only 45.9 billion are activated for each token, using 32 experts with a top-2 routing strategy. This allows for a large model capacity without a corresponding increase in computational cost per token. Vision-Language Model (MiniMax-VL-01): The vision-language model (MiniMax-VL-01) builds upon MiniMax-Text-01 by integrating a lightweight Vision Transformer (ViT) module. It uses a dynamic resolution strategy, resizing input images to various sizes (from 336x336 to 2016x2016) and concatenating features from both resized patches and a standard thumbnail. It does not use pooling or downsampling on the visual features, relying instead on the long-context capabilities of the architecture. Demonstrates the viability of linear attention at a massive scale, achieving performance comparable to top-tier models while significantly extending the context window. Long-Context Capability: Supports context inputs of up to 4 million tokens, with strong performance in long-context evaluations. Efficient Training and Inference Framework: Introduces several novel algorithmic and engineering optimizations to handle the hybrid architecture, MoE, and long contexts efficiently. Pre-training: A curated corpus incorporating academic literature, books, web content, and programming code. Vision-Language Pre-training (VL-01): A substantial image-caption dataset (694 million unique pairs) and a dataset of 100 million images with fine-grained descriptions. Vision-Language Instruction Data (VL-01): A diverse instruction-based dataset synthesized from a wide array of image-related tasks. Hybrid Attention: The core fusion method combines the efficiency of lightning attention (linear) with the retrieval capabilities of softmax attention. MoE Routing: The top-2 routing strategy selectively activates experts, enhancing model capacity without increasing compute per token; a global router supports load balancing. Vision-Language Fusion (VL-01): Visual features from the ViT are projected into the LLM embedding space using a two-layer MLP. The raw, high-dimensional visual features are used without pooling or downsampling, leveraging the architecture's long-context capabilities. Varlen Ring Attention and LASP+: These algorithms enable efficient handling of long, variable-length sequences and data packing during training and inference.
MiniCPM-o-2.6 is a powerful 8B parameter multimodal large language model (MLLM) that excels in vision, speech, and multimodal live streaming, achieving performance comparable to GPT-4o in several benchmarks, while maintaining high efficiency for deployment on edge devices.
OpenBMB
Official architecture diagram. MiniCPM-o 2.6 end-to-end omni-modal streaming architecture, using time-division multiplexing to combine visual, audio, and query streams in a shared backbone with streaming speech decoding. Primary source. Figure notice.
Figure 4. MiniCPM-o 4.5 architecture; a later release in this family. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
MiniCPM-o-2.6 employs an end-to-end omni-modal architecture. It integrates several pre-trained components: Vision Encoder: SigLip-400M Audio Encoder: Whisper-medium-300M Text-to-Speech (TTS): ChatTTS-200M Large Language Model (LLM): Qwen2.5-7B. These components are connected and trained end-to-end. A key innovation is the "Omni-modal Live Streaming Mechanism." This involves: Online Modality Encoders/Decoders: The offline encoders and decoders are transformed into online versions to handle streaming inputs and outputs. Time-Division Multiplexing (TDM): A TDM mechanism within the LLM backbone processes omni-modal streams. It divides parallel streams (video, audio) into sequential information within short time slices. Configurable Speech Modeling: A multimodal system prompt (including text and audio prompts) allows for flexible voice configuration during inference, enabling voice cloning and description-based voice creation.
MiniCPM-o 4.5, released on February 3, 2026 (technical report: April 29), evolves this family toward real-time full-duplex omni-modal interaction. It retains streaming vision, speech, and text while improving simultaneous perception and generation; the update belongs here because it extends the same time-multiplexed, end-to-end omni-modal lineage. Paper · Repository
Eagle 2 is a family of vision-language models (VLMs) developed with a data-centric approach, focusing on post-training data strategies to achieve state-of-the-art performance. The models build upon open-source components and prioritize data diversity and quality, using a three-stage training recipe and a tiled mixture of vision encoders (MoVE) architecture, achieving results that match or surpass those of larger, proprietary models.
Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Matthieu Le, De-An Huang, Ilia Karmanov, Lukas Voegtle, Jose M. Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Karan Sapra, Amala Deshmukh, Tuomas Rintamaki, Philipp Fischer, Timo Roman, Tong Lu, Guilin Liu, Zhiding Yu
Figure 11. Eagle 2's tiled mixture of vision encoders, combining SigLIP and ConvNeXt features through dynamic image splitting, feature concatenation, pixel shuffle, and an MLP connector to the LLM. Source paper, PDF p. 7. Figure notice.
Figure 2. Eagle 2.5 architecture; a later release in this family. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Eagle 2 adopts a "diversity first, then quality" data strategy, beginning with a large, diverse pool of over 180 data sources, followed by rigorous filtering and selection. The architecture uses a tiled mixture of vision encoders (MoVE), specifically SigLIP and ConvNeXt-XXLarge, with image tiling to handle high resolutions. Each image tile is encoded by channel-concatenated MoVE. The vision encoder outputs are concatenated and aligned with the LLM (Qwen2.5) via a simple MLP connector. A three-stage training recipe is used: Stage 1 trains the connector to align modalities; Stage 1.5 trains the full model on a large, diverse dataset; and Stage 2 fine-tunes on a high-quality instruction-tuning dataset. Crucially, all available visual instruction data is used in Stage 1.5, not just captioning/knowledge data. Balanced data packing addresses limitations in existing open-source frameworks. The core contribution is the detailed data strategy. This involves: (1) Data Collection: Building a highly diverse data pool (180+ sources) through both passive gathering (monitoring arXiv and Hugging Face) and proactive searching (addressing "bucket effect" via error analysis). (2) Data Filtering: Removing low-quality samples based on criteria like mismatched question-answer pairs, irrelevant image-question pairs, repeated text, and numeric formatting issues. (3) Data Selection: Choosing optimal subsets based on data source diversity, distribution, and K-means clustering on SSCD image embeddings to ensure balance across types (especially useful for chart data, etc.). (4) Data Augmentation: Mining information from input images through techniques like Chain-of-Thought (CoT) explanation generation, rule-based QA generation, and expanding short answers into longer ones. (5) Data Formating: remove unnecessary decorations. Training uses a three-stage approach: Stage 1: Aligns language and image modalities by training the MLP connector. Stage 1.5: Trains the full model using a large-scale, diverse dataset (21.6M samples). All available visual instruction data is used here, unlike common two-stage approaches, leading to substantial improvements. Stage 2: Fine-tunes the full model on a carefully curated, high-quality visual instruction tuning dataset (4.6M samples). The model is trained with AdamW. Eagle 2 demonstrates strong performance across a wide range of multimodal benchmarks, matching or outperforming frontier open-source and some closed-source VLMs. Eagle 2.5 (April 16, 2025 model release; April 21 paper) extends the family to long videos and large images. Its 8B checkpoint connects a SigLIP2 visual encoder to Qwen2.5-7B through an MLP. Image Area Preservation retains image geometry during tiling; Automatic Degrade Sampling budgets visual content while preserving text. Progressive mixed post-training expands context from 32K to 128K, using public video/document data and Eagle-Video-110K with clip-level and story-level annotations. The supplemental paper figure above shows the tiling and sampling pathway. Paper · Model
Sa2VA is a unified model for dense grounded understanding of both images and videos, integrating the SAM-2 video segmentation model with the LLaVA vision-language model. It supports a wide array of image and video tasks, like referring segmentation and conversation, by treating all inputs (text, images, videos) as tokens in a shared LLM space, generating instruction tokens that guide SAM-2 for precise mask production.
Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, Ming-Hsuan Yang
Figure 2. Sa2VA model: text, prompts, images, and videos are encoded for an LLM, whose segmentation token is combined with SAM 2 features to decode image or video masks. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
Sa2VA leverages a pre-trained LLaVA-like model (containing a visual encoder, visual projection layer, and LLM) and appends SAM-2 alongside it. Crucially, it uses a decoupled design, where SAM-2's decoder and memory module are frozen. This preserves SAM-2's perception and tracking capabilities and allows Sa2VA to be a plug-and-play module, updatable with newer MLLMs. The connection between the LLM and SAM-2 is a special [SEG] token. The LLM generates this token, and its hidden states act as a spatial-temporal prompt for SAM-2's decoder, which produces segmentation masks. The model is trained end-to-end, demonstrating scalability. The training uses a unified instruction-tuning format for various tasks: referring segmentation, visual question answering (VQA), and grounded conversation generation (GCG) for both images and videos. It treats all images, videos and prompts as visual tokens. A key aspect is the co-training with multiple datasets, including image and video data. The authors introduce Ref-SAV, an auto-labeled dataset with over 72,000 object expressions in complex video scenes, and manually validate 2,000 video objects in Ref-SAV for benchmarking referring video object segmentation. A simple mask tracking method re-utilizes SAM-2's knowledge. The model formulates all tasks as a single instruction-tuning process. Datasets used for co-training are: LLAVA 1.5 (665K), RefCOCO (17K), RefCOCO+ (17K), RefCOCOg (22K), Grand-f (214K), ChatUniVi (100K). Ref-YTVOS (3.5K), MeVIS (0.6K), ReVOS (1.7K) and Ref-SAV (37K).
VideoChat-Flash is a system designed for handling long-form video content in multimodal large language models (MLLMs). It introduces a Hierarchical visual token Compression (HiCo) method to reduce computational load while preserving essential details, and uses a multi-stage learning approach with a new long-video dataset (LongVid) to achieve state-of-the-art performance on both long and short video benchmarks.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, Limin Wang
Figure 3. VideoChat-Flash framework with hierarchical video-token compression: shared encoders and connectors first compress clips, then the LLM performs video-level compression for long-context inference. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Hierarchical visual token Compression (HiCo): This is the core innovation. It compresses video information at two levels: Clip-level Compression: The video is divided into clips. A visual encoder (UMT-L) processes each clip, and a compressor (token merging with MLP) reduces the number of visual tokens. This exploits inter-frame redundancy. Video-level Compression: During the LLM (Qwen2-7B) interaction, visual tokens are further reduced using a progressive visual dropout strategy. This leverages the idea that the LLM focuses on the entire context at shallow layers and specific details at deeper layers. It combines uniform dropout (shallow layers) and text-guided selection (deep layers). Visual Encoder: UMT-L@224 (a video encoder, shown to be more efficient than image encoders like SigLIP). Visual-Language Connector: A compressor (token merging) followed by an MLP projection. Large Language Model (LLM): Qwen2-7B. Multi-stage Short-to-Long Learning: This is a crucial training strategy: Stage 1: Video-Language Alignment: Train the compressor and MLP with image-text and short video-text pairs (0.5M each). Stage 2: Short Video Pre-training: Enhance visual understanding with more images (3.5M) and short videos (2.5M). Stage 3: Joint Short & Long Video Instruction Tuning: Fine-tune on a mix of images (1.1M), short videos (1.7M), and long videos (0.7M) with instruction-following data. Stage 4: Efficient High-Resolution Post-finetuning: Adapt to higher resolutions (224 to 448) by fine-tuning the video encoder on a subset (25%) of Stage 3 data.Dynamic Video Sampling: Uses a dual sampling strategy: dense sampling (15 fps) for short videos (capturing motion) and sparse sampling (1 fps) for long videos (capturing events). Timestamp-aware Prompt: Uses a text prompt to provide timestamp information to the model. LongVid: A new large-scale long video instruction-tuning dataset introduced in the paper. It contains 114,228 long videos and 3,444,849 question-answer pairs across five task types. It leverages existing datasets (Ego4D, HowTo100M, HD-Vila, MiraData) and generates dense event labels. Mixed Training Data: Uses a combination of short and long videos during training. NIAH (Needle In A video Haystack). A newly created dataset for testing models capabilities for understanding long contexts.
OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference
OmniVLM is a sub-billion-parameter vision-language model designed for efficient on-device inference, featuring a token compression mechanism that reduces visual token sequence length from 729 to 81, drastically cutting computational overhead while maintaining visual-semantic fidelity. It uses Qwen2.5-0.5B-Instruct model, Google's SigLIP-400M.
Wei Chen, Zhiyuan Li, Shuo Xin
Figure 1. OmniVLM architecture: a vision transformer feeds a reshape-based projector that compresses image tokens before they join text tokens in the Qwen2.5-0.5B-Instruct language model. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
OmniVLM addresses the challenges of deploying vision-language models (VLMs) on resource-constrained edge devices. It achieves this through a novel token compression mechanism and a multi-stage training pipeline. The core innovation is the image token compression, which transforms the embedding dimensions from [batch_size, 729, hidden_size] to [batch_size, 81, hidden_size] within the projection layer. This 9x reduction in token count is achieved through reshaping, chosen after empirical comparison against convolution-based methods. The model architecture (Figure 1) builds upon the LLaVA framework, employing Google's SigLIP-400M as the vision encoder, Qwen2.5-0.5B-Instruct as the base language model, and a Multi-Layer Perceptron (MLP) as the projection layer. The training pipeline consists of three stages: (1) Pretraining on large-scale image-caption pairs (primarily from the LLaVA pretraining dataset) to learn visual-linguistic alignments, training only the projection layer; (2) Supervised Fine-Tuning (SFT) on a mix of datasets (LLaVA, UnimmChat, and internal data) to improve contextual understanding and conversational coherence, training the projector and LLM while freezing the vision encoder; and (3) Minimal-Edit Direct Preference Optimization (DPO), using a teacher model to create minimally edited corrections to the base model's outputs, forming chosen-rejected pairs for preference learning, again freezing the vision encoder and training the projector and LLM. The DPO process leverages GPT-4V to generate synthetic training pairs. Extensive experiments show that the 81-token configuration provides the optimal balance between computational efficiency and model performance. OmniVLM outperforms nanoLLAVA on benchmarks like ScienceQA, POPE, and MMMU, demonstrating improved reasoning, multimodal comprehension, and generalization. Crucially, it achieves significantly faster inference speeds (9.1x faster time-to-first-token and 1.5x higher decoding speed compared to nanoLLAVA on a laptop, and 8x faster TTFT on a mobile device), making it suitable for deployment on edge devices like smartphones and laptops.
Apollo is a family of Large Multimodal Models (LMMs) designed for video understanding. Its study introduces "Scaling Consistency" and examines video sampling, architecture, data composition, and training schedules; at publication, the authors reported that Apollo-3B outperformed most compared 7B models.
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia
Figure 8. Apollo architecture: image and video encoders process N-frame clips, interpolated features are concatenated channel-wise, and a Perceiver resampler produces a fixed token set for the language model. Source paper, PDF p. 21. Figure notice.
ℹ️ More Information
Apollo leverages the Qwen2.5 series of Large Language Models (LLMs) with 1.5B, 3B, and 7B parameters. The key architectural innovation is the combination of a SigLIP-SO400M image encoder and an InternVideo2 video encoder. Features from both encoders are interpolated and concatenated channel-wise before being fed into a Perceiver Resampler, which outputs 32 tokens per frame. This combination was empirically found to be superior to other encoder choices. The model uses a 3-stage training approach. Critically, the paper introduces the concept of "Scaling Consistency," demonstrating that design decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. This allows for more efficient experimentation. The paper also advocates for frames-per-second (fps) sampling during training, as opposed to uniform frame sampling, and demonstrates its superiority. The optimal number of tokens is 8-32 per frame. It also includes a curated benchmark, ApolloBench, that reduces evaluation time by 41x compared to existing benchmarks while maintaining high correlation and focusing on temporal reasoning and perception. The exploration also includes Token Resampling showing that Perceiver resampling has a good performace. Token Integration is also discussed: Adding tokens (text, learned, etc.) between the video tokens derived from different frames or clips is sufficient for efficient token integration. Training Stages is also disscussed, concluding that progressively unfreezing the different components in different stages leads to superior model training dynamics. Finally, training the Video Encoder is discussed. The paper concludes that Finetuning video encoders on only video data further improves overall performance, especially on reasoning and domain-specific tasks. Data Composition is also studied. It concludes that Data mixture matters, and including a moderate amount of text data and maintaining a slight video-heavy mix leads to optimal performance.
DeepSeek-VL2 is an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL, by incorporating a dynamic tiling vision encoding strategy for high-resolution images and leveraging DeepSeekMoE models with Multi-head Latent Attention for efficient inference. It is trained on a large vision-language dataset, shows top performance in tasks.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, and et al.
Figure 2. DeepSeek-VL2 architecture: dynamic image tiling feeds a vision encoder and vision-language adapter whose image tokens join text tokens in a DeepSeek-MoE language model. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
DeepSeek-VL2 builds upon a LLaVA-style architecture. it consists of three core modules: (1) a vision encoder, (2) a vision-language adaptor, and (3) a Mixture-of-Experts language model. It introduces two major enhancements: a dynamic tiling strategy and it uses DeepSeekMOE language model that has Multi-head Latent Attention. The dynamic tiling strategy addresses the limitations of fixed-resolution encoders by splitting high-resolution images into tiles. It uses a single SigLIP-SO400M-384 vision encoder. A set of candidate resolutions CR = {(m· 384, η · 384) | m∈ N, n ∈ N, 1 ≤ m, n,mn ≤ 9} is defined, representing different aspect ratios. For an input image, the optimal resolution from CR that minimizes padding is selected. The resized image is then divided into m₁ × n₁ local tiles of 384 × 384 pixels, plus one global thumbnail tile. The SigLIP-SO400M-384 processes all (1 + m¡ × n₁) tiles, yielding 729 visual embeddings (27x27) of 1152 dimensions per tile. Dynamic tiling is disabled for multiple (>2) images for efficiency. A 2x2 pixel shuffle compresses each tile's visual tokens to 14x14 (196 tokens). Special tokens are added: 14 <tile_newline> tokens at the end of each row in the global thumbnail (total 210 tokens); m₁ · 14 <tile_newline> tokens at the end of the final column of the local tiles; and a <view_separator> token between the global thumbnail and local tiles. The total visual sequence length is 210 + 1 + m₁ · 14 × (nį · 14 + 1). This sequence is projected into the LLM's embedding space by a two-layer MLP. The language model utilizes DeepSeekMoE, featuring Multi-head Latent Attention (MLA) to compress the Key-Value (KV) cache, improving inference speed and throughput. The MoE architecture further enhances efficiency. A global bias term is used during MoE training for load balancing. DeepSeek-VL2 comes in three variants (Tiny, Small, and Base) with 1.0B, 2.8B, and 4.5B activated parameters, respectively. The training data is constructed in three stages: (1) VL alignment, (2) VL pretraining, and (3) supervised fine-tuning (SFT). The alignment stage uses ShareGPT4V (1.2M samples). Pretraining data combines VL and text-only data (70/30 ratio), including interleaved image-text data (WIT, WikiHow, OBELICS, Wanjuan, and in-house data), image captioning data (various open-source datasets with quality enhancements and filtering), OCR data (LaTeX OCR, 12M RenderedText, and in-house data), general VQA data, table/chart/document understanding data (PubTabNet, FinTabNet, Docmatix), web-to-code and plot-to-Python data (Websight, and Python plots), QA with visual prompts, visual grounding data and grounded conversation data. SFT data includes enhanced general visual question-answering data, cleaned OCR and document understanding data, enhanced table and chart understanding data, improved reasoning/logic/math data, textbook/academic questions, and expanded web-to-code and plot-to-Python data, visual grounding data, grounded conversation data. Text only datasets were used during SFT stage. The training methodology involves a three-stage pipeline. Stage 1 trains the vision encoder and vision-language adaptor MLP, keeping the language model fixed, using image-text paired data. Stage 2 performs vision-language pre-training with all parameters unlocked, using ~800B image-text tokens. Stage 3 conducts supervised fine-tuning. Visual understanding is emphasized, and the loss is computed only on text tokens. Unlike previous work, the fixed-resolution vision encoder is adapted during training.
Maya is an open-source Multimodal Multilingual Vision Language Model (mVLM) designed to address the limitations of current VLMs in handling low-resource languages and diverse cultural contexts, achieved by creating a new multilingual image-text pretraining dataset, performing toxicity analysis and mitigation, and fine-tuning for enhanced cultural and linguistic comprehension.
Nahid Alam, Karthik Reddy Kanjula, Bala Krishna S Vegesna, S M Iftekhar Uddin, Drishti Sharma, Abhipsha Das, Shayekh Bin Islam, Surya Guthikonda, Timothy Chung, Anthony Susevski, Ryan Sze-Yin Chan, Roshan Santhosh, Snegha A, Chen Liu, Isha Chaturvedi, Ashvanth.S, Snehanshu Mukherjee, Alham Fikri Aji
Figure 7. Maya's LLaVA-derived architecture, projecting multilingual SigLIP vision features into the embedding space of a multilingual language model for instruction following. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
Architecture: Maya builds upon the LLaVA 1.5 framework. It uses the Aya-23 8B model as its Large Language Model (LLM) due to Aya's strong multilingual capabilities (trained on 23 languages). Critically, it replaces the CLIP vision encoder used in LLaVA with SigLIP. This is motivated by SigLIP's superior performance, multilingual support, and ability to handle variable-length image patches (allowing for more flexible input sizes). The visual features from SigLIP (Zv = g(Xv)) are passed through a trainable projection matrix (W, a 2-layer MLP with GELU activation) to align them with the LLM's embedding space, producing visual features Hv. The architecture is fairly standard for this type of model, concatenating visual and textual features for input to the LLM. The training process involves two main phases: pretraining and finetuning. Pretraining: The model is pretrained on a newly created multilingual image-text dataset. This dataset is derived from the English-only LLaVA pretraining dataset (558k image-text pairs) and translated into seven additional languages (Chinese, French, Spanish, Russian, Hindi, Japanese, and Arabic) using a sophisticated translation pipeline. This pipeline uses the Aya 35B model, optimized prompt engineering (determined empirically using BLEU and N-gram scores), and a batch processing approach with quality checks. Crucially, this dataset undergoes toxicity filtering. LLaVAGuard and Toxic-BERT are used to identify and remove toxic image-caption pairs, creating a "toxicity-free" version of the dataset (removing 7,531 toxic images). The pretraining uses a learning rate of 1e-3 and a cosine scheduler. Only the projection matrix is trained during pretraining. Finetuning: The pretrained model is then instruction-tuned using the PALO 150K instruction-tuning dataset (which covers 10 languages). Full finetuning is performed (as opposed to LoRA), with frozen vision encoder and LLM. The core alignment technique is the trainable projection matrix (the 2-layer MLP) that maps the SigLIP visual features into the embedding space of the Aya-23 LLM. This is a simple but effective method, common in many VLMs. The paper explicitly states they did not use more complex alignment techniques like gated soft-attention (Flamingo) or Q-Former (BLIP-2) in this phase, reserving those for future work. Pretraining Dataset: A new multilingual dataset created by translating and filtering the LLaVA pretraining dataset. This dataset is a key contribution of the paper. The translation process and toxicity filtering are described in detail. Instruction Tuning Dataset: PALO 150K instruction-tuning dataset. Evaluation Datasets: PALO multilingual evalution, VizWiz, GQA, ScienceQA, TextVQA, POPE, MMBench, MM-Vet, MME. Multilingual Image-Text Pretraining Dataset: A new dataset of 558,000 images in eight languages. Toxicity Analysis and Mitigation: A thorough analysis of toxicity in the original LLaVA dataset and the creation of a toxicity-free version. This is a significant and novel aspect. Multilingual Model: A model (Maya) that shows improved performance in understanding cultural and linguistic nuances, especially in comparison to models trained primarily on English data. The results show that Maya performs comparably to or better than models of similar size (LLaVA-7B) and often approaches the performance of larger models (PALO-13B) on multilingual benchmarks. The toxicity filtering has a minimal impact on overall performance, suggesting that valuable information isn't lost by removing toxic content. The paper includes both quantitative benchmark results and qualitative examples demonstrating the model's capabilities.
InternVL 2.5: Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
InternVL 2.5 is an advanced Multimodal Large Language Model (MLLM) series that builds upon InternVL 2.0, maintaining its core architecture while enhancing training and testing strategies, and data quality, to rival leading commercial models like GPT-4o and Claude-3.5-Sonnet.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang
Figure 2. InternVL 2.5's ViT-MLP-LLM architecture with pixel-unshuffle visual-token compression. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
InternVL 2.5 retains the "ViT-MLP-LLM" architecture of its predecessors, combining a pre-trained InternViT (either InternViT-6B or InternViT-300M) with LLMs of varying sizes (InternLM 2.5, Qwen 2.5) via a 2-layer MLP projector. A key feature is the pixel unshuffle operation, reducing visual tokens from 1024 to 256 per 448x448 image tile, improving scalability for high-resolution processing. The model architecture supports dynamic resolution, adapting to image aspect ratios by dividing images into 448x448 tiles. Crucially, InternVL 2.0 and 2.5 incorporate multi-image and video data, in addition to single-image and text-only data. The training strategy involves a three-stage pipeline: (1) MLP warmup, where only the MLP projector is trained, (2) optional ViT incremental learning, where the vision encoder and MLP are trained to enhance visual feature extraction, particularly for domains rare in web-scale data, and (3) full model instruction tuning, where the entire model is trained on high-quality multimodal instruction datasets. A progressive scaling strategy is employed, starting with smaller LLMs and scaling up, allowing for efficient alignment of the vision encoder with larger LLMs. Training enhancements include random JPEG compression (for robustness to real-world image quality) and loss reweighting (to balance contributions from responses of different lengths). Data organization is optimized using parameters like nmax (maximum tile number) and a repeat factor (r) to control data sampling frequency. A data-packing strategy concatenates multiple samples into longer sequences to improve GPU utilization. A significant contribution is a data filtering pipeline to remove low-quality samples, particularly those with repetitive patterns, mitigating the risk of repetitive generation, a common issue in MLLMs. The data mixture includes a wide range of tasks (captioning, general QA, mathematics, charts, OCR, etc.) and modalities (single-image, multi-image, video, text). The model was evaluated comprehensively on diverse benchmarks including multi-discipline reasoning (MMMU, MMMU-Pro), document understanding (DocVQA), multi-image/video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing.
PaliGemma 2 is an upgraded family of open Vision-Language Models (VLMs) based on Gemma 2 language models, combined with the SigLIP-So400m vision encoder. It offers models in three sizes (3B, 10B, 28B) and three resolutions (224px², 448px², 896px²), trained in multiple stages for broad knowledge transfer. PaliGemma 2 achieves state-of-the-art results on various tasks, including OCR-related challenges like table/molecular/music score recognition, and long-form captioning.
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer and Xiaohua Zhai
Figure 1. PaliGemma 2 processes variable-resolution images with SigLIP, a linear projector, and Gemma 2. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
PaliGemma 2 closely follows the architecture of its predecessor, PaliGemma. It uses a pre-trained SigLIP-So400m vision encoder. The embeddings from this encoder are mapped to the input space of the Gemma 2 language model using a linear projection. The combined visual and text embeddings are then fed into the Gemma 2 model, which autoregressively generates the output. The model comes in three size variants (2B, 9B, and 27B parameters in the Gemma 2 component, corresponding to 3B, 10B, and 28B total parameters) and is trained at three resolutions (224x224, 448x448, and 896x896 pixels). This allows for analysis of the interplay between model size, resolution, and transfer performance. The input image gets concatenated with the input text tokes and Gemma 2 autoregressively completes this prefix with an answer. PaliGemma 2's training follows a three-stage approach, similar to the original PaliGemma: Stage 1: The pre-trained SigLIP-So400m and Gemma 2 checkpoints are combined and trained jointly on a multimodal task mixture of 1 billion examples. The image resolution is 224px². Stage 2: Training continues for 50 million examples at 448px² resolution, then for 10 million examples at 896px². Tasks benefiting from higher resolution are upweighted. Stage 3: Fine-tuning the checkpoints from stage 1 or 2 on the target tasks. The training data mixture includes captioning, grounded captioning, OCR, visual question answering (VQA), detection, and instance segmentation. Notably, the training data relies heavily on machine-generated labels from publicly available specialist models, avoiding the use of large commercial VLMs for label generation. Gemma 2 Language Models: The core upgrade is the use of the more recent and capable Gemma 2 family of language models, replacing the original Gemma model in PaliGemma. Resolution and Model Size Scaling: PaliGemma 2 systematically explores the impact of both image resolution and language model size on transfer performance. This is a key contribution, as most prior work did not jointly study these factors with consistent training recipes.
ShowUI is a lightweight 2B vision-language-action model built on Qwen2-VL for screenshot grounding and GUI navigation. It reduces redundant screenshot tokens through UI-guided selection and unifies grounding and navigation examples as interleaved vision-language-action streams.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, Mike Zheng Shou
Released: 2024-11-26
Figure 3. UI-guided visual-token selection and interleaved action history. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
ShowUI adapts Qwen2-VL-2B into an end-to-end GUI agent that predicts textual responses and executable actions from pixels. Its UI-Guided Visual Token Selection models screenshot patches as a connected graph, identifies visually redundant regions, and retains representative tokens during selected self-attention blocks. This exploits the repeated colors and structures common in interfaces: the paper reports removing about 33% of visual tokens during training and a 1.4× speedup without sacrificing grounding accuracy.
Training uses Interleaved Vision-Language-Action Streaming to express multiple GUI tasks in one causal format. For navigation, screenshots, instructions, and actions are arranged as temporal trajectories so the model can use visual-action history. For grounding, multiple query-action pairs can share a screenshot, improving data reuse and training efficiency. A resampling strategy prevents abundant task or interface categories from overwhelming smaller ones. The resulting model is trained on approximately 256K curated examples.
The instruction mixture covers web, mobile, and desktop interfaces and combines screenshot grounding with sequential navigation data. It incorporates carefully filtered public GUI sources and recaptioned annotations, then evaluates transfer on ScreenSpot-style grounding and on Mind2Web, AITW, and MiniWob navigation environments. The repository releases model weights, datasets, training and evaluation code, and the implementation of UI-guided token selection.
SmolVLM is a 2B parameter vision-language model (VLM) that achieves state-of-the-art performance for its memory footprint, offering a small, fast, and memory-efficient solution for multimodal tasks. It is fully open-source, with all model checkpoints, datasets, training recipes, and tools released under the Apache 2.0 license, enabling local deployment, reduced inference costs, and user customization.
Andres Marafioti, Merve Noyan, Miquel Farré, Elie Bakouch, Pedro Cuenca
Official architecture diagram. SmolVLM's SigLIP vision encoder, aggressive pixel-shuffle compression, and SmolLM2 language backbone. Primary source. Figure notice.
ℹ️ More Information
SmolVLM builds upon the architecture of Idefics3, leveraging a similar implementation in transformers but with key differences to enhance efficiency. It replaces the Llama 3.1 8B language backbone with the smaller SmolLM2 1.7B model. A more aggressive image compression strategy is employed, using a pixel shuffle strategy that reduces visual information by a factor of 9 (compared to 4x in Idefics3). This allows for 384x384 patches, and a shape-optimized SigLIP is used as the vision backbone with 14x14 inner patches. The model demonstrates superior memory usage compared to other VLMs in transformers, enabling efficient on-device inference. For instance, encoding a single image and prompt requires only 1.2k tokens, significantly less than models like Qwen2-VL. This efficiency translates to faster prefill and generation throughputs. SmolVLM achieves strong performance on benchmarks such as MMMU, MathVista, MMStar, DocVQA, and TextVQA. It also shows promising results in basic video analysis, leveraging its long context capabilities. Training involved extending the context window of SmolLM2 to 16k tokens using techniques like RoPE base value adjustment and fine-tuning on a mixture of long- and short-context datasets. A curated training dataset, largely based on The Cauldron and Docmatix, was used for the VLM training. Checkpoint selection was based on a weighted metric across multiple vision-language benchmarks. The model is integrated with VLMEvalKit for easy evaluation, and it can be readily used and fine-tuned with the transformers library. TRL integration allows for applying Direct Preference Optimization (DPO). A notebook is provided for fine-tuning on VQAv2, with options for LoRA, QLoRA, or full fine-tuning, even within the constraints of consumer GPUs.
Released on February 20, 2025, SmolVLM2 extends the same architecture from still images to multi-image and video input while preserving edge-oriented 256M, 500M, and 2.2B scales. It adds temporal sampling and video instruction tuning to the Idefics3-derived visual-token pipeline rather than introducing a separate family. Announcement
AIMv2 is a family of generalist vision encoders that autoregressively generates both image patches and text tokens, achieving state-of-the-art performance in multimodal image understanding and strong results in vision benchmarks like localization, grounding, and classification, demonstrating scalability and efficiency.
Enrico Fini, Mustafa Shukor, David Haldimann, Sai Aitharaju, Alexander Toshev, Marcin Eichner, Moin Nabi, Xiujun Li, Philipp Dufter, Michal Klein, Victor G. Turrisi da Costa, Louis Béthune, Zhe Gan, Alaaeldin El-Nouby
Figure 1. AIMv2's prefix-attention vision encoder and joint autoregressive multimodal decoder. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
AIMv2 (Autoregressive Image Models v2) introduces a novel pre-training method for large-scale vision encoders that extends autoregressive pre-training to a multimodal setting, encompassing both images and text. The core architecture pairs a Vision Transformer (ViT) encoder with a causal multimodal decoder. The vision encoder processes raw image patches (using prefix attention), while the multimodal decoder autoregressively generates both image patches (using pixel MSE loss) and text tokens (using cross-entropy loss). Crucially, image patches and text tokens are treated as a single, unified sequence. This allows the model to learn a joint representation of visual and textual information. The image is always prepended to the beginning of text sequence. The training process is streamlined and efficient. It resembles that of AIM and LLMs, relying solely on the autoregressive objective. There are no specialized inter-batch communication methods or excessively large batch sizes are required. This contrasts with contrastive methods (e.g., CLIP, SigLIP), which are often more challenging to train and scale. The training data consists of a mixture of publicly available (DFN-2B, COYO) and proprietary datasets (HQITP), comprising both alt-text and synthetic captions. AIMv2 demonstrates strong scaling properties, consistently improving performance with increased data or model parameters. The model family includes variants ranging from 300 million to 3 billion parameters. A key optimization is the use of prefix attention within the vision encoder, enabling bidirectional attention during inference without fine-tuning. Other architectural choices include the incorporation of SwiGLU and RMSNorm, inspired by recent successes in language modeling. AIMv2 excels in a variety of tasks. It performs favorably on multimodal understanding benchmarks compared to state-of-the-art vision-language pre-trained methods . It also exhibits strong performance on open-vocabulary object detection and referring expression comprehension, surpassing DINOv2. Additionally, it achieves impressive recognition performance with a frozen trunk. The model supports native image resolution and adaptation to zero-shot recognition, demonstrating its flexibility. Post-training strategies, including high-resolution adaptation, further enhance the model's capabilities. Ablation studies demonstrate the importance of joint image and text modeling, validate design choices, and explore scaling characteristics.
LLaVA-CoT is a novel Vision-Language Model (VLM) designed to perform autonomous, multi-stage reasoning, enabling it to tackle complex visual question-answering tasks by independently engaging in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation.
Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, Li Yuan
Figure 4. LLaVA-CoT's Best-of-N, stage-wise beam-search, and stage-wise retracing inference procedures. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
LLaVA-CoT builds upon the Llama-3.2-Vision model and introduces a structured, four-stage reasoning process: Summary (briefly outlines the task), Caption (describes relevant image parts), Reasoning (detailed analysis), and Conclusion (provides the final answer). Each stage is marked with specific tags (
, , , ) to maintain clarity. Unlike traditional Chain-of-Thought (CoT) prompting, LLaVA-CoT promotes structured thinking by first organizing the problem and known information, then performing detailed reasoning, and finally deriving a conclusion. The model is trained on the newly compiled LLaVA-CoT-100k dataset. This dataset integrates samples from various visual question answering sources and providing structured reasoning instructions. The dataset contains 99k image and Question answer pairs using GPT-4o to provide details. Data is gathered from general VQA datasets (ShareGPT4V, ChartQA, A-OKVQA, DocVQA, PISC, CLEVR) and Science targeted VQA (AI2D, GeoQA+, ScienceQA, CLEVR-Math). The paper also proposes a novel inference-time stage-level beam search method. This method generates multiple candidate results at each stage of the reasoning process, selecting the best to continue, improving performance and scalability. This contrasts with traditional best-of-N or sentence-level beam search. The entire model is trained using the Supervised-Fine Tuning.
LLM2CLIP is a fine-tuning approach which integrates Large Language Models (LLMs) with pre-trained CLIP visual encoders. It improves the model by using the LLM's ability to proccess and understant long captions, open-world knowledge.
Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Liang Hu, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, Lili Qiu
Figure 1. LLM2CLIP fine-tunes an LLM for caption discrimination before using it to train stronger CLIP representations. Source paper, PDF p. 1. Figure notice.
ℹ️ More Information
LLM2CLIP is fine-tuning approach. It integrates LLM (Large Language Models) to already pretrained CLIP visual encoders. The main problem which is tried to be solved is that; LLM's text understanding capability is not reflected on CLIP models. The authors highlight that directly incorporating LLMs into CLIP often fails due to the poor separability of LLM output features. To tackle this, they introduce a two-stage approach. Stage 1: Caption Contrastive (CC) Fine-tuning: The LLM (specifically Llama-3 8B) is fine-tuned using a contrastive learning framework on a dataset of image captions (CC3M). This stage doesn't train for autoregressive capabilities, instead, it is transforming the causal attention to bidirectional, to function it as an encoder. This stage aims to improve the discriminative power of the LLM's output space, making it easier to distinguish between different captions, using supervised SimCSE loss. Stage 2: CLIP Vision Encoder Fine-tuning: The pre-trained CLIP visual encoder is fine-tuned using the CC-fine-tuned LLM, now acting as a "super" text encoder. The LLM's gradients are frozen during this stage to preserve its acquired knowledge and reduce computational cost. Learnable adapters (linear layers) are added after the LLM to facilitate alignment with the CLIP visual encoder. Instead of the typical image-text contrastive loss, a caption-to-caption contrastive framework is used during LLM fine-tuning. This forces the LLM to produce distinct representations for different captions describing the same image. It uses Supervised SimCSE. Makes the model encoder. Freezing the LLM during CLIP fine-tuning is crucial for efficiency and preserving the LLM's knowledge. These adapters bridge the gap between the frozen LLM and the CLIP visual encoder. The method is surprisingly efficient, requiring only a small amount of open-source data (15M or even 3M image-text pairs) and a single epoch of training in some cases. It leverages LoRA (Low-Rank Adaptation) for efficient fine-tuning. LLM2CLIP can effectively leverage dense captions (detailed image descriptions), a known limitation of standard CLIP. Uses ShareCaptioner-modified CC-3M (for CC fine-tuning), Wikitext-103, and a combination of CC-3M, CC-12M, YFCC-15M, and Recaption-1B for CLIP fine-tuning. The paper demonstrates that, after fine-tuning of the output space of the LLM, using LLM has a significant impact and it substantially improves the performance on downstream tasks.
Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding
Tarsier2 is a state-of-the-art large vision-language model (LVLM) that excels in generating detailed and accurate video descriptions and demonstrates superior general video understanding capabilities. It scales pre-training data, performs fine-grained temporal alignment during supervised fine-tuning, and uses model-based sampling with Direct Preference Optimization (DPO) to improve performance, outperforming models like GPT-4o and Gemini 1.5 Pro.
Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, Yuan Lin
Architecture figure: The paper contains capability, dataset, DPO-construction, and benchmark figures but inherits Qwen2-VL and provides no architecture overview; a training diagram would be misleading.
ℹ️ More Information
Tarsier2 utilizes a straightforward architecture consisting of a vision encoder, a vision adaptor, and a large language model (LLM), specifically building upon Qwen2-VL. The model undergoes a three-stage training process: pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL) using Direct Preference Optimization (DPO). A key improvement over its predecessor, Tarsier, is the significant expansion of the pre-training dataset from 11 million to 40 million video-text pairs. This expansion includes the meticulous collection and filtering of 11 million commentary videos (explanations and analyses of movies and TV shows), providing rich contextual information. During the SFT stage, Tarsier2 is trained on a dataset containing 150K instances, each with a detailed video description and specific frame annotations corresponding to each described event. This fine-grained temporal alignment provides supervision that improves accuracy and reduces hallucinations compared to traditional video-caption alignment. The SFT phase is conducted in two steps. The initial step is frame to event allignment. Then, the model's output to make a more human-like style. The final training stage employs DPO with automatically generated preference data. Negative samples are created by corrupting videos (clip-switching, clip-reversing, clip-cropping, and down-sampling), and a preference data filtering method (using AutoDQ) ensures high-quality pairs. Tarsier2 achieves state-of-the-art results on 15 public benchmarks, demonstrating its versatility across tasks such as video question-answering, video grounding, hallucination tests, and embodied question-answering. A recaptioning dataset, Tarsier2-Recap-585K, is also released.
Janus decouples the visual pathways used for understanding and generation while sharing one autoregressive transformer; Janus-Pro scales and improves that architecture through a revised curriculum and substantially larger data mixtures.
DeepSeek-AI
Released: 2024-10-17
Figure 3. Janus-Pro decouples visual understanding and generation encoders around a shared autoregressive transformer. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Janus uses a unified autoregressive transformer but separate visual encoders for understanding and generation. A SigLIP encoder and understanding adaptor supply semantic features, while a VQ tokenizer and generation adaptor turn images into discrete codes. This separation prevents the fine-grained reconstruction demands of image generation from compromising the high-level representations needed for comprehension.
Janus-Pro, released in January 2025, retains that split and improves it through a three-stage curriculum, more balanced multimodal, text, and generation mixtures, and scaling from 1.5B to 7B language backbones. Its expanded data includes roughly 90M understanding examples and 72M synthetic aesthetic generation examples. Treating both releases as one lineage preserves the original architectural contribution rather than presenting the later checkpoint as the family's beginning.
ARIA is an open-source, multimodal native Mixture-of-Experts (MoE) model designed to seamlessly integrate and understand diverse modalities like text, code, images, and video, achieving state-of-the-art performance in its class. It features a fine-grained MoE decoder for efficient parameter utilization, a lightweight visual encoder, and a 4-stage training pipeline that builds capabilities in language understanding, multimodal comprehension, long context handling, and instruction following.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, Chongyan Zhu, Xiaoyi Ren, Chao Li, Yifan Ye, Peng Liu, Lihuan Zhang, Hanshu Yan, Guoyin Wang, Bei Chen, Junnan Li
Architecture figure: ARIA's visual encoder, projection layer, and fine-grained MoE are described in prose and a configuration table; its only numbered model figure visualizes expert specialization, not architecture.
ℹ️ More Information
ARIA's architecture is centered around a fine-grained Mixture-of-Experts (MoE) decoder, which is more efficient than traditional dense decoders. This MoE approach activates 3.5B parameters per text token and 3.9B per visual token, out of a total of 24.9B parameters. The model uses 66 experts in each MoE layer, with 2 shared across all inputs for common knowledge, and 6 activated per token by a router. The visual encoder is a lightweight (438M parameter) Vision Transformer (ViT) combined with a projection module. The ViT processes images at various resolutions (medium, high, and ultra-high), preserving aspect ratios. The projection module uses cross-attention and an FFN layer to convert image embeddings into visual tokens, which are then integrated with text tokens by the MoE. ARIA's training uses a 4-stage pipeline: (1) Language pre-training (6.4T text tokens, 8K context window); (2) Multimodal pre-training (400B multimodal tokens, including interleaved image-text, synthetic image captions, document transcriptions and QA, video captions and QA); (3) Multimodal long-context pre-training (extending context to 64K tokens); and (4) Multimodal post-training (instruction following with 20B tokens). The data curation process is rigorous, incorporating techniques like de-duplication, quality filtering, and data clustering. The training infrastructure avoids pipeline parallelism, using a combination of expert parallelism and ZeRO-1 data parallelism, which contributes to efficient training without the need for tensor parallelism. A load-balancing loss and z-loss are used to stabilize training. The paper demonstrates that, despite having modality-generic experts, ARIA naturally develops expert specialization during pre-training. Analysis of expert activation shows distinct visual specialization in several layers, particularly for image, video, and PDF content. ARIA also shows excellent performance in handling long-context multimodal data, surpassing other open models and competing favorably with proprietary models in tasks like long video and document understanding.
Emu3 tokenizes text, images, and video into discrete sequences and trains a single 8B autoregressive transformer from scratch to predict the next token. It unifies multimodal understanding and visual generation without a diffusion model, CLIP vision encoder, or separately pretrained LLM.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, et al.
Released: 2024-09-27
Figure 1. A single next-token objective across text, image, and video tokens. Source paper, PDF p. 1. Figure notice.
ℹ️ More Information
Emu3 first maps images and videos into discrete codes with a learned vision tokenizer, then interleaves those codes with ordinary text tokens. A single decoder-only transformer models every resulting sequence with the same causal next-token objective. Images are serialized with explicit spatial structure, while videos extend the representation across time. Because generation and perception share the same token space and transformer, Emu3 can caption and answer questions about images, generate images from text, and causally generate or continue video without adding a diffusion decoder or a compositional vision-language bridge.
The 8B transformer is pretrained from scratch on approximately 0.2 trillion visual tokens and 0.8 trillion text tokens. Pretraining establishes general multimodal generation and understanding; separate post-training paths then produce Emu3-Chat for instruction-following perception and Emu3-Gen for controllable image generation. Generation post-training includes supervised fine-tuning and preference optimization, whereas the chat model is tuned on multimodal conversations and perception tasks.
The pretraining data combines images, videos, interleaved image-text sequences, and multilingual text and code gathered and filtered at scale. Post-training uses curated image descriptions, visual question-answering and dialogue examples for Emu3-Chat, plus high-quality text-image pairs and preference data for Emu3-Gen. The project separately releases the base-stage model, chat and generation checkpoints, and the Emu3 vision tokenizer.
Molmo is Ai2's family of open vision-language models, while PixMo is the collection of human- and programmatically-created datasets used to train them. Molmo joins a CLIP vision encoder to OLMo, OLMoE, or Qwen2 language backbones and supports text generation, visual grounding, and pointing.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, et al.
Released: 2024-09-25
Figure 2. Vision encoding, multimodal connection, language decoding, and grounded pointing. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Molmo uses a pretrained CLIP ViT-L/14 vision encoder and a lightweight connector to supply image features to an autoregressive language model. The family spans MolmoE-1B, whose mixture-of-experts backbone has roughly 1B active and 7B total parameters, dense 7B variants based on OLMo or Qwen2, and a 72B Qwen2-based model. Multi-crop high-resolution processing improves fine-detail perception, while coordinate tokens let the model answer with points as well as natural language.
Training separates multimodal pretraining from supervised fine-tuning. Pretraining teaches image understanding primarily through dense captions, after which instruction tuning develops question answering, document reading, counting, pointing, and conversational behavior. The pipeline starts from pretrained vision and language components but releases the resulting weights, training and evaluation code, intermediate checkpoints, and data. Crucially, the team avoids distilling image understanding from proprietary VLM outputs.
PixMo names the data suite, not the model. PixMo-Cap contains detailed captions collected from people through speech, then transcribed and cleaned. Fine-tuning sources include PixMo-AskModelAnything, PixMo-CapQA, PixMo-Points, PixMo-Point-Explanations, PixMo-Docs, PixMo-Clocks, and PixMo-Count. Together they cover free-form image questions, caption-derived QA, referring expressions and point annotations, explanations with inline points, synthetic documents and charts, clock reading, and counting.
Llama 3.2-Vision extends the Llama 3 text-only model with multimodal capabilities, allowing it to process both text and images. This model, available in 11B and 90B parameter sizes, leverages a vision adapter with cross-attention layers to integrate image representations from a separate vision encoder into the core Llama 3 LLM, achieving strong performance on visual recognition, image reasoning, captioning, and visual question answering.
Meta
Architecture figure: The official source describes the image encoder and cross-attention adapter in prose; benchmark tables and marketing artwork are not legitimate substitutes.
ℹ️ More Information
Llama 3.2-Vision builds upon the Llama 3 architecture, an auto-regressive language model using an optimized transformer. It adds a vision adapter, comprised of cross-attention layers, to incorporate visual information. This adapter receives input from a separate vision encoder (not part of the core Llama 3 model), allowing the model to process images without directly converting them into text tokens. The <|image|> tag within the prompt signifies the presence of an image and dictates where the visual information is integrated via cross-attention. This integration occurs after the image tag and influences subsequent text tokens. The model supports a context length of 128k tokens and utilizes Grouped-Query Attention (GQA). The model family was trained on 6B image-text pairs. Pretraining data cutoff is December 2023, supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. However image-text tasks are only in English. The model's training involves supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) for instruction-tuned versions. The base models are suitable for text completion, with or without an image, using specific prompt formats. Instruction-tuned models excel at tasks like Visual Question Answering (VQA), Document VQA (DocVQA), image captioning, and image-text retrieval. The training process includes stages of pretraining and annealing, leveraging a vast amount of data and significant computational resources (H100 GPUs). Key capabilities include handling both text and image inputs, answering questions about images, generating captions, and performing visual reasoning. The model does not support built-in tool calling (like brave_search or wolfram_alpha) when an image is present in the prompt; tool calling is only available for text-only inputs. The intended use cases cover a wide range of applications, but usage is restricted by the Llama 3.2 Community License and Acceptable Use Policy, particularly regarding languages and potential misuse. Meta emphasizes a responsible deployment approach, including providing tools like Llama Guard for safety and encouraging developers to implement appropriate safeguards. The model underwent extensive evaluations, including red teaming and assessments for critical risks such as CBRNE, child safety, and cyber attacks.
NVLM 1.0 is a family of multimodal large language models (LLMs) achieving state-of-the-art results on vision-language tasks, rivaling proprietary and open-access models. It demonstrates improved text-only performance after multimodal training and offers a comprehensive comparison of decoder-only and cross-attention-based architectures, introducing a novel hybrid architecture and a 1-D tile-tagging design for high-resolution images.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
Figure 3. NVLM-X, NVLM-H, and NVLM-D share a dynamic-high-resolution visual pathway but integrate vision differently. Source paper, PDF p. 9. Figure notice.
ℹ️ More Information
NVLM (NVIDIA Vision Language Model) introduces a family of models with three primary architectures: NVLM-D (Decoder-only), NVLM-X (Cross-attention-based), and NVLM-H (Hybrid). All models share a common vision pathway, employing a frozen InternViT-6B-448px-V1-5 vision encoder with dynamic high-resolution (DHR) processing. DHR involves dividing input images into tiles (up to 6, with varying aspect ratios) and a downscaled global "thumbnail" tile. These tiles are processed by the vision encoder, and the resulting 1024 tokens per tile are downsampled to 256 via pixel shuffling. NVLM-D (Decoder-only): Connects the vision encoder to the LLM (Qwen2-72B-Instruct or Nous-Hermes-2-Yi-34B) via a 2-layer MLP projector. It introduces a novel 1-D tile-tagging design for handling high-resolution images. Text-based tile tags (e.g., <tile_1>) are inserted before the flattened image tokens of each tile to provide positional information to the LLM. Training involves pretraining (frozen LLM and vision encoder, training only the MLP) and supervised fine-tuning (SFT) (unfrozen LLM and MLP). Crucially, a high-quality text-only SFT dataset is included to maintain/improve text-only performance. NVLM-X (Cross-attention-based): Uses gated cross-attention layers to process image tokens, similar to Flamingo, but without a Perceiver resampler. Image features are projected to the LLM's hidden dimension with a one-layer MLP. Gated X-attention layers are interleaved with LLM self-attention layers. Training also has pretraining and SFT stages. The LLM backbone is unfrozen during SFT, and a high-quality text-only dataset is used. 1-D tile tags are also used, but within the X-attention layers. NVLM-H (Hybrid): Combines aspects of NVLM-D and NVLM-X. The thumbnail image tokens are processed by the LLM's self-attention layers (like NVLM-D), while the regular tile tokens are processed by gated cross-attention (like NVLM-X). This aims to balance multimodal reasoning with computational efficiency. It also uses 1-D tile tags in cross-attention. The 1-D tile-tagging design significantly improves performance, especially on OCR-related tasks, compared to simply concatenating image tokens or using 2D grid/bounding box tags. The authors emphasize that dataset quality and task diversity are more important than sheer scale, even during pretraining. NVLM models achieve strong performance on both vision-language and text-only tasks. This is achieved by including a high-quality text-only dataset during SFT and incorporating multimodal math and reasoning data. Decoder VS X-Attention: Cross attention based models are more efficient in high-resolution images. However, Decoder models provides unified multimodel reasoning and higher accuracy in OCR-related tasks. Curated from open-source datasets, including captioning (COCO, CC3M, SBU, LAION-115M), VQA (VQAv2, Visual Genome, DVQA), document understanding (Docmatix), OCR/Scene-Text (various datasets), and Math (CLEVR-Math). Emphasis on quality over quantity. A diverse collection of task-oriented datasets, including captioning, VQA, chart/diagram understanding, document understanding, OCR, math, and science datasets. High-quality text-only data from various sources (ShareGPT, SlimOrca, EvolInstruct, etc.) and categories (general, math, coding) is crucial for maintaining/improving text-only performance. Refined using GPT-40 and GPT-40-mini. NVLM models are evaluated on a wide range of vision-language benchmarks (MMMU, MathVista, OCRBench, AI2D, ChartQA, DocVQA, TextVQA, RealWorldQA, VQAv2) and text-only benchmarks (MMLU, GSM8K, MATH, HumanEval).
Pixtral 12B is a 12-billion-parameter multimodal language model developed by Mistral AI, designed to excel in both understanding images and text, achieving leading performance on various multimodal benchmarks. The core of the VLM is built upon the transformer architecture. A strong aspect of the VLM is, Pixtral 12B is trained with a new vision encoder from scratch to natively support variable image sizes and aspect ratios.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, et al. (Mistral AI Science Team)
Figure 3. Pixtral combines a variable-resolution vision encoder with a 128K-context multimodal decoder. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Pixtral 12B has two main components, vision encoder (Pixtral-ViT), which tokenizes images and a multimodal decoder, which predicts the next token given a sequence of text and images. Pixtral can take an arbitrary number of images as an input, provided they fit within its 128K context window. The vision encoder (Pixtral-ViT) is trained from scratch with a novel ROPE-2D implementation, allowing it to process images at their native resolution and aspect ratio. The model can flexibly process images at low resolution in latency-constrained settings, while processing images at high resolution when fine-grained reasoning is required. For distinguishing between images with same number of patches but different aspect ratios, [IMAGE BREAK] tokens are inserted between image rows. Additionally, an [IMAGE END] token marks the end of image sequence. The model employs a gated FFN architecture, implementing gating in the hidden layer in place of standard feedforward layer in the attention block. For processing images within a single batch, the model flattens images along the sequence dimension and concatenates them. A block diagonal mask is constructed to prevent attention leakage between patches of different images. Traditional learned and absolute position embeddings are replaced by ROPE-2D, which allows handling variable image sizes. The multimodal decoder of Pixtral is built on top of Mistral Nemo 12B, a 12-billion parameter decoder-only language model. The decoder uses a causal self-attention. The vision encoder is connected to the multimodal decoder by a two-layer fully connected network. The paper describes Pixtral as an instruction-tuned model, pre-trained on large-scale interleaved image and text documents. The Paper contributes an open-source benchmark called MM-MT-Bench, for evaluating vision-language models. Pixtral excels at multimodal instruction following, surpassing comparable open-source models on the MM-MT-Bench benchmark.
The Pixtral pathway later scaled to Pixtral Large in November 2024 and became the visual component of Mistral's general-purpose multimodal releases. Mistral Small 4, released in 2026, folds Pixtral-style perception together with reasoning and agentic coding in one open model; because Mistral has not disclosed a distinct replacement vision architecture, it belongs to this lineage rather than a duplicate first-class entry. Pixtral Large · Mistral Small 4
VILA-U uses a shared visual tokenizer and a single next-token objective for image, video, and language understanding and generation, avoiding separate diffusion decoders or separate visual pathways.
VILA-U Team
Released: 2024-09-06
Figure 1. A shared visual tokenizer, autoregressive model, and modality decoders. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
VILA-U converts images and video frames into discrete visual tokens and processes them together with text through one autoregressive language-model backbone. Its key component is a unified vision tower trained to preserve the information needed for both semantic understanding and visual reconstruction. Aligning its discrete visual tokens with language during pretraining improves perception while allowing the same tokens to be generated directly by next-token prediction.
Training mixes image-text understanding, video understanding, pure text, and high-quality text-to-image examples, then instruction-tunes the model for conversational and generative tasks. The design demonstrates that competitive image generation does not necessarily require a diffusion module and that understanding and generation can share a fully tokenized visual interface.
Qwen2-VL is the latest iteration of the Qwen vision-language model family, building upon the Qwen-VL architecture and introducing significant enhancements for improved understanding of images and videos. It excels in various tasks, including visual question answering, dialogue, content creation, and even agent-based control of devices like mobile phones and robots.
Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren
Figure 3. Qwen2-VL's M-RoPE decomposes multimodal position encoding into temporal, height, and width components. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
Qwen2-VL continues to leverage the core architecture of Qwen-VL, combining a Vision Transformer (ViT) with approximately 600M parameters and Qwen2 language models. This ViT is designed to handle both image and video inputs seamlessly. The key architectural improvements in Qwen2-VL include Naive Dynamic Resolution support and Multimodal Rotary Position Embedding (M-ROPE). Naive Dynamic Resolution allows the model to handle arbitrary image resolutions by mapping them into a dynamic number of visual tokens. This ensures that the model input accurately reflects the information content of the image, regardless of its original resolution. This approach is more aligned with human visual perception, which adapts to different image sizes and resolutions. M-ROPE enhances the model's ability to capture positional information in multimodal inputs. It deconstructs the original rotary embedding into three parts, representing temporal, height, and width information. This allows the LLM to simultaneously process and integrate 1D textual, 2D visual (image), and 3D video positional information, leading to a more comprehensive understanding of the input sequence. These architectural enhancements, combined with a robust training process, enable Qwen2-VL to achieve state-of-the-art performance on various visual understanding benchmarks, including MathVista, DocVQA, RealWorldQA, and MTVQA. It can also understand videos over 20 minutes long, enabling high-quality video-based question answering, dialogue, and content creation. Furthermore, Qwen2-VL's capabilities in complex reasoning and decision-making allow it to be integrated with devices like mobile phones and robots for automatic operation based on visual input and text instructions. The model also supports multilingual understanding of text within images, including most European languages, Japanese, Korean, Arabic, and Vietnamese, broadening its applicability to a global user base.
EAGLE is a family of open-source Multimodal Large Language Models (MLLMs) that leverage a mixture of vision encoders to achieve state-of-the-art performance on various benchmarks, particularly in tasks involving OCR and document understanding. The study focuses on systematically exploring the design space of MLLMs with multiple vision encoders, aiming to identify optimal design choices and improve MLLM perception.
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, Guilin Liu
Figure 2. EAGLE explores mixtures of vision experts and alternative fusion strategies for multimodal language models. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
EAGLE builds upon the LLaVA architecture, consisting of a large language model, a vision encoder, and a projection layer. The core innovation lies in integrating multiple vision experts, each pre-trained on different tasks and resolutions, to enhance the model's ability to perceive and comprehend diverse visual information. The study explores various aspects of this design space, including high-resolution adaptation, fusion paradigms, and optimal encoder combinations. It introduces a Pre-Alignment training stage to address representational inconsistencies between vision-focused encoders and language tokens. The training process consists of three progressive stages: vision-language pre-alignment, joint-projector training, and supervised fine-tuning. EAGLE achieves state-of-the-art performance on various benchmarks, demonstrating significant advantages in OCR and document understanding tasks. The study highlights the importance of systematic design space exploration and the effectiveness of combining multiple vision experts with a streamlined fusion strategy and a pre-alignment training stage for building high-performing MLLMs.
Show-o unifies visual understanding, text generation, and image generation by combining causal autoregression with masked discrete diffusion inside a single transformer.
Show Lab
Released: 2024-08-22
Figure 2. Causal text attention and full-attention discrete image diffusion. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Show-o represents images with discrete visual tokens and processes text and images through one omni-attention transformer. Language and multimodal-understanding sequences use autoregressive next-token prediction, whereas image generation uses iterative masked-token prediction, giving the same network a discrete-diffusion generation path. Its attention mask changes with the task: causal dependencies are retained where autoregression is required, while visual generation tokens can exchange bidirectional context.
Training combines large-scale text, image-text pairs, visual question answering and instruction data, and text-to-image data, followed by task-oriented fine-tuning. The resulting model supports captioning, VQA, text-to-image generation, inpainting, extrapolation, and interleaved mixed-modality generation without attaching an external diffusion model.
Idefics3-8B is a powerful open-source vision-language model (VLM) that significantly outperforms its predecessor, Idefics2-8B, while being trained efficiently and exclusively on open datasets. It leverages a straightforward pipeline and introduces Docmatix, a massive dataset for document understanding, to achieve state-of-the-art performance within its size category across various multimodal benchmarks.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, Léo Tronchon
Figure 1. Idefics3 maps vision-encoder features into interleaved visual tokens consumed by an autoregressive language model. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Idefics3-8B builds upon the foundation of pre-trained unimodal models, specifically Llama 3.1 instruct as the language model and SigLIP-SO400M as the vision encoder. It adopts a self-attention architecture, where visual features are treated as tokens and concatenated with text tokens before being fed into the LLM. To enhance OCR capabilities and address the bottleneck of limited visual tokens per image, Idefics3-8B replaces the perceiver resampler used in Idefics2 with a simple pixel shuffle strategy, similar to InternVL-1.5. This strategy reduces the number of image hidden states by a factor of 4, allowing for the encoding of larger images (up to 364x364 pixels) into 169 visual tokens. The model utilizes an image-splitting strategy during both training and inference, dividing the original image into a matrix of 364x364 pixel tiles. To preserve the 2D structure and positional information of these tiles, a text token '\n' is inserted after each row of tiles, and the downscaled original image is appended to the sequence. Additionally, each tile is prepended with textual tokens indicating its position in the matrix. The training process consists of three stages of pre-training followed by supervised fine-tuning. In the first pre-training stage, the backbones (LLM and vision encoder) are frozen, and only the newly initialized parameters are trained. The maximum image resolution is gradually increased from 364² to 1820². From the second stage onward, the backbones are efficiently trained using DoRA (a variant of LoRA), and larger images are introduced into the training data. The final pre-training stage focuses on training with large synthetic datasets, including Docmatix, Websight, LNQA, PixelProse, and ChartGemma. During supervised fine-tuning, NEFTune noise is applied to the inputs, and the loss is calculated only on the answer tokens. The learning rate is kept constant for the first two pre-training stages and linearly decayed to zero during the final pre-training stage and supervised fine-tuning. Idefics3-8B demonstrates significant improvements over Idefics2, particularly in document understanding tasks, achieving a 13.7-point improvement on DocVQA. This highlights the effectiveness of the Docmatix dataset and the architectural choices made in Idefics3-8B. The model also achieves state-of-the-art performance within its size category across various multimodal benchmarks, including MMMU, MathVista, MMStar, and TextVQA, showcasing its strong capabilities in visual understanding and reasoning.
Transfusion trains one transformer over mixed text-and-image sequences while applying the representation and loss best suited to each modality: autoregressive next-token prediction for text and diffusion for continuous image patches.
Meta FAIR
Released: 2024-08-20
Figure 1. A shared Transformer with autoregressive text and continuous image diffusion. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
Unlike unified models that quantize images into discrete codes, Transfusion retains continuous image representations and denoises them with a diffusion objective. Text tokens and noisy image patches occupy the same sequence; modality-specific input and output layers connect them to a shared transformer. Causal attention governs text, while image patches can interact bidirectionally within each image.
The paper derives scaling behavior from models pretrained from scratch and scales the recipe to 7B parameters and two trillion multimodal tokens. Additional modality-specific encoding and decoding layers improve image fidelity and can compress an image to as few as 16 latent patches. This makes Transfusion an important alternative to fully discrete, next-token-only multimodal architectures.
mPLUG-Owl3 keeps long visual inputs outside the ordinary language-token stream and lets language tokens retrieve relevant visual evidence through hyper-attention, making multi-image and long-video reasoning more efficient.
mPLUG-Owl Team
Released: 2024-08-09
Figure 2. Vision encoding and Hyper-Attention blocks inside the language model. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Instead of concatenating every vision feature with the language sequence and paying full self-attention cost, mPLUG-Owl3 introduces hyper-attention blocks that use language states to selectively aggregate separately maintained visual features. These blocks are inserted into the language model and create a language-guided semantic space for single images, interleaved image-text documents, retrieved visual knowledge, and long videos.
The approach reduces interference from irrelevant frames and avoids consuming the entire language context with visual tokens. Training combines single-image instruction data with multi-image, interleaved, retrieval-augmented, and video examples. The accompanying Distractor Resistance evaluation measures whether the model can retain the relevant visual evidence when long sequences contain competing images.
VITA is the first open-source Multimodal Large Language Model (MLLM) capable of simultaneously processing and analyzing video, image, text, and audio modalities while offering an advanced multimodal interactive experience. It addresses the limitations of existing open-source models, which often excel in either understanding or interaction but rarely both, by integrating architectural innovations with advanced training and development strategies.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, Xing Sun
Figure 2. VITA unifies text, audio, image, and video inputs with state tokens, an LLM, and speech output. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
VITA starts with the Mixtral 8x7B model as its language foundation, chosen for its strong performance and sparse mixture of experts (SMoE) architecture. To enhance its Chinese language capabilities, the vocabulary is expanded with Chinese terms, and the model undergoes bilingual instruction tuning using a high-quality bilingual text corpus. This ensures proficiency in both Chinese and English. For visual modality, VITA employs InternViT-300M-448px as the visual encoder, processing images at 448x448 resolution and generating 256 tokens after passing through a two-layer MLP visual connector. High-resolution images are handled using a dynamic patching strategy, while videos are treated as special cases of images, with frame sampling based on video length. For audio modality, a Mel Filter Bank block is used to process the input audio, followed by 4xCNN downsampling layers and a 24-layer transformer, resulting in 25 tokens for every 2 seconds of audio. A two-layer MLP serves as the audio-text modality connector. The training pipeline consists of three stages: LLM instruction tuning, multimodal alignment, and multimodal instruction tuning. LLM instruction tuning focuses on enhancing the base LLM's bilingual capabilities. Multimodal alignment aims to bridge the representation gap between text and other modalities by training individual encoders and connectors for each modality. This involves collecting and curating a large-scale, high-quality multimodal dataset, including image descriptions, general image QA, OCR and diagram data, general video descriptions, general video QA, and pure text data. Multimodal instruction tuning further refines the model's ability to follow instructions and understand different modalities. A specially designed state token is introduced to distinguish the type of input query (effective audio, noisy audio, or text), enabling non-awakening interaction during inference. To achieve natural multimodal human-computer interaction, VITA introduces two key innovations: non-awakening interaction and audio interrupt interaction. These are implemented using a duplex pipeline during deployment. Two VITA models run concurrently: one for generating responses to user queries (Generation model) and the other for monitoring environmental audio (Monitoring model). The Monitoring model uses SileroVAD for voice activity detection and filters out noisy audio based on the state token. If an effective audio query is detected, the Monitoring model interrupts the Generation model, consolidates the historical context, and responds to the latest query. The two models then swap identities, ensuring continuous monitoring and seamless interaction.VITA demonstrates strong performance on various unimodal and multimodal benchmarks, showcasing its robust foundational capabilities in multilingual, vision, and audio understanding. While still lagging behind closed-source counterparts in certain areas, VITA represents a significant step towards open-source interactive omni-modal LLMs, paving the way for future research and development in this field.
LLaVA-OneVision is a family of open large multimodal models (LMMs) designed to excel in various computer vision scenarios, including single-image, multi-image, and video understanding. It pushes the performance boundaries of open LMMs by consolidating insights from the LLaVA-NeXT blog series, focusing on data, models, and visual representations. Notably, LLaVA-OneVision demonstrates strong transfer learning capabilities, enabling it to excel in video understanding tasks by leveraging knowledge learned from image data.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li
Figure 1. LLaVA-OneVision extends the minimal LLaVA vision-encoder, projector, and LLM architecture to multiple visual signals. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
LLaVA-OneVision inherits the minimalist design of the LLaVA series, aiming to effectively leverage pre-trained capabilities of both the LLM and the visual model while facilitating strong scaling. The architecture consists of three key components: a large language model (LLM), a vision encoder, and a projector. The authors choose Qwen-2 as the LLM due to its strong language capabilities and various model sizes available. For the vision encoder, they opt for SigLIP, which has shown to yield higher LMM performance among open vision encoders. A 2-layer MLP is used as the projector to map image features into the word embedding space, creating a sequence of visual tokens. The model utilizes a flexible visual representation strategy called Higher AnyRes, which builds upon the original AnyRes strategy introduced in LLaVA-NeXT. This strategy involves dividing the input image into crops, each with a resolution suitable for the vision encoder, and then applying bilinear interpolation to reduce the number of tokens per crop if needed. This allows the model to handle high-resolution images and videos efficiently while preserving important visual details. The specific configuration of Higher AnyRes is adapted for different scenarios: single-image, multi-image, and video. For single-image data, a large maximum spatial configuration is used to maintain the original image resolution and a large number of visual tokens are allocated to effectively represent the visual signal. For multi-image data, only the base image resolution is considered, eliminating the need for multi-crop and saving computational resources. For video data, each frame is resized to the base image resolution and bilinear interpolation is used to reduce the number of tokens per frame, allowing for the processing of a larger number of frames. The training process follows a three-stage curriculum learning approach: language-image alignment, high-quality knowledge learning, and visual instruction tuning. The first stage focuses on aligning visual features with the LLM's embedding space using the LLaVA align dataset. The second stage refines and enhances the model's knowledge base using high-quality data from three major categories: re-captioned detailed description data, document/OCR data, and Chinese and language data. The final stage involves visual instruction tuning, where the model is trained on a diverse set of visual tasks with preferred responses. This stage is further divided into two phases: single-image training and OneVision training. Single-image training focuses on single-image scenarios, while OneVision training expands the model's capabilities to multi-image and video scenarios, enabling task transfer and emerging capabilities. LLaVA-OneVision demonstrates state-of-the-art performance on various benchmarks, including single-image, multi-image, and video tasks, showcasing its effectiveness and versatility in handling diverse visual scenarios.
Released by arXiv v1 on September 28, 2025, LLaVA-OneVision-1.5 preserves the established vision-encoder-projector-language-model design but makes the complete construction pipeline reproducible: an 85M-sample concept-balanced mid-training set, a 26M-sample instruction set, 64B compressed multimodal tokens, and offline parallel data packing reduce reported training cost to approximately $16,000. It is a major open-training update, but architecturally remains part of LLaVA-OneVision. Paper · Repository
VILA² (VILA-augmented-VILA) introduces a novel approach to address the limitations of data quantity and quality in training Visual Language Models (VLMs). Instead of relying on costly human annotation or distillation from proprietary models, VILA² leverages the VLM itself to iteratively refine and augment its pretraining data, leading to significant performance improvements and achieving state-of-the-art results on the MMMU leaderboard among open-sourced models.
Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, Hongxu Yin
Figure 1. VILA² improves itself through generic model-in-the-loop recaptioning and specialist-model augmentation. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
VILA² employs a two-step iterative process: self-augmenting and specialist-augmenting. The self-augmenting loop focuses on enhancing the general knowledge of the VLM by using the model itself to re-caption its pretraining data. This process starts with an initial VLM (VILA0) trained on a dataset with typically short and brief captions, like COYO. VILA0 is then used to generate longer and more detailed captions for the same images, creating a synthetic dataset. This augmented dataset, combined with the original data, is used to train the next iteration of the VLM (VILA1). This loop can be repeated multiple times, with each iteration improving the caption quality and subsequently the VLM's performance. However, this self-augmentation process eventually reaches saturation. To overcome this limitation, VILA² introduces the specialist-augmenting loop. This involves fine-tuning the self-augmented VLM on specific downstream tasks, creating specialist VLMs with expertise in areas like spatial awareness, OCR, and grounding. These specialists are then used to re-caption the pretraining data, focusing on their specific domain knowledge. The self-augmented VLM is then retrained on this specialist-recaptioned data, further boosting its performance. This approach leverages the synergy between the vast amount of data in pretraining and the specialized knowledge acquired during fine-tuning. The architecture of VILA² follows the standard auto-regressive VLM design, consisting of a large language model (LLM), a visual encoder, and an image-text projector. The authors experiment with different LLMs (Llama2-7B, Llama3-8B-Instruct, and Yi-34B) and visual encoders (SigLIP and InternViT-6B). They also introduce a 4x downsampling of visual tokens to reduce computational cost. The training process follows the typical three-stage paradigm: projector initialization, vision-language pre-training, and visual instruction-tuning. VILA² demonstrates significant performance improvements over previous state-of-the-art methods on various benchmarks, including general VQA, text-oriented VQA, general multimodal benchmarks, and image captioning. This highlights the effectiveness of the proposed self- and specialist-augmentation techniques in enhancing VLM training and achieving state-of-the-art results.
INF-LLaVA is a novel Multimodal Large Language Model (MLLM) designed to effectively process high-resolution images. It addresses the limitations of existing cropping-based and dual-encoder methods by introducing two innovative modules: Dual-perspective Cropping Module (DCM) and Dual-perspective Enhancement Module (DEM). DCM segments high-resolution images into sub-images from both local and global perspectives, preserving detailed and contextual information. DEM facilitates efficient interaction between local and global features, enhancing the model's understanding of complex visual relationships. Extensive evaluations demonstrate INF-LLaVA's superior performance on various benchmarks, establishing a new state-of-the-art in vision-language tasks.
Yiwei Ma, Zhibin Wang, Xiaoshuai Sun, Weihuang Lin, Qiang Zhou, Jiayi Ji, Rongrong Ji
Figure 2. INF-LLaVA combines dual-perspective cropping, CLIP encoding, feature recombination, enhancement, and language reasoning. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
INF-LLaVA pushes the boundaries of Multimodal Large Language Models (MLLMs) by tackling the critical challenge of high-resolution image perception. It aims to leverage the richness of detail present in high-resolution images without succumbing to the computational limitations imposed by traditional MLLM architectures. INF-LLaVA achieves this through a sophisticated approach that combines innovative cropping and feature enhancement techniques, resulting in a model capable of simultaneously capturing fine-grained local details and comprehensive global context. At the core of INF-LLaVA lies the Dual-perspective Cropping Module (DCM), a strategic cropping strategy that surpasses conventional approaches by integrating both local and global perspectives. This dual-perspective approach ensures that each extracted sub-image retains not only the intricate details essential for accurate analysis but also the broader contextual information crucial for understanding the relationships between objects. While local-perspective cropping preserves continuous visual information at high resolution, capturing the essence of individual objects and regions, global-perspective cropping leverages a unique interleaving technique to preserve the overall spatial relationships between objects within the high-resolution image. This balanced combination ensures that the model can perceive both the "trees" and the "forest," enabling a holistic understanding of the visual scene. To further enhance the model's understanding, INF-LLaVA introduces the Dual-perspective Enhancement Module (DEM). This module facilitates efficient and effective interaction between the local and global features extracted by the vision encoder, enriching the representation with multi-scale information. Instead of relying on computationally expensive cross-attention directly on high-resolution features, DEM employs a more resource-efficient strategy. It leverages 2D positional priors to concatenate global-perspective sub-image features back into the original image's shape, effectively recreating a high-resolution representation of the global context. These recombined features are then re-cropped from a local perspective, and cross-attention is performed between corresponding local and global sub-images to enhance global features with fine-grained local details. A symmetrical process enhances local features with global context. This meticulously designed interaction between local and global features ensures that the resulting representation is not only rich in detail but also cognizant of the broader context. The dual-enhanced features are then projected into a format compatible with the LLM through a linear connector. The LLM then processes the combined visual and textual information to generate a coherent and contextually relevant response. Through extensive evaluations on a diverse set of benchmarks, including ScienceQA-img, OKVQA, SEEDBench, MMBench, AI2D, LLaVA-Bench-in-the-wild, and MMMU, INF-LLaVA demonstrates its superior performance over existing MLLMs. Its ability to effectively handle high-resolution images while maintaining computational efficiency establishes a new state-of-the-art in the field. The open-source release of INF-LLaVA, along with its pretrained model and code, paves the way for further research and exploration of high-resolution image perception in multimodal large language models, pushing the boundaries of multimodal understanding and enabling the development of more powerful and versatile AI systems.
SlowFast-LLaVA (SF-LLaVA) is a training-free video large language model that effectively captures both detailed spatial semantics and long-range temporal context in videos without requiring any additional fine-tuning on video data. It achieves this by leveraging a two-stream SlowFast design inspired by action recognition models, allowing it to process a larger number of frames and outperform existing training-free methods on various video benchmarks.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, Afshin Dehghan
Figure 2. SlowFast-LLaVA combines low-frame-rate spatial detail with high-frame-rate motion features. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
SF-LLaVA builds upon the LLaVA-NeXT framework and utilizes a two-stream approach, similar to SlowFast networks in action recognition, to process video inputs. The model first uniformly samples N frames from the input video. These frames are then processed independently by a visual encoder, such as CLIP-L, followed by a visual-language adapter for feature alignment. The resulting frame features are then fed into two separate pathways: Slow and Fast. The Slow pathway focuses on capturing detailed spatial semantics by processing a smaller number of frames (Nslow) at a higher spatial resolution (e.g., 8 frames with 24x24 tokens). It applies spatial pooling with a small stride (e.g., 1x2) to aggregate features and reduce the number of tokens. The Fast pathway focuses on capturing temporal context and motion cues by processing all N frames (Nfast = N) at a lower spatial resolution (e.g., 64 frames with 4x4 tokens). It applies aggressive spatial pooling to each frame to prioritize temporal information. The features from both pathways are then flattened and concatenated, forming a comprehensive video representation that balances spatial details and temporal context. This aggregated feature vector, along with the text prompt and question, is then fed into the LLM (LLaVA-NeXT) to generate the final answer. This training-free approach eliminates the need for expensive fine-tuning on video datasets, making SF-LLaVA highly efficient and adaptable to various video scenarios. The authors demonstrate the effectiveness of SF-LLaVA on three different video question-answering tasks (Open-Ended VideoQA, Multiple Choice VideoQA, and Text Generation) across eight benchmarks, showcasing its superior performance compared to existing training-free methods and even surpassing some state-of-the-art supervised fine-tuned video LLMs.
EVLM is an efficient multimodal language model designed to minimize computational costs while maximizing the model's ability to perceive visual signals comprehensively. It addresses the challenges of handling long sequences of visual signals, particularly in video data, by employing a cross-attention mechanism and hierarchical ViT features, achieving competitive performance in tasks like image and video captioning.
Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, Changyi Liu, Dewen Fan, Huihui Xiao, Jiahong Wu, Fan Yang, Size Li, Di Zhang
Figure 2. EVLM injects hierarchical EVA2-CLIP features into the language model through gated cross-attention layers. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
EVLM is built upon the Flamingo architecture, incorporating a visual encoder, a large language model, and a Gated Cross-Attention Layer. To enhance visual perception, EVLM utilizes the 4.4B EVA2-CLIP-E-Plus model as the visual encoder, extracting hierarchical visual features by uniformly sampling 8 feature sequences from the last 40 layers of the transformer. These features are then sequentially fed into different Gated Cross-Attention layers of the Flamingo model. Unlike Flamingo, which uses a single media token image, EVLM replaces it with a set of 16 learnable tokens, aiming to capture visual features similar to Q-former. The attention mechanism is designed to allow each set of learnable tokens to interact only with the corresponding image, while text sequences interact only with the previous image in the multimodal sequence. This approach ensures efficient interaction between visual and textual information. For the language model, EVLM employs the Qwen-14B-Chat 1.0, chosen for its strong performance in content understanding and logical reasoning. A gated cross-attention layer is inserted before every transformer layer of the language model to condition it on visual inputs. To further enhance model effectiveness and scale trainable parameters, a Mixture of Experts (MoE) mechanism is applied to the Cross Attention layer. This involves replicating and segmenting the FFN of the base model into multiple fine-grained experts, with a routing layer selecting the appropriate set of experts for each token. The model undergoes a three-stage training process: multi-modal pre-training, multi-task continual pre-training, and multi-modal instruction fine-tuning. Pre-training focuses on cross-modal alignment and modeling intrinsic relationships within multimodal data, using a large-scale dataset of bilingual image-text captions and web-type multimodal data. Continual pre-training further enhances the model's visual question-answering ability, while instruction fine-tuning activates its instruction-following capabilities using a diverse range of high-quality instruction tuning data.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
InternLM-XComposer-2.5 (IXC-2.5) is a versatile Large Vision Language Model (LVLM) designed to handle long-contextual input and output, excelling in various text-image comprehension and composition tasks. It achieves performance comparable to GPT-4V with a significantly smaller 7B LLM backend, demonstrating its efficiency and scalability.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang
Figure 5. InternLM-XComposer-2.5's framework supports text, single and multiple images, and video inputs. Source paper, PDF p. 6. Figure notice.
ℹ️ More Information
InternLM-XComposer-2.5 builds upon its previous iterations (IXC-2 and IXC-2-4KHD) and features a three-component architecture: a lightweight OpenAI ViT-L/14 vision encoder, a powerful InternLM2-7B LLM, and Partial LoRA for efficient alignment between the visual and language modalities. IXC-2.5 supports diverse input modalities, including text, single/multiple images, and videos. It utilizes a Unified Dynamic Image Partition strategy to handle high-resolution images and videos, resizing and padding them into smaller patches while preserving aspect ratios. For videos, frames are sampled and concatenated along the short side, creating a high-resolution composite image. The model is pre-trained in three stages: general semantic alignment, world knowledge alignment, and vision capability enhancement, using a diverse range of datasets. During pre-training, the LLM is frozen, and the vision encoder and Partial LoRA are fine-tuned to align visual tokens with the LLM. Supervised fine-tuning is then performed on a collection of datasets covering various tasks, including captioning, visual question answering, multi-turn QA, science QA, chart QA, math QA, OCR QA, video understanding, and conversation. This fine-tuning process involves jointly training all components with a weighted data sampling strategy and specific learning rate schedules for each component. IXC-2.5 also introduces two novel applications: crafting webpages and composing high-quality text-image articles. For webpage generation, the model is trained on a combination of synthetic and real-world web data, enabling it to generate HTML, CSS, and JavaScript code based on screenshots, instructions, or resume documents. For article composing, IXC-2.5 leverages Chain-of-Thought (CoT) and Direct Preference Optimization (DPO) techniques to enhance the quality of written content. This involves rewriting original prompts using CoT, generating diverse responses using different random seeds, and training a reward model to select preferred responses, ultimately leading to more creative and high-quality article generation.
OMG-LLaVA presents a novel framework that unifies image-level, object-level, and pixel-level reasoning and understanding within a single Multimodal Large Language Model (MLLM). It leverages the power of a frozen universal segmentation model (OMG-Seg) for visual encoding and a Large Language Model (LLM) for text understanding and response generation, enabling a wide range of multimodal tasks within a single, elegant architecture.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, Shuicheng Yan
Figure 3. OMG-LLaVA connects OMG-Seg visual tokens and prompts to an LLM that can decode segmentation outputs. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
OMG-LLaVA consists of two main components: a frozen universal perception module (based on OMG-Seg) and a Large Language Model (LLM). The universal perception module is responsible for encoding the input image and visual prompts into three types of visual tokens: pixel-centric, object-centric, and object-centric derived from visual prompts. The pixel-centric tokens are generated by a ConvNeXt-L based CLIP image encoder, capturing dense image features. The object-centric tokens are generated by the OMG decoder, which takes learnable object queries and visual prompt queries as input and attends to the image features to extract object-level information. This decoder can handle point, box, and mask prompts by applying constraints on the attention masks. To bridge the gap between the frozen perception module and the LLM, a novel "perception prior embedding" strategy is introduced. This strategy fuses the image features with the object queries from the OMG decoder using a mask score derived from the segmentation masks and confidence scores. The resulting weighted object queries are then added to the image features to generate the pixel-centric visual tokens, providing the LLM with rich object-level information. The object-centric visual tokens are directly taken from the foreground object queries of the OMG decoder. Both types of visual tokens, along with the text instruction tokens, are fed into the LLM, which is responsible for understanding the user's intent and generating the appropriate response. The LLM outputs text responses and object-centric visual tokens, which are then decoded by the frozen OMG decoder to produce segmentation masks. This unified architecture allows OMG-LLaVA to perform a wide range of tasks, including image captioning, visual question answering, referring segmentation, reasoning segmentation, grounded conversation generation, and region captioning, all within a single model.
Cambrian-1 is a family of 8B, 13B, and 34B multimodal LLMs centered on stronger visual representation and spatial grounding. Its Spatial Vision Aggregator dynamically combines complementary high-resolution features while limiting the number of visual tokens passed to the language model.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, Saining Xie
Released: 2024-06-24
Figure 8. Spatial Vision Aggregation across multiple visual encoders and decoder layers. Source paper, PDF p. 13. Figure notice.
ℹ️ More Information
Cambrian-1 treats the language model as an interface for systematically studying visual representations. The paper evaluates more than twenty encoders and builds its strongest systems from complementary CLIP, SigLIP, DINOv2, and ConvNeXt features. Its Spatial Vision Aggregator uses learnable visual queries and auxiliary connectors to gather local, spatially aligned evidence from these feature grids. This preserves high-resolution information while producing a fixed, substantially smaller set of tokens for the LLM. Released variants pair the connector with Phi-3, Llama 3, Vicuna 1.5, or Hermes-2-Yi backbones.
Training has two stages. First, the vision encoders and LLM remain frozen while the Spatial Vision Aggregator is trained on approximately 2.5M alignment samples. Second, the connector and language model are instruction-tuned together on Cambrian-7M. The authors' scaling studies emphasize balanced data composition, limiting overrepresented sources, and retaining demanding vision-centric examples rather than simply maximizing sample count.
Cambrian-7M is the curated subset of a larger Cambrian-10M pool assembled from public visual-question-answering, visual-conversation, OCR, chart, counting, math, coding, science, and embodied-interaction sources. The project releases the models, code, alignment data, instruction data, training recipes, and CV-Bench, its vision-centric evaluation suite.
EVE is an encoder-free vision-language model (VLM) that directly processes images and text within a unified decoder-only architecture, eliminating the need for a separate vision encoder. It achieves competitive performance with encoder-based VLMs of similar size on multiple vision-language benchmarks using only 35M publicly accessible data, with the model efficiently handling high-resolution images with arbitrary aspect ratios.
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, Xinlong Wang
Figure 2. EVE combines patch embedding, a decoder-only backbone, patch alignment, and next-word prediction without a separate vision encoder. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
EVE (Encoder-free Vision-language modEl): This model distinguishes itself by completely removing the vision encoder component typically found in VLMs. Instead, it directly integrates visual information into a decoder-only architecture (based on Vicuna-7B). This is achieved through a novel Patch Embedding Layer (PEL) that processes image patches directly, combined with a Patch Aligning Layer (PAL) that facilitates learning from a pre-trained vision encoder (CLIP-ViT-L/14) without updating the encoder itself. Crucially, EVE does not use a traditional image encoder during inference. The PEL uses a convolution layer and average pooling to create 2D feature maps from the input image. It then employs cross-attention (CA1) within a limited receptive field to enhance these features. A special <CLS> token provides a holistic view of each patch feature, and a learnable newline token <SPL> is inserted after each row of patch features to represent the 2D structure. The PAL aligns EVE's patch features with those from a frozen, pre-trained vision encoder (CLIP-ViT-L/14). This is done hierarchically, aggregating features across multiple layers of the decoder and using a layer-wise cross-attention (CA3) mechanism. A Mean Squared Error (MSE) loss between EVE's features and the vision encoder's features encourages alignment. This "implicit" supervision from the vision encoder improves visual understanding. Importantly, PAL is only used during training, not inference. The training process occurs in three stages: LLM-guided Pre-training: Only the PEL and PAL are trained, aligning the visual features with the frozen LLM (Vicuna-7B). This stage uses a subset (16M) of the total training data. Generative Pre-training: The entire model (including the LLM) is trained, using the full 33M dataset. Both text prediction (cross-entropy loss) and visual alignment (MSE loss) are used. Supervised Fine-tuning: The entire model is fine-tuned on instruction-following datasets (LLaVA-mix-665K and others). The key innovations that allow EVE to work well without a vision encoder are: LLM-Centric Pre-alignment: Stage 1 is critical for preventing model collapse and accelerating convergence. Aligning visual features before fully training the LLM is essential. Vision Recognition Capability via Extra Supervision: The PAL provides supervision from a pre-trained vision encoder during training, which enhances visual understanding without requiring the encoder during inference. Flexible Input Handling: The architecture naturally handles images of arbitrary aspect ratios and resolutions, without needing resizing, padding, or partitioning. No reliance on vision encoder: The image are directly input into the LLM model.
EVE uses a curated dataset of 33M publicly available image-text pairs for pre-training, with captions generated by Emu2 and LLaVA-1.5. Supervised fine-tuning utilizes datasets like LLaVA-mix-665K, AI2D, DocVQA, and others.
Ovis replaces the usual direct projection of continuous vision features with a learnable visual vocabulary, structurally aligning visual embeddings with the lookup-table mechanism used for text tokens.
Alibaba International Digital Commerce
Released: 2024-06-14
Figure 3. Visual-token probability distributions and structural visual embedding lookup. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
The original Ovis vision encoder predicts distributions over a learnable visual embedding table; weighted combinations of those embeddings become the visual tokens consumed by the language model. This mirrors textual embedding lookup more closely than a conventional MLP connector and makes the visual tokenizer and LLM independently replaceable. Ovis1.5 and 1.6 improved high-resolution processing, data quality, and preference alignment.
Ovis2, released January 26, 2025, paired an AIMv2-based visual tokenizer with Qwen2.5 backbones and expanded the family from 1B to 34B, adding stronger OCR, video, multi-image, and reasoning behavior. Ovis2.5, released August 15, 2025, moved to a native-variable-resolution SigLIP 2 vision transformer and Qwen3 backbones. Its five-stage curriculum progresses from visual and multimodal pretraining through instruction tuning, DPO, and GRPO, enabling an optional reflective thinking mode.
Parrot tackles the issue of "multilingual erosion" in Multimodal Large Language Models (MLLMs), where models trained primarily on English-centric data struggle to understand and respond in other languages. It achieves this by using textual guidance to align visual tokens with language-specific embeddings, effectively enhancing the model's multilingual capabilities.
Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, Han-Jia Ye
Figure 5. PARROT aligns multilingual visual features through a projector, multilingual mixture of experts, and language model. Source paper, PDF p. 6. Figure notice.
ℹ️ More Information
Parrot builds upon the LLaVA framework, utilizing a pre-trained CLIP ViT-L/14 as the vision encoder and Qwen1.5-Chat as the LLM. The architecture consists of three main components: a vision encoder, a large language model (LLM), and a multilingual Mixture-of-Experts (MoE) module. The vision encoder processes the input image and generates visual features, which are then projected into the embedding space of the LLM using a learned projector. To address the multilingual challenge, Parrot introduces a novel textual guidance mechanism. It first calculates cross-attention between the class token of the visual features and the text embeddings derived from the input prompt. This cross-attention output is then fed into the MoE module's router, which predicts the probability of activating each language expert. Each expert is a specialized MLP trained to transform the English-biased visual embeddings into language-specific representations. The router selects the most relevant experts based on the input language, and their outputs are combined to generate the final language-specific visual embeddings. These embeddings are then combined with the original visual embeddings using a weighted sum, ensuring that the model retains its ability to process visual information effectively across different languages. This entire process allows Parrot to align visual tokens with textual embeddings at the language level, effectively mitigating multilingual erosion and enhancing the model's ability to understand and respond in multiple languages.
ConvLLaVA addresses the limitations of Vision Transformers (ViTs) in high-resolution Large Multimodal Models (LMMs) by replacing them with a hierarchical backbone, ConvNeXt, as the visual encoder. This architectural shift aims to reduce the computational burden caused by excessive visual tokens and quadratic complexity often associated with ViTs, especially when dealing with high-resolution images.
Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, Bo Zheng
Figure 1. ConvLLaVA uses a hierarchical ConvNeXt vision encoder to compress visual tokens between stages. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
ConvLLaVA leverages the inherent information compression capabilities of ConvNeXt, a hierarchical convolutional neural network. ConvLLaVA, unlike traditional LMMs that rely on ViTs, employs a five-stage ConvNeXt architecture as its visual encoder. This encoder progressively compresses visual information across its stages, significantly reducing the number of visual tokens generated compared to ViT. The architecture mirrors other popular general LMMs like LLaVA, Qwen-VL, and VILA, consisting of a vision encoder (ConvNeXt), a large language model (LLM - Vicuna in this case), and a vision-language projector (a two-layer MLP). The ConvNeXt encoder processes the input image and generates latent visual embeddings. These embeddings are then projected into the embedding space of the LLM by the vision-language projector. Finally, the projected visual embeddings are concatenated with the text embeddings generated by the LLM's tokenizer, and this combined input is fed into the LLM. The entire model is trained using a language modeling loss. To further enhance ConvLLaVA's performance, the authors introduce two key optimizations: firstly, they update the pretrained ConvNeXt weights instead of freezing them, allowing the model to adapt to high-resolution inputs and improve the quality of visual representations. Secondly, they introduce an additional ConvNeXt stage, effectively creating a five-stage architecture (ConvNeXt†) that further compresses visual information, enabling the model to handle even higher resolutions (up to 1536x1536) while generating a manageable number of visual tokens (576). This hierarchical compression approach, combined with the linear spatial complexity of ConvNeXt, significantly reduces the computational burden on the LLM compared to ViT-based models, making ConvLLaVA a more efficient and scalable solution for high-resolution multimodal tasks.
Phi-3-Vision brought chart, document, diagram, and image reasoning to a compact 4.2B-parameter model; Phi-3.5-Vision retained the small footprint while extending the family to multi-image inputs.
Microsoft Phi Team
Released: 2024-05-21
Architecture figure: The cited Phi-3 report contains no Phi-3-Vision architecture diagram.
ℹ️ More Information
Phi-3-Vision connects a vision encoder to the Phi-3 Mini language backbone through a learned projection and uses a dynamic high-resolution image pipeline, preserving detail through global and cropped image features while supporting a 128K-token context. Its 4.2B parameters make it substantially smaller than many contemporary VLMs.
Training combines carefully filtered public material, image-text and interleaved examples, synthetic reasoning data, and instruction data emphasizing OCR, charts, tables, diagrams, slides, and general visual reasoning. Microsoft released Phi-3-Vision on May 21, 2024. Phi-3.5-Vision followed on August 22, 2024, maintaining the same compact family design while improving visual quality, multilingual behavior, and multi-frame or multi-image reasoning.
CogVLM2 is a family of open-source visual language models designed to push the boundaries of image and video understanding. This new generation builds upon the success of previous CogVLM models, focusing on enhanced vision-language fusion, efficient high-resolution architecture, and broader modalities and applications.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong, Jie Tang
Figure 2. CogVLM2 processes high-resolution images and video frames through a ViT encoder, adapter, and visual-language decoder. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
CogVLM2 is a new generation visual language model designed for comprehensive image and video understanding. It leverages a powerful ViT encoder to extract visual features from high-resolution images or video sequences, which are then downsampled by a convolutional layer and aligned with linguistic representations through a SwiGLU module. This adapter efficiently bridges the visual and language modalities while preserving critical image information. The model then utilizes a visual expert architecture, integrating visual features into both the attention and FFN modules of the language decoder. This approach allows for deep vision-language fusion without compromising the model's inherent language capabilities. Notably, CogVLM2-Video extends this architecture to handle videos, incorporating timestamps alongside multi-frame inputs to enable temporal localization and question-answering capabilities. The CogVLM2 family has achieved state-of-the-art results on various benchmarks, including MMBench, MM-Vet, TextVQA, MVBench, and VCG-Bench, showcasing its versatility and effectiveness across a wide range of image and video understanding tasks.
Chameleon is an early-fusion, token-based model that represents text and images in one discrete vocabulary and processes arbitrary interleaved sequences with a single autoregressive transformer. The same architecture can understand and generate either modality without a separate vision encoder attached to a language model.
Chameleon Team
Released: 2024-05-16
Figure 1. Mixed-modal early-fusion tokenization and autoregressive generation. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
Chameleon converts images into discrete codes with a learned image tokenizer and combines them with text tokens in a unified vocabulary. A decoder-only transformer then models mixed sequences through next-token prediction, enabling text-to-text, image-to-text, text-to-image, and interleaved image-text generation within one architecture. Meta trains 7B- and 34B-parameter variants and introduces stabilization measures tailored to the mixed-modal setting, including query-key normalization, revised normalization placement, and modality-aware normalization.
Training proceeds through large-scale mixed-modal pretraining followed by an alignment stage. The pretraining run covers approximately 4.4 trillion tokens and mixes text-only sequences, image-text pairs, and interleaved documents. The alignment recipe uses supervised fine-tuning together with preference optimization to improve response quality and safety while preserving the model's ability to produce long mixed-modal sequences. The released artifacts provide inference support for the text-output models; the paper's full research system also evaluates image and mixed-modal generation.
The corpus combines licensed text, publicly available text, image-caption pairs, and interleaved web documents in which images occur within their textual context. Additional high-quality instruction and preference data are used during alignment. Keeping the modalities in their original interleaved order is essential: Chameleon is trained to model complete mixed-modal documents, rather than treating every image as an isolated attachment to a prompt.
PaliGemma is a compact, open-source vision-language model designed to be easily transferable to a diverse range of tasks. It combines a powerful SigLIP image encoder with the Gemma-2B language model, achieving strong performance on over 40 diverse tasks, including standard VLM benchmarks, remote-sensing, and segmentation. PaliGemma is pretrained using a multi-stage approach, focusing on maximizing the density of learning signal and providing different checkpoints with varying image resolutions. This versatile foundation model is easily fine-tuned for specific tasks and serves as a valuable tool for researchers and practitioners exploring the capabilities of VLMs.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, Xiaohua Zhai
Figure 1. PaliGemma connects a SigLIP image encoder to a Gemma autoregressive decoder language model. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
PaliGemma stands out as a highly versatile and transferable 3-billion parameter Vision-Language Model (VLM) meticulously designed for broad applicability across a wide spectrum of visual-language tasks. Its foundation lies in the integration of two powerful components: a SigLIP-So400m vision encoder, known for its exceptional performance despite its compact size, and the Gemma-2B language model, a pretrained autoregressive decoder-only model from the Gemma family. This combination enables PaliGemma to effectively process and understand both visual and textual information, making it adept at handling tasks ranging from image captioning and visual question answering to more specialized tasks like remote-sensing and segmentation. PaliGemma's architecture is streamlined and efficient. It uses a simple linear projection to align the visual features extracted by the SigLIP encoder with the vocabulary tokens of the Gemma language model, enabling seamless fusion of the two modalities. A key aspect of PaliGemma's training is the emphasis on "density of learning signal," prioritizing a broad range of skills and knowledge over achieving high zero-shot performance. This approach involves a multi-stage pretraining process that starts with unimodal pretraining of individual components using publicly available checkpoints, followed by extensive multimodal pretraining on a diverse mixture of large-scale vision-language tasks. Notably, PaliGemma deviates from the common practice of freezing the image encoder during pretraining, allowing it to learn spatial and relational understanding from complex tasks like captioning. To further enhance its capabilities, PaliGemma undergoes a resolution increase stage, where it is trained on higher-resolution images, enabling it to handle tasks that benefit from finer visual details. This multi-stage pretraining process results in a family of three PaliGemma checkpoints at varying image resolutions (224px, 448px, and 896px), each pretrained with broad visual knowledge. These checkpoints serve as strong base models that can be easily transferred to specific downstream tasks. PaliGemma's transferability is demonstrated through its impressive performance on over 30 academic benchmarks, including those involving multiple images, such as NLVR2 and short-video understanding tasks. The model's ability to adapt quickly to new tasks with minimal fine-tuning highlights its versatility and makes it a valuable tool for exploring and advancing the capabilities of VLMs. Furthermore, the model's open-source nature, along with its straightforward architecture and training recipe, encourages further research and experimentation within the VLM community, driving progress towards more powerful and general-purpose multimodal AI systems.
xGen-MM (BLIP-3): An Open-Source Framework for Building Powerful and Responsible Large Multimodal Models
xGen-MM (BLIP-3) is a comprehensive framework developed by Salesforce for training a series of open-source large multimodal models (LMMs) designed to excel in a variety of visual language tasks. It provides meticulously curated datasets, a streamlined training recipe, model architectures, and a suite of open LMMs capable of performing various visual language tasks. xGen-MM focuses on scalability, using a simplified architecture and a unified training objective to enable training on larger, more diverse datasets. The framework also includes a safety-tuned model to mitigate harmful behaviors and promote responsible AI development.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, Ran Xu
Figure 2. BLIP-3 feeds interleaved visual and text tokens through a scalable vision-token sampler into a pretrained language model. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
xGen-MM (BLIP-3), short for xGen-MultiModal, addresses limitations of previous open-source efforts by providing a complete ecosystem for LMM development. Central to its approach is the utilization of diverse, large-scale, and high-quality multimodal data, which enables xGen-MM to achieve competitive performance against both open-source and proprietary LMMs. Instead of relying on the intricate Q-Former architecture and multiple training objectives used in its predecessor, BLIP-2, xGen-MM streamlines the process by employing a more scalable vision token sampler (perceiver resampler) and unifying the training objective to a single auto-regressive loss on text tokens. This simplification enables larger-scale training and focuses the model on effectively learning from the rich multimodal context. Furthermore, xGen-MM incorporates safety measures, introducing a safety-tuned model with DPO to mitigate potential harmful behaviors like hallucinations and promote responsible AI development. By open-sourcing its models, datasets, and fine-tuning code, xGen-MM aims to empower the research community and foster advancements in the field of LMMs, making these powerful tools more accessible and encouraging further exploration of their capabilities.
MANTIS is a family of open-source large multimodal models that demonstrate state-of-the-art performance on multi-image visual language tasks. By focusing on instruction tuning with a carefully curated multi-image dataset, MANTIS achieves superior results using significantly less data than models trained with massive web datasets. This efficient approach opens new avenues for developing powerful multi-image LMMs with limited resources.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, Wenhu Chen
Architecture figure: The paper contains capability examples, dataset statistics and illustrations, and case studies, none of which is a model architecture or system diagram.
ℹ️ More Information
Mantis: a powerful and efficient multi-image Large Multimodal Models (LMMs), demonstrating that massive pre-training on noisy web data is not the only path towards achieving state-of-the-art performance in complex visual-language tasks. Instead, MANTIS focuses on instruction tuning using high-quality, academic-level data, achieving remarkable results on various multi-image benchmarks while using significantly less data than its counterparts. Central to MANTIS's success is the meticulously curated MANTIS-INSTRUCT dataset, a collection of 721K multi-image instruction data carefully designed to instill four crucial skills: co-reference, comparison, reasoning, and temporal understanding. These skills equip MANTIS with a comprehensive toolkit for tackling the challenges of multi-image understanding. Co-reference enables the model to understand references like "second image" in natural language and correctly identify the corresponding image within the input. Comparison fosters the ability to analyze and identify subtle differences and commonalities between multiple images, a skill crucial for tasks like visual similarity assessment and difference description. Reasoning empowers the model to go beyond simple comparisons and make complex inferences by combining its world knowledge with the information extracted from multiple images, allowing it to solve intricate logical reasoning puzzles and answer challenging multi-image questions. Finally, temporal understanding equips MANTIS with the capability to process and understand image sequences, capturing the dynamic aspects of videos, comics, and other temporal visual data. MANTIS leverages a simple yet effective architecture based on existing pre-trained LLMs like LLaMA-3 and vision transformer encoders from CLIP or SigLIP. A multimodal projector, similar to the one used in LLaVA, aligns the visual embeddings with the text embeddings, facilitating their seamless integration within the LLM. This streamlined approach avoids the complexity of previous architectures like Q-Former while retaining high performance. Extensive evaluations on five multi-image benchmarks, including NLVR2, QBench, BLINK, MVBench, and a newly curated Mantis-Eval dataset, demonstrate MANTIS's superior performance, exceeding existing open-source LMMs and even matching the results of the powerful GPT-4V. Notably, MANTIS surpasses Idefics2-8B, a model pre-trained on 200x larger interleaved multi-image data, showcasing the effectiveness of instruction tuning with high-quality academic-level data. Furthermore, MANTIS retains strong single-image performance on par with existing state-of-the-art models, demonstrating its versatility and adaptability. MANTIS's impressive results, combined with its efficient training and open-source nature, offer a compelling alternative to traditional pre-training-heavy approaches, opening new possibilities for researchers and practitioners seeking to develop powerful and versatile multi-image LMMs with minimal computational resources.
Moondream is a compact (1.9B parameters) vision-language model (VLM) that prioritizes practical usability and accessibility, offering features like structured output (JSON, XML, Markdown, CSV), improved OCR, and a novel experimental Gaze Detection capability, while maintaining fast performance and ease of deployment.
Architecture figure: The rolling model card and official repository provide implementation details but no legitimate architecture or system figure.
ℹ️ More Information
Moondream distinguishes itself by being exceptionally small (1.9B parameters) while supporting a wide range of functionalities typically found in larger, more specialized models. The architecture is not explicitly detailed in the provided text, but it mentions improvements to the "vision layer" for better OCR performance. This suggests a structure where visual input is processed by a vision encoder, and then integrated with a language model. The key feature is its ability to perform multiple Vision AI tasks ("capabilities") within a single, unified model, including: object detection, captioning, visual querying, pointing (x,y coordinate retrieval), and the newly added gaze detection. The model also newly supports structured output formats, generating outputs directly as JSON, XML, Markdown, or CSV, making integration with applications much easier. The "Gaze Detection" capability is specifically highlighted as a novel and experimental feature, indicating a focus on real-world applications beyond standard benchmarks. The training data and process are not thoroughly described, although the text notes increased training on "document querying and understanding" for OCR enhancement. The model's creators express a cautious approach to benchmarks, acknowledging their limitations and potential for manipulation, yet also highlight improved benchmark scores in this release, suggesting a balance between practical utility and measurable performance. It does not rely on external apis.
IDEFICS2, an 8B parameter open-source vision-language model, efficiently processes interleaved image and text sequences by combining a SigLIP vision encoder, a Mistral-7B LLM, and a Perceiver pooling layer with MLP projection for robust text encoding, excelling in tasks like OCR and document understanding.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor Sanh
Figure 2. Idefics2 maps vision-encoder features into interleaved visual tokens consumed by an autoregressive language model. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
IDEFICS2 is an 8B parameter open-source vision-language model adept at handling interleaved image and text sequences. IDEFICS2 utilizes a vision-language architecture designed for efficient processing of image and text sequences. It employs the SigLIP model as the vision encoder, extracting features from images in their native resolutions and aspect ratios. The Mistral-7B model serves as the LLM backbone, providing language understanding and generation capabilities. For text encoding, IDEFICS2 leverages a Perceiver pooling layer followed by an MLP projection to integrate visual features with the LLM's embedding space. This combination of vision encoder, LLM, and text encoder enables IDEFICS2 to handle various multimodal tasks, with a particular focus on OCR and document understanding. The model is trained on a diverse dataset encompassing OBELICS, LAION Coco, and PMD, with additional data for OCR tasks. Fine-tuning is performed on instruction datasets like The Cauldron and OpenHermes-2.5.
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
InternLM-XComposer2-4KHD, building on its predecessor, pioneers high-resolution image handling in LVLMs by employing dynamic resolution with automatic patch configuration, adapting to resolutions from 336 pixels up to 4K HD for enhanced visual understanding without distortion.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang
Figure 4. InternLM-XComposer2-4KHD dynamically partitions high-resolution images into local patches while retaining a global thumbnail. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
InternLM-XComposer2-4KHD: Cutting-edge Large Vision-Language Model (LVLM) designed to handle ultra-high resolutions, up to 4K HD and beyond, while also supporting diverse resolutions from 336 pixels. The model builds upon the InternLM-XComposer2 architecture, incorporating a novel dynamic resolution with automatic patch configuration technique. This allows the model to dynamically adjust patch layouts and counts based on the input image's aspect ratio, enabling efficient processing of high-resolution images while preserving their original proportions. To address potential ambiguity arising from variable patch configurations, a newline token is introduced to delineate rows of patch tokens, significantly improving performance. InternLM-XComposer2-4KHD is pre-trained on a diverse dataset, including image-caption pairs, concept knowledge, and OCR datasets, focusing on enhancing high-resolution and structural image understanding. Supervised fine-tuning further incorporates a mixed-resolution strategy, employing higher resolution for tasks requiring fine-grained detail, like HD-OCR tasks, and dynamically adjusted resolution for other tasks. This approach enables the model to excel in both high-resolution scenarios and general vision-language understanding tasks.
MM1 is Apple's family of dense and mixture-of-experts multimodal language models, built from a pretrained image encoder, a vision-language connector, and a decoder-only language model. Its controlled ablations show that visual capacity and pretraining data composition matter substantially more than connector complexity.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, et al.
Released: 2024-03-14
Architecture figure: MM1 presents an architecture design space and ablations, but no definitive final-model diagram.
ℹ️ More Information
MM1 combines a pretrained vision encoder with a vision-language connector that maps image features into tokens consumed by a decoder-only language model. Apple studies dense models from 3B to 30B parameters alongside mixture-of-experts variants. Controlled experiments vary the image encoder, input resolution, number of visual tokens, connector, and language-model initialization. The results identify the image encoder, resolution, and visual-token budget as consequential choices, while the difference among connector designs is comparatively small once the other components are well configured.
Pretraining uses a deliberate mixture of image-caption pairs, interleaved image-text documents, and text-only sequences. The paper finds that caption data supports strong zero-shot behavior, interleaved documents are especially important for few-shot and in-context learning, and retaining text-only data protects language capability. The scaled recipe gives MM1 multi-image reasoning and few-shot chain-of-thought behavior before supervised fine-tuning; subsequent multimodal instruction tuning improves conversational and benchmark performance.
The pretraining corpus is organized into three corresponding data families: web-scale image-caption pairs, long interleaved documents containing multiple images and surrounding text, and large text-only corpora. The caption component includes both original and synthetically generated captions. Because the paper's principal contribution is a controlled study of mixture and architecture choices, it reports data by source type and experimental mixture rather than presenting a separately released named dataset.
DeepSeek-VL, utilizing a hybrid vision encoder combining SigLIP-L and SAM-B, excels in real-world vision-language understanding by efficiently processing high-resolution images and integrating extracted features with a DeepSeek LLM backbone through a two-layer hybrid MLP adapter.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, Chong Ruan
Figure 3. DeepSeek-VL trains its hybrid vision encoder, vision-language adaptor, and language model across three progressive stages. Source paper, PDF p. 12. Figure notice.
ℹ️ More Information
DeepSeek-VL: Employs a hybrid vision encoder architecture, fusing a SigLIP-L encoder for semantic understanding with a SAM-B encoder for high-resolution detail extraction. This allows for efficient processing of 1024x1024 images while capturing both global and fine-grained visual features. A two-layer hybrid MLP adapter then integrates these features with the DeepSeek LLM backbone. The model is pre-trained on a diverse dataset encompassing web screenshots, PDFs, OCR, charts, and knowledge-based content from sources like Common Crawl, Web Code, E-books, and arXiv articles. This pretraining is further refined using a curated instruction-tuning dataset based on real user scenarios and categorized into a comprehensive taxonomy covering recognition, conversion, analysis, reasoning, evaluation, and safety tasks. By combining this diverse data with its unique architecture and fusion strategies, DeepSeek-VL aims to deliver robust performance across a wide range of real-world vision-language applications.
AnyGPT treats text, speech, images, and music as token sequences in one autoregressive language model, showing that new modalities can be incorporated largely through tokenization and data construction rather than architectural surgery.
OpenMOSS
Released: 2024-02-19
Figure 1. Unified discrete sequence modeling across speech, text, image, and music. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
AnyGPT converts every supported modality into discrete tokens and trains a LLaMA-family decoder with the ordinary next-token objective. Modality-specific tokenizers encode and reconstruct speech, images, and music, while special tokens delimit each modality inside interleaved sequences.
Training begins with multimodal alignment data that connects modality tokens to text, then uses the authors' 108,000-example any-to-any instruction dataset for multi-turn conversations containing arbitrary input and output modalities. The central result is architectural simplicity: speech, image, music, and text generation share one vocabulary-level interface and one autoregressive backbone, without separate cross-attention modules or task-specific generation heads.
SPHINX-X refines multi-modal large language models by streamlining its architecture to use two visual encoders, CLIP-ConvNeXt and DINOv2, and implementing an efficient single-stage training process for enhanced performance across diverse multi-modal tasks.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, Yu Qiao
Figure 3. SPHINX-X combines mixed visual experts, padded-tile skip tokens, high-resolution partitioning, and unified training. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
SPHINX-X: Represents an advanced iteration in the development of Multi-modal Large Language Models (MLLM), building upon its predecessor, SPHINX, by optimizing both architecture and training efficiency. The core modifications introduced in SPHINX-X include the elimination of redundant visual encoders, the incorporation of learnable skip tokens to bypass fully-padded sub-images, and the simplification of the multi-stage training process into a singular, all-in-one training paradigm. This approach is designed to enhance the model's efficiency and effectiveness across a broad spectrum of multi-modal tasks. The architecture of SPHINX-X retains two key visual encoders, CLIP-ConvNeXt and DINOv2, ensuring robust text-image alignment capabilities, especially for high-resolution images and varied aspect ratios. This streamlined model architecture enables a unified encoding approach for both vision and text, emphasizing scalable and efficient training methodologies. The training strategy is comprehensive, directly engaging all model parameters across a wide-ranging multi-modal dataset, which encompasses public resources covering language, vision, and vision-language tasks. Additionally, SPHINX-X enriches this dataset with specially curated OCR-intensive and Set-of-Mark datasets to further extend the model's versatility and generalization capabilities. The datasets utilized in SPHINX-X aim to foster a deep, comprehensive understanding across multiple domains, enhancing the model's performance in OCR, document layout detection, and fine-grained multi-modal understanding. By training over various base Large Language Models (LLMs) with different parameter sizes and multilingual capabilities, SPHINX-X achieves a spectrum of MLLMs that showcase a strong correlation between multi-modal performance and the scales of data and parameters involved. This strategy allows SPHINX-X to set a new benchmark in multi-modal large language model performance, significantly advancing the field's capabilities in handling complex, multi-domain tasks.
LLaVA-NeXT advances on LLaVA-1.5 by incorporating high-resolution image processing, enhancing visual reasoning and OCR capabilities, while maintaining a data-efficient design through knowledge transfer from its predecessor and a refined training process.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee
Official architecture diagram. LLaVA-NeXT's AnyRes scheme partitions high-resolution images into a configurable grid of local views. Primary source. Figure notice.
ℹ️ More Information
LLaVA-NeXT: Represents a significant step forward in the evolution of large language models with visual capabilities, building upon the foundations laid by LLaVA-1.5. This model introduces several enhancements aimed at improving image resolution, visual reasoning, optical character recognition (OCR), and the integration of world knowledge, all while retaining the minimalist and data-efficient design of its predecessor. The architecture of LLaVA-NeXT is optimized for high performance, supporting input image resolutions up to 672x672, 336x1344, and 1344x336 pixels. This improvement facilitates a more detailed visual perception, which, coupled with an enhanced visual instruction tuning data mixture, significantly bolsters the model's reasoning and OCR capabilities. Furthermore, LLaVA-NeXT achieves efficient deployment through the use of SGLang, a feature that underscores its design's focus on performance and data efficiency.Training LLaVA-NeXT requires less than 1 million visual instruction tuning samples, leveraging the pre-trained connector from LLaVA-1.5 for efficient knowledge transfer. The training process, remarkably swift, utilizes 32 A100 GPUs and completes in approximately one day, a testament to the model's efficient design and deployment strategy. The alignment techniques in LLaVA-NeXT are particularly noteworthy, utilizing high-resolution images and a high-quality data mixture to enhance the model's capabilities in visual conversation and instruction following. The model's use of dynamic high-resolution techniques, known as 'AnyRes', allows for effective handling of images with varying resolutions, improving the model's overall visual understanding.The datasets employed in training LLaVA-NeXT, including LAION-GPT-V, ShareGPT-4V, DocVQA, SynDog-EN, ChartQA, DVQA, and AI2D, are meticulously chosen to augment the model's visual reasoning, OCR capabilities, and comprehension of charts and diagrams. This strategic selection aims to elevate the model's performance across a wide range of multimodal tasks, emphasizing its enhanced ability to process and understand complex visual information. Through these improvements, LLaVA-NeXT sets a new benchmark for models at the intersection of language and vision, offering unprecedented capabilities in visual reasoning, OCR, and the application of world knowledge in multimodal contexts.
MiniCPM-V is a series of efficient Multimodal Large Language Models (MLLMs) designed for deployment on end-side devices like mobile phones and personal computers. The latest iteration, MiniCPM-Llama3-V 2.5, achieves performance comparable to GPT-4V, Gemini Pro, and Claude 3 while being significantly smaller and more efficient, demonstrating the feasibility of deploying powerful MLLMs on resource-constrained devices.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun
Figure 3. MiniCPM-V combines a visual encoder, shared compression layer, language model, and adaptive high-resolution encoding. Source paper, PDF p. 5. Figure notice.
Figure 1. MiniCPM-V 4.5 architecture; a later release in this family. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
MiniCPM-V focuses on achieving a balance between performance and efficiency, crucial for real-world applications on end-side devices. The model architecture consists of three key modules: a visual encoder, a compression layer, and an LLM. For the visual encoder, MiniCPM-V utilizes SigLIP SoViT-400m/14, chosen for its efficiency and effectiveness. To handle high-resolution images with varying aspect ratios, the model employs an adaptive visual encoding approach. This involves dividing the input image into slices that better match the ViT's pre-training settings in terms of resolution and aspect ratio. A score function is used to select the optimal partition of slices, ensuring a good match with the ViT's pre-training. Each slice is then resized proportionally and interpolated to fit the ViT's input size. After visual encoding, each slice is represented by 1024 tokens, resulting in a large number of tokens for multiple slices. To address this, a token compression module is employed, using one-layer cross-attention with a moderate number of queries to compress the visual tokens of each slice into 64 or 96 tokens. This significantly reduces the computational cost and memory footprint, making the model suitable for end-side deployment. A spatial schema is also introduced to indicate the position of each slice relative to the whole image, further enhancing the model's understanding of spatial relationships. The compressed visual tokens, along with the text input, are then fed into the LLM, which is based on MiniCPM 2B for earlier versions and Llama3-Instruct 8B for MiniCPM-Llama3-V 2.5. The training process consists of three phases: pre-training, supervised fine-tuning, and RLAIF-V (Reinforcement Learning from AI Feedback for Vision). Pre-training aims to align the visual modules with the LLM's input space and learn foundational multimodal knowledge. It involves three stages: warming up the compression layer, extending the input resolution of the visual encoder, and training the visual modules with the adaptive visual encoding strategy. Supervised fine-tuning further enhances the model's knowledge and interaction capabilities using high-quality visual question answering datasets. The SFT data is categorized into two parts: one focusing on basic recognition capabilities and the other on generating detailed responses and following instructions. Finally, RLAIF-V is employed to mitigate the hallucination problem common in MLLMs. This involves generating multiple responses for an instruction, evaluating their correctness using a divide-and-conquer strategy, and then optimizing the model using Direct Preference Optimization (DPO) on a preference dataset. MiniCPM-V demonstrates impressive performance on various benchmarks, including general multimodal benchmarks, OCR benchmarks, and multilingual multimodal interaction, while being efficient enough for deployment on mobile phones. This highlights the potential of pushing the boundaries of end-side MLLMs and bringing powerful AI capabilities to user devices.
MiniCPM-V 4.5, released on August 26, 2025 (technical report: September 16), keeps the family's efficient visual pathway and compact language backbone while introducing a unified 3D-Resampler for highly compressed image and video encoding, a unified document-knowledge and text-recognition recipe, and hybrid reinforcement learning for short and long reasoning modes. MiniCPM-V 4.6, released in May 2026, returns to an approximately 1B edge footprint with SigLIP2-400M, Qwen3.5-0.8B, LLaVA-UHD v4, and switchable 4x/16x visual-token compression. Both are lineage updates rather than separate top-level architectures. 4.5 paper · 4.6 model · Repository MiniCPM-V 4.7 (October 12, 2026) adds a 35B/A3B MoE variant alongside the compact model. Early compression inside the vision encoder supports mixed 4x/16x visual-token reduction. Lookahead Expert Scheduling prefetches experts without changing the router's chosen computation; layer-wise quantization and the October 15 DSpark heads target local inference. The release links Self-Replay Fine-Tuning for downstream adaptation but does not disclose its complete pretraining mixture. The figures above depict the original family and 4.5, not a new 4.7 schematic; the reviewed 4.7 card publishes deployment and evaluation graphics rather than a model architecture figure. 4.7 model · Release history
MouSi pushes the boundaries of VLMs by incorporating multiple visual experts like CLIP and SAM, utilizing a poly-expert fusion network to combine their outputs and interface with powerful LLMs like Vicuna, thereby enabling a more comprehensive understanding and processing of visual information.
Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
Figure 2. MouSi integrates heterogeneous visual experts through a fusion network and projector into a language model. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
MouSi: Represents an innovative approach to Vision-Language Models (VLMs) by integrating multiple visual experts into a unified architecture, aiming to surpass the limitations inherent to models reliant on a singular visual component. This architecture leverages a poly-expert fusion network, which incorporates outputs from varied visual experts, such as CLIP for image-text matching and SAM for image segmentation. This network facilitates an efficient interface with pre-trained Large Language Models (LLMs), notably utilizing a model like Vicuna v1.5. MouSi distinguishes itself by employing a multi-expert visual encoder that selects relevant experts from a pool, and it features two types of poly-expert fusion networks: a projection fusion method and a Q-Former fusion method. The training methodology of MouSi is characterized by a two-phase approach. Initially, during the pre-training phase, both the text-only LLM and the multi-expert encoder are kept static, with the training focus squarely on the poly-visual fusion network. Subsequently, in the fine-tuning phase, the LLM is activated for training in conjunction with the poly-visual fusion network, using high-quality supervised datasets. This methodology ensures that MouSi benefits from robust pre-existing language models while simultaneously enhancing its capability to process and integrate complex visual information. For alignment and fusion of the multimodal inputs, MouSi employs its poly-expert fusion network to amalgamate the outputs from the various visual experts, aligning them with the vision input tokens. This alignment is critical for encoding vision and text cohesively, a process facilitated by either the projection fusion method or the more complex Q-Former fusion method. These methods allow for the effective compression of multi-channel visual information into a format that can be efficiently processed alongside textual data. The datasets used in MouSi's training regimen include LCS-558K and the LAION-CC-SBU collection for pre-training, aimed at aligning text and image representation spaces, and diverse, high-quality SFT datasets for fine-tuning, enhancing the model's performance across a broad spectrum of multimodal tasks.
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
InternLM-XComposer2 excels in free-form text-image composition and comprehension by connecting a CLIP pre-trained vision encoder with the powerful InternLM-2 LLM using a novel Partial LoRA module, enabling efficient alignment of visual and language tokens for enhanced multimodal understanding.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, Jiaqi Wang
Figure 2. InternLM-XComposer2 applies Partial-LoRA only to visual tokens while preserving language-only behavior. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
InternLM-XComposer2: This model introduces a sophisticated architecture that leverages a vision encoder and a Large Language Model (LLM), interconnected through a Partial Low-Rank Adaptation (LoRA) module. This innovative setup allows InternLM-XComposer2 to effectively process both images and text, employing visual tokens generated by the vision encoder alongside language tokens derived from the tokenized text. The vision encoder, pre-trained using CLIP for image-language contrastive learning, and InternLM-2, which serves as the LLM with multi-lingual capabilities, are key components of this architecture. The Partial LoRA module distinguishes itself by aligning visual and language tokens through low-rank adaptation applied specifically to visual tokens, enhancing the model's multimodal understanding and processing efficiency. The training methodology of InternLM-XComposer2 is multifaceted, focusing on fine-tuning the vision encoder and Partial LoRA to align visual tokens with the LLM across various datasets. This process involves general semantic alignment, world knowledge alignment, and vision capability enhancement to refine the model's ability to interpret image information and compose text-image content. Supervised fine-tuning further includes multi-task training and free-form text-image composition, aiming to optimize the model's performance in leveraging image information for comprehensive text-image generation and understanding. Alignment techniques and fusion methods in InternLM-XComposer2 utilize the Partial LoRA module for the effective integration of different modalities, thereby enriching the LLM with modality-specific knowledge while preserving its inherent capabilities. This selective enhancement of visual tokens through Partial LoRA enables the model to exhibit robust performance across visual and textual domains, facilitating detailed perception, logical reasoning, and extensive knowledge integration in multimodal understanding. The model employs a diverse array of datasets, including ShareGPT4V-PT, COCO, Nocaps, TextCaps, and many others, for pre-training and supervised fine-tuning. These datasets serve to equip InternLM-XComposer2 with a broad range of capabilities, including general semantic alignment, world knowledge alignment, vision capability enhancement, and the facilitation of free-form text-image composition, marking a significant advancement in the field of vision-language large models.
MoE-LLaVA introduces a novel approach by incorporating Mixture of Experts (MoE) within a large vision-language model, using learnable routers to selectively activate expert modules for processing specific tokens, thereby enhancing efficiency and enabling nuanced understanding of multimodal inputs.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Munan Ning, Li Yuan
Figure 2. MoE-LLaVA routes multimodal tokens through sparse experts added to a vision-encoder, projector, and language-model backbone. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
MoE-LLaVA: Represents an innovative leap in the development of large vision-language models through the integration of Mixture of Experts (MoE) within a sophisticated architectural framework. This model is characterized by its sparse design, wherein individual tokens are directed towards a selection of experts based on learnable routers, ensuring that only the top-k experts are activated for any given token's processing. Such an approach not only enhances the model's efficiency but also its capability to handle diverse and complex data inputs by leveraging specialized processing paths for different types of information. At the heart of MoE-LLaVA's architecture are several critical components, including a vision encoder, a visual projection MLP layer, word embedding layers, multi-head self-attention blocks, feed-forward neural networks, and notably, the MoE blocks themselves. These elements are seamlessly integrated through the use of layer normalization and residual connections, establishing a robust and adaptable framework capable of deep multimodal understanding. The training methodology for MoE-LLaVA is meticulously structured in three stages, each designed to gradually enhance the model's proficiency in integrating and processing visual and textual data. This includes initial adaptation of image tokens, training of all LLM parameters excluding the vision encoder, and specialized training of the MoE layers, with the latter utilizing initialization weights from previous stages for optimal performance. Alignment techniques and fusion methods employed by MoE-LLaVA are pivotal in achieving a harmonious integration of text and image modalities. By utilizing learnable routers to dynamically allocate tokens to the most apt experts and subsequently processing these through a combination of LLM and MoE blocks, the model achieves a nuanced understanding of multimodal inputs. The datasets employed throughout the training phases—ranging from LLaVA-PT for pretraining to Hybrid-FT for multimodal instruction tuning, and LLaVA-FT for fine-tuning the MoE layers—further underscore the model's ability to refine its understanding across a broad spectrum of multimodal tasks. This strategic deployment of diverse datasets not only facilitates a comprehensive tuning of the model's capabilities but also underscores its potential in advancing the field of vision-language processing.
moondream1 and moondream2 are vision-language models with moondream2 building upon moondream1's SigLIP vision encoder and Phi-1.5 language backbone by incorporating an MLP projector for enhanced visual and textual representation alignment.
@vikhyatk
Architecture figure: Official repository and model cards describe the implementation but provide no legitimate family architecture or system figure.
ℹ️ More Information
moondream1 and moondream2: A series of vision-language models. moondream1 is a 1.6B parameter model that leverages SigLIP as the vision encoder and Phi-1.5 as the language backbone, trained on the LLaVA dataset. moondream2 expands upon this foundation, utilizing a 1.86B parameter model initialized with weights from SigLIP and Phi-1.5. It incorporates an MLP projector to bridge the visual and textual representations, potentially leading to enhanced vision-language alignment and improved performance across various tasks.
FireLLaVA breaks new ground by combining the CodeLlama 34B Instruct model for advanced language understanding with a CLIP-ViT-based visual interpretation component, training on a unique dataset incorporating bounding box labels and captions to excel in visual language conversations.
Architecture figure: A generic LLaVA diagram would not document FireLLaVA's own contribution and should not be substituted.
ℹ️ More Information
FireLLaVA: As the first of its kind within the LLaVA lineage, FireLLaVA integrates a dual-component architecture that leverages the CodeLlama 34B Instruct model for nuanced language understanding and a visual interpretation component akin to OpenAI's CLIP-ViT. This model is distinctive for its use of bounding box labels and captions to generate visual language conversations, a method that underscores its innovative approach to multi-modal training. The training regimen for FireLLaVA is meticulously crafted, utilizing 588K lines of visual question answering and conversation data. This dataset amalgamates permissive original LLaVA data with newly generated data from Fireworks.ai, demonstrating a unique approach to instruction fine-tuning that enhances the model's ability to comprehend and articulate responses that bridge textual and visual inputs. The integration of bounding box labels and captions not only serves as a mechanism for generating training data but also facilitates the alignment of text and image data, a crucial step in achieving coherent multi-modal understanding. Although the specific methods employed for alignment fusion within FireLLaVA's architecture remain under-described, it is inferred that embedding fusion plays a critical role in synthesizing vision and text inputs. By drawing on original LLaVA training materials and Fireworks.ai's proprietary data, FireLLaVA sets a precedent for the development of VLMs capable of navigating the complexities of commercial applications. This model embodies a significant advancement in the field of visual language modeling, offering insights into the potential of OSS models to contribute to the evolving landscape of multi-modal AI research and deployment.
COSMO presents a streamlined multimodal framework by combining a Vision Transformer with a partitioned Large Language Model, optimizing the processing of interleaved data sequences through a combination of language modeling and contrastive loss functions.
Alex Jinpeng Wang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin, Zhengyuan Yang, Lijuan Wang, Mike Zheng Shou
Figure 3. CosMo handles image and video inputs through a language model trained with contrastive and language-modeling objectives. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
COSMO: This framework is distinctive for its architecture that merges a visual encoder, leveraging the Vision Transformer (ViT) from Open-CLIP, with a partitioned Large Language Model (LLM). The LLM is systematically divided into segments dedicated to unimodal text processing and multimodal data handling, aiming to streamline the overall processing of interleaved data sequences. The introduction of an additional contrastive loss component stands out as a strategy to improve performance across both classification and generation tasks. Training of COSMO is carried out through a unique combination of language modeling loss and contrastive loss, focusing on the efficient management of interleaved text and visual sequences. This process is optimized with the use of the AdamW optimizer, a cosine learning rate schedule, and the implementation of DeepSpeed fp16 precision, distributed across 128 NVIDIA V100 GPUs. The partitioning strategy of the LLM into dedicated components is a testament to the framework's commitment to computational efficiency and efficacy in handling extensive data sequences. The model's alignment techniques are notably advanced, featuring a learnable query that facilitates global attention across all tokens, alongside an additional query for Text Fusion Layers, optimizing the model's understanding of token sets and enhancing image-text alignment through contrastive loss. The gated cross-attention layers for multimodal fusion introduce a significant reduction in learnable parameters by introducing bottlenecks in input and output feature channels. This method of lightweight fusion is pivotal in integrating visual information for precise next-token prediction. COSMO's training leverages a diverse array of datasets including CC3M, SBU, LAION400M, DataComp1B, MMC4, WebVid, and Howto-Interlink7M. The introduction of Howto-Interlink7M, in particular, underscores the model's innovative approach to improving video-language understanding through high-quality annotated captions, demonstrating its effectiveness across 14 diverse downstream tasks.
TinyGPT-V prioritizes efficiency in multimodal large language models by combining a compact EVA-ViT visual encoder with linear projection layers and the powerful Phi-2 language model, achieving robust performance in vision-language tasks despite its smaller size.
Zhengqing Yuan, Zhaoxu Li, Lichao Sun
Figure 4. TinyGPT-V projects frozen visual-backbone and Q-Former outputs through two linear layers into Phi-2. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
TinyGPT-V: introduces a compact yet powerful architecture tailored for efficient multimodal large language model applications, leveraging small backbones for streamlined processing. This model integrates a visual encoder, specifically EVA of Vision Transformer (ViT), with linear projection layers and the Phi-2 language model, constituting its core components. The visual encoder remains inactive during training, focusing on image resolution adjustments across various stages to enhance image understanding. The linear projection layers, particularly with the incorporation of the Q-Former layer from BLIP-2, aim to efficiently embed visual features into the language model, reducing the number of parameters needing training. The Phi-2 large language model backbone, a 2.7 billion-parameter model, excels in reasoning and language comprehension, effectively handling vision-language operations including spatial location tasks through textual bounding box depictions. The training of TinyGPT-V unfolds across four stages: warm-up, pre-training, instruction fine-tuning, and multi-task learning. Each stage is meticulously designed to progressively enhance the model's capabilities in understanding and generating language based on visual inputs, with a special emphasis on human-like learning and conversation abilities in later stages. The use of datasets such as LAION, CC3M, SBU, and more, across these stages, supports the model's development in vision-language understanding, generation, and task execution like visual question answering and image captioning. A noteworthy aspect of TinyGPT-V's architecture is the implementation of normalization techniques and LoRA (Low-Rank Adaptation) to stabilize training and optimize the model's performance across different modalities. Addressing challenges like NaN or INF values in multimodal data computation, these mechanisms enhance training stability and efficiency. Furthermore, the model employs a multi-task instruction template to manage task ambiguity, utilizing MiniGPT-v2 tokens for task-specific instructions, facilitating precise and accurate task execution.
MobileVLM offers a mobile-optimized vision-language model that combines a CLIP ViT-L/14 visual encoder with the efficient MobileLLaMA language model and a Lightweight Downsample Projector (LDP), enabling effective multimodal processing and alignment within the constraints of mobile devices.
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, Chunhua Shen
Figure 1. MobileVLM connects a visual encoder to MobileLLaMA through a lightweight downsample projector. Source paper, PDF p. 5. Figure notice.
ℹ️ More Information
MobileVLM: Introduces a compact yet robust architecture designed to facilitate efficient vision-language tasks on mobile devices, distinguishing itself through a blend of specialized components and a streamlined training methodology tailored for edge computing environments. At its core, MobileVLM integrates a visual encoder based on the CLIP ViT-L/14 model with a resolution of 336x336, MobileLLaMA—a language model optimized for mobile devices, and a Lightweight Downsample Projector (LDP) that bridges the gap between visual and textual data with minimal computational overhead. This synergy between components ensures that MobileVLM can process and align multimodal inputs effectively, making it well-suited for mobile applications where resource efficiency is paramount. The training regimen for MobileVLM unfolds in three distinct phases, each contributing uniquely to the model's development. Initially, the language model undergoes pre-training using the text-centric RedPajama v1 dataset, laying a solid linguistic foundation. Subsequent supervised fine-tuning leverages multi-turn dialogues between humans and ChatGPT, refining the model's conversational abilities. The final stage involves training the integrated vision-language model on diverse multimodal datasets, equipping MobileVLM with the capacity to interpret and respond to both visual and textual stimuli. This comprehensive training approach ensures that MobileVLM achieves a balance between performance and efficiency, making it adept at handling complex vision-language interactions on mobile platforms. Central to MobileVLM's effectiveness is the Lightweight Downsample Projector (LDP), a novel component designed for the efficient alignment of visual and textual features. By employing mobile-friendly operations such as depth-wise convolution, LDP manages to downsample visual tokens to match the language model's input dimensions, preserving spatial information while minimizing computational demands. This alignment mechanism, in conjunction with the efficient fusion of vision and text embeddings, enables MobileVLM to maintain high levels of accuracy and responsiveness in mobile environments. Through the use of carefully selected datasets, including RedPajama v1 for linguistic pre-training and various multimodal datasets for comprehensive vision-language modeling, MobileVLM showcases its capability to navigate the challenges of mobile-based vision-language tasks with remarkable efficiency.
Alpha-CLIP builds upon the CLIP model by incorporating region awareness through the addition of an alpha channel to the image encoder, trained on millions of RGBA region-text pairs, enabling precise control over image emphasis and enhancing performance across various tasks requiring detailed spatial understanding.
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang
Figure 3. Alpha-CLIP extends CLIP with an alpha channel that focuses visual encoding on a specified region. Source paper, PDF p. 4. Figure notice.
ℹ️ More Information
Alpha-CLIP: Introduces a significant enhancement to the original CLIP model, incorporating region awareness to its repertoire of capabilities. This model is fine-tuned on millions of RGBA region-text pairs, enabling it to maintain CLIP's visual recognition prowess while offering precise control over the emphasis of image content. By integrating an additional alpha channel into the CLIP image encoder, Alpha-CLIP allows for detailed segmentation and region-specific processing without modifying the foundational CLIP weights, thus facilitating a nuanced approach to image understanding that respects the spatial dynamics of visual data. The training of Alpha-CLIP leverages a novel data generation pipeline designed to produce a vast array of RGBA-region text pairs. This process involves the creation of natural images equipped with foreground alpha channels and their corresponding referring expressions for specific regions. Such a methodology not only enables the fine-tuning of the model with an additional alpha channel input but also underpins its ability to perform with heightened specificity across various tasks. These tasks range from image recognition to multimodal large language models, and extend into both 2D and 3D generation domains, showcasing Alpha-CLIP's versatility and broad applicability. Datasets like LAION-400M, LAION-5B, and GRIT play a crucial role in training Alpha-CLIP, providing a wide spectrum of images for initial training and fine-grained mask-level labels for enhancing local perception capabilities. This strategic choice of datasets ensures that Alpha-CLIP is not only well-equipped for general visual recognition tasks but also capable of nuanced, region-specific processing and understanding, setting a new standard for models at the intersection of language and vision.
Nous-Hermes-2-Vision builds upon OpenHermes-2.5 by integrating the efficient SigLIP-400M vision encoder and incorporating a custom dataset with function calling capabilities, enabling it to not only understand visual and textual information but also extract specific text from images, advancing its functionality as a Vision-Language Action Model.
This project is led by qnguyen3 and teknium.
Architecture figure: The card describes SigLIP integration and training data in prose but contains no legitimate architecture or system figure.
ℹ️ More Information
Nous-Hermes-2-Vision: Represents a notable advancement in the realm of Vision-Language Models, marking its distinction through the integration of two key enhancements that elevate its capabilities beyond traditional models. This model is an evolution from its predecessor, OpenHermes-2.5-Mistral-7B, and distinguishes itself by incorporating the SigLIP-400M for significantly improved performance and efficiency, moving away from the standard reliance on larger 3B vision encoders. Additionally, it introduces a custom dataset that includes function calling capabilities, transforming it into a more dynamic Vision-Language Action Model. The training of Nous-Hermes-2-Vision utilized a diverse dataset comprising 220K images from LVIS-INSTRUCT4V, 60K from ShareGPT4V, 150K private function calling data, and 50K conversations from teknium's OpenHermes-2.5. Such a varied dataset ensures the model's proficiency across a broad spectrum of vision-language tasks, including object recognition, instruction following, and conversational understanding. The model's innovative approach to integrating vision and language, particularly through the use of custom datasets for function calling, allows for encoding vision and text together in a way that supports action-oriented tasks and automation. A key feature of Nous-Hermes-2-Vision is its ability to interact with images to extract valuable text information from visual content, thus enabling detailed analyses and responses in natural language. This capability is underscored by the model's utilization of the SigLIP-400M, opting for a more lightweight and efficient architecture while enhancing performance in vision-language tasks. The model is further enriched with a custom dataset that includes function calling, allowing for the extraction of written information from images through specific tags, thus broadening its application scope for developers and researchers alike. Despite its innovative features, early usage of Nous-Hermes-2-Vision has revealed some challenges, such as hallucinations and spamming of EOS tokens. Recognizing these issues, the research team, led by Quan Nguyen and Teknium, has committed to releasing an updated version to address these problems, demonstrating their dedication to refining the model's capabilities.
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
SPHINX pushes the boundaries of multi-modal LLMs by jointly mixing model weights, tasks, and visual embeddings during training, utilizing a two-stage approach that unfreezes the LLM (LLaMA-2) during pre-training for enhanced cross-modal learning and achieving impressive performance on a variety of vision-language tasks.
Figure 3. SPHINX jointly mixes tuning tasks, visual embeddings, and model weights in one multimodal architecture. Source paper, PDF p. 6. Figure notice.
ℹ️ More Information
SPHINX: stands out as a multi-modal large language model (MLLM) designed to enhance the integration of language and vision through an innovative approach that includes the joint mixing of model weights, tuning tasks, and visual embeddings. This model is particularly distinguished by its methodology of unfreezing the large language model during pre-training to foster more effective cross-modal learning. The architecture of SPHINX is built upon a foundation that combines vision encoders, two linear projection layers, and leverages LLaMA-2 as the language model backbone. It adopts a two-stage training paradigm that emphasizes pre-training for vision-language alignment followed by fine-tuning aimed at visual instruction-following tasks. In the realm of training methodologies, SPHINX employs a strategy that emphasizes the joint mixing of model weights, tuning tasks, and visual embeddings, setting a precedent for robust cross-modal knowledge acquisition. This approach is complemented by a pre-training regimen that utilizes both real-world and synthetic data, thereby ensuring a comprehensive understanding across various visual instruction tasks. The model introduces an efficient strategy for processing high-resolution images, utilizing mixed scales and sub-images to accommodate diverse visual inputs. Moreover, SPHINX achieves vision-language alignment by integrating comprehensive visual embeddings, unfreezing the LLM during pre-training, and employing a weight-mixing strategy that bridges domain-specific knowledge across different network architectures and training paradigms. The datasets utilized in training SPHINX, including LAION-400M, LAION-COCO, RefinedWeb, VQAV2, GQA, OKVQA, A-OKVQA, OCRVQA, TextCaps, COCO, LVIS, RefCOCO, VG, and Flickr30k, serve a multifaceted purpose. They are instrumental in achieving multi-modal alignment, language-only tuning, and addressing a wide spectrum of visual question answering and general vision tasks. These tasks range from object detection and human pose estimation to referring object localization and understanding descriptions within the context of image regions. SPHINX, through its meticulous design and strategic training approach, sets a new benchmark in the field of multi-modal large language models, advancing the capabilities in vision-language integration.
Florence-2 presents a significant advancement in vision foundation models, aiming to achieve a single, versatile representation capable of handling a wide spectrum of vision and vision-language tasks through a unified, prompt-based approach. Unlike previous models that often specialize in specific tasks, Florence-2 is designed to be a generalist, adept at performing tasks with simple text instructions, similar to how Large Language Models (LLMs) operate.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, Lu Yuan
Figure 2. Florence-2 combines an image encoder and multimodality encoder-decoder through a unified task-prompt interface. Source paper, PDF p. 3. Figure notice.
ℹ️ More Information
Florence-2 lies a sophisticated architecture comprised of two key components: an image encoder and a multi-modality encoder-decoder. The image encoder, powered by the powerful DaViT architecture, transforms the input image into a sequence of visual token embeddings, effectively capturing the visual information. These visual embeddings are then combined with text embeddings derived from task-specific prompts. This fusion of visual and linguistic information is processed by a standard transformer-based multi-modality encoder-decoder. This component acts as the brain of the model, meticulously analyzing the combined input and generating the desired output in textual form. This unified architecture, with a single set of parameters governing various tasks, eliminates the need for task-specific modifications, leading to a streamlined and efficient model. This design philosophy mirrors the trend in the NLP community, where models with consistent underlying structures are preferred for their versatility and ease of development. Florence-2's capabilities span a multitude of tasks, showcasing its remarkable adaptability. It excels at generating detailed image captions, capturing the essence of an image through rich textual descriptions. Its prowess extends to visual grounding, accurately pinpointing specific objects or regions within an image based on textual phrases. Florence-2 also demonstrates impressive performance in open-vocabulary object detection, identifying objects by their names, even if those objects were not part of its training data. This capability highlights the model's ability to generalize its knowledge and understand novel visual concepts. Furthermore, Florence-2 tackles dense region captioning, providing detailed descriptions for multiple regions within an image, and even performs optical character recognition (OCR), extracting text from images. This broad range of capabilities makes Florence-2 a powerful tool for numerous applications, pushing the boundaries of multimodal understanding in AI.
u-LLaVA introduces a novel projector-based architecture that unifies multi-modal tasks by connecting specialized expert models with a central Large Language Model (LLM), enabling seamless modality alignment and efficient multi-task learning through a two-stage training approach.
Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Yanchun Xie, Yi-Jie Huang, Yaqian Li
Figure 1. u-LLaVA unifies modality alignment with task-specific projectors, decoders, and patched downstream modules. Source paper, PDF p. 2. Figure notice.
ℹ️ More Information
u-LLaVA: Represents a pioneering approach in the integration of Large Language Models (LLMs) with specialized expert models to address a wide array of multi-modal tasks. This architecture is designed to leverage the strengths of LLMs as a central hub, facilitating seamless modality alignment and multi-task learning. Through a novel projector-based structure that incorporates CLIP's Vision Transformer (ViT-L/14) and LLaMA2, u-LLaVA introduces a flexible framework capable of handling diverse modalities and tasks. The system integrates special tokens for modality and task expressions, alongside dedicated modules for segmentation, grounding, and in-painting, to enrich its multi-modal capabilities. The training methodology of u-LLaVA is executed in two distinct stages, beginning with a coarse-grained alignment to ensure the alignment of representation spaces across different modalities. This foundational step is crucial for establishing a common ground for further, more nuanced task-specific adaptations. Following this, a fine-grained alignment phase focuses on the refinement of task-specific instruction data, optimizing the model's performance for targeted applications. This dual-stage training approach ensures that u-LLaVA can efficiently adapt to a variety of tasks with minimal additional training requirements. Central to u-LLaVA's effectiveness is its innovative use of projector-based alignment techniques and fusion methods, which enable the integration of visual and textual representations within the LLM's framework. By mapping hidden states and text embeddings through projectors, u-LLaVA facilitates modality fusion, leveraging the extensive knowledge embedded within LLMs for complex task solving. The datasets utilized for training, including LLaVA CC3M, Conversation-58K, Detail-23K, and others, are meticulously curated to support the model's versatile capabilities across tasks such as image captioning, video captioning, visual question answering (VQA), referential expression comprehension (RES), semantic segmentation, and salient object detection/segmentation. This strategic selection and organization of datasets underscore u-LLaVA's commitment to advancing multi-modal task unification through Large Language Models.
LLaVA-Plus pioneers the creation of multimodal agents by integrating diverse vision and vision-language models into a skill repository, enabling the agent to learn and use tools effectively through end-to-end training on comprehensive multimodal instruction-following data.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, Chunyuan Li
