Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Official implementation of paper "VLM³: Vision Language Models Are Native 3D Learners".
| Date | Stars |
|---|---|
| 2026-07-31 | 403 |
| 2026-08-03 | 405 |
| 2026-08-06 | 405 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# [VLM³: Vision Language Models Are Native 3D Learners](https://arxiv.org/pdf/2605.30561)
[]([https://arxiv.org/abs/XXXX.XXXXX](https://arxiv.org/abs/2605.30561))
| Model | Coming Soon! |
|:----:|:-------------------------------------------------------------------------------------------------:|
<div align=center>
<img width=100% src="./media/teaser.svg"/>
</div>
## Summary
We show that **standard VLMs** are native 3D learners. We propose VLM³, which **without complex data augmentations and any architecture/loss change**, can make standard VLMs:
- Surpass SpatialRGPT on object-level 3D understanding (both qualitative and quantitative in SpatialRGPT-bench), without using extra encoders.
- Match [UnidepthV2](https://arxiv.org/abs/2502.20110) and [Moge-2](https://arxiv.org/pdf/2507.02546) on metric depth estimation, improving the accuracy of [DepthLM](https://github.com/facebookresearch/DepthLM_Official) from 0.84 to 0.9;
- Surpass [DKM](https://openaccess.thecvf.com/content/CVPR2023/papers/Edstedt_DKM_Dense_Kernelized_Feature_Matching_for_Geometry_Estimation_CVPR_2023_paper.pdf) and [RoMa](https://arxiv.org/pdf/2305.15404) for pixel correspondence estimation;
- Match [DepthAnything3](https://arxiv.org/abs/2511.10647) and surpass [VGGT](https://arxiv.org/pdf/2503.11651) for camera pose estimation;
VLM³ opens up a new paradigm for simple and scalable 3D learning. Now you dont need to spend a year designing:
- complex models with different backbones, prediction heads, routings.
- complex losses for different prediction heads, balancing weights for different losses
- complex data augmentations like image cropping, rotatin, translation, appearance augmentation etc.
**All you need to do is collect data, and scale the training with a standard VLM!**
Our findings provide a new perspective on what is and is not necessary for 3D vision:
- Large models, task-specific architectures, losses, data-augmentations, and even the regression formulation that sets the foundation of most SOTA 3D expert vision models, are all not necessary conditions for effective 3D learning.
- A generalist foundation model (VLM) with unified output domain (text) + data scaling are sufficient.
## Method Overview
Given the input images, VLM³ first resizes them so that the focal length is the same for all input images (e.g., 1000 pixels). This solves camera ambiguity without the need for adding extra VLM encoders/modules. To refer to an object or pixel, VLM³ simply uses text with the pixel range normalized (e.g., [0, 2000) or [0, 1000)) for both horizontal and vertical axes. This requires no architecture or marker rendering, and makes VLM³ much more flexible and scalable. Standard VLM architectures and text-based training (SFT) are used to train the model.
<div align=center>
<img width=100% src="./media/pipeline.svg"/>
</div>
## Results
<div align=center>
<img width=100% src="./media/visualizations.svg"/>
</div>
<div align=center>
<img width=100% src="./media/table1.png"/>
</div>
<div align=center>
<img width=100% src="./media/table2.png"/>
</div>
## Contact
Zhipeng Cai, Meta Inc, homepage: https://zhipengcai.github.io/, email: czptc2h at gmail dot com.
# Quickstart
Install transformers to do inference with our model.
pip install transformers>=5.4.0
Since VLM³ maintain the architecture of the base model (Qwen3-vl-4B), we can call the model for inference the same as the original VLM.
### Using 🤗 Transformers to Chat
Here we show a code snippet to show you how to use the chat model with `transformers`:
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
# default: Load the model on the available device(s)
model = AutoModelForImageTextToText.from_pretrained(
"facebook/VLM3-depth", dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_prExcerpt of 6,434 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a2ec24e2c2dddf0d, topic:large-language-models