Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[Fully open] [Encoder-free MLLM] Vision as LoRA
| Date | Stars |
|---|---|
| 2026-07-24 | 388 |
| 2026-07-25 | 389 |
| 2026-07-28 | 389 |
| 2026-07-30 | 389 |
| 2026-08-06 | 389 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# VoRA: Integrating Visual Capabilities into LLMs
<div align="center">
[](https://georgeluimmortal.github.io/vora-homepage.github.io/)
[](https://arxiv.org/pdf/2503.20680)
[](https://huggingface.co/collections/Hon-Wong/vora-67ee34c9d32e9ac2358106ae)
<p style="font-size: larger; margin-top: -5px;">
<a href="https://arxiv.org/pdf/2503.20680">Vision as LoRA</a>
</p>
<div align="center" style="width: 100%; margin: 0 auto;">
<img src="assets/framework.gif" alt="Framework" width="70%">
</div>
</div>
## News
* **2025-04-16:** Training code released.
* **2025-04-06:** [LMMs-Eval](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/616) has supported VoRA.
* **2025-04-04:** [VoRA Weights and training data](https://huggingface.co/collections/Hon-Wong/vora-67ee34c9d32e9ac2358106ae) are released.
<h3 align="center">Abstract</h3>
<p>We introduce Vision as LoRA (VoRA), a novel paradigm for transforming an LLM into an MLLM. Unlike prevalent MLLM architectures that rely on external vision modules for vision encoding, VoRA internalizes visual capabilities by integrating vision-specific LoRA layers directly into the LLM. This design allows the added parameters to be seamlessly merged into the LLM during inference, eliminating structural complexity and minimizing computational overhead. Moreover, inheriting the LLM's ability of handling flexible context, VoRA can process inputs at arbitrary resolutions.</p>
<p>To further strengthen VoRA’s visual capabilities, we introduce a block-wise distillation method that transfers visual priors from a pre-trained ViT into the LoRA layers, effectively accelerating training by injecting visual knowledge. Additionally, we apply bi-directional attention masks to better capture the context information of an image. We successfully demonstrate that with additional pre-training data, VoRA can perform comparably with conventional encode-based MLLMs.</p>
## Install
Clone this repository and install dependencies:
```bash
git clone https://github.com/Hon-Wong/VoRA.git
cd VoRA
pip3 install -e .
```
## Data Preparation
We have collected or generated the following datasets for VoRA. The image bytes are also included, so there’s no need to download the images from URLs. The captions were created using a variety of prompts to ensure diversity.
| HF dataset 🤗 | #Samples | Source | Generated by |
| ------------ | -------------- | ------------ | ------------ |
| [VoRA-Recap-8M](https://huggingface.co/datasets/Hon-Wong/VoRA-Recap-8M) | 8M | DataComp-1B | [Qwen2-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-72B-Instruct) |
| [VoRA-Recap-29M](https://huggingface.co/datasets/Hon-Wong/VoRA-Recap-29M) | 29M | DataComp-1B | [Qwen2-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-72B-Instruct) |
| [VoRA-Recap-GLDv2-1.4M](https://huggingface.co/datasets/Hon-Wong/VoRA-Recap-29M) | 1.4M | GLDv2 | [Qwen2-VL-72B-Instruct](https://huggingface.co/datasets/Hon-Wong/VoRA-Recap-GLDv2-1.4M) |
| [VoRA-TextQA-Mixed](https://huggingface.co/datasets/Hon-Wong/VoRA-TextQA-Mixed) | 6.3M | Cambrian, LLaVA-ov, Infinity-Instruction, SmolTalk | - |
1. download the pre-training datasets from HF.
datasets for ablation study
```bash
apt-get install git-lfs
git-lfs install
cd {raw_data_dir}
git clone https://huggingface.co/datasets/Hon-Wong/VoRA-Recap-8M
```
datasets for Pre-training
```bash
apt-get install git-lfs
git-lfs install
cd {raw_data_dir}
git clone https://huggingface.co/datasets/Hon-Wong/VoRA-Recap-29M
git clone https://huggingface.co/datasets/Hon-Wong/VoRA-Recap-GLDv2-1.4M
git clone https://huggingface.co/datasets/Hon-Wong/VoRA-TextQA-Mixed
```
2. convert parquet to json
For ablation:
```bash
python3 tools/parquet2json.py --dataset_dir={Excerpt of 7,142 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:25ea755e279e0eba, topic:lora, desc:lora, readme:lora
matched fp:25ea755e279e0eba, topic:llm
matched fp:25ea755e279e0eba, topic:vlm