Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization
| Date | Stars |
|---|---|
| 2026-07-24 | 585 |
| 2026-07-25 | 585 |
| 2026-07-28 | 585 |
| 2026-07-30 | 585 |
| 2026-08-06 | 585 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center">
<h1> Groma: Grounded Multimodal Assistant </h1>
> [**Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models**](https://arxiv.org/abs/2404.13013)
> **Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, Xiaojuan Qi**
>
><a href="https://arxiv.org/abs/2404.13013"><img src='https://img.shields.io/badge/arXiv-Groma-red' alt='Paper PDF'></a>
><a href='https://groma-mllm.github.io/'><img src='https://img.shields.io/badge/Project_Page-Groma-green' alt='Project Page'></a>
><a href='https://huggingface.co/FoundationVision/groma-7b-finetune'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a>
><a href='https://huggingface.co/datasets/FoundationVision/groma_instruct'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow'></a>
<img src='docs/teaser.png' align="center" width="80%">
<p align="left">Groma is an MLLM with exceptional region understanding and visual grounding capabilities. It can take user-defined region inputs (boxes) as well as generate long-form responses that are grounded to visual context.</p>
<img src='docs/paradigm.png' align="center" width="80%">
<p align="left">Groma presents a novel paradigm of grounded MLLMs. (a) LLM for localization (e.g., Kosmos-2, Shikra); (b) External modules for localization (e.g., Lisa); and (c) <b>Visual tokenier for localization (Groma)</b>.
</div>
## Contents
- [Install](#installation)
- [Model](#model-weights)
- [Data](#prepare-data)
- [Training](#training)
- [Inference](#inference)
- [Evaluation](#evaluation)
## Performance
State-of-the-art performance on referring expression comprehension (REC) benchmarks among multimodal
large language models.
<table>
<thead>
<tr>
<th rowspan="2">Method</th>
<th colspan="3">RefCOCO</th>
<th colspan="3">RefCOCO+</th>
<th colspan="2">RefCOCOg</th>
<th rowspan="2">Avergae</th>
</tr>
<tr>
<th>val</th>
<th>testA</th>
<th>testB</th>
<th>val</th>
<th>testA</th>
<th>testB</th>
<th>val</th>
<th>test</th>
</tr>
</thead>
<tbody>
<tr align="center">
<td>Shikra</td>
<td>87.01</td>
<td>90.61</td>
<td>80.24</td>
<td>81.60</td>
<td>87.36</td>
<td>72.12</td>
<td>82.27</td>
<td>82.19</td>
<td>82.93</td>
</tr>
<tr align="center">
<td>Ferret</td>
<td>87.49</td>
<td>91.35</td>
<td>82.45</td>
<td>80.78</td>
<td>87.38</td>
<td>73.14</td>
<td>83.93</td>
<td>84.76</td>
<td>83.91</td>
</tr>
<tr align="center">
<td>MiniGPT-v2</td>
<td>88.69</td>
<td>91.65</td>
<td>85.33</td>
<td>79.97</td>
<td>85.12</td>
<td>74.45</td>
<td>84.44</td>
<td>84.66</td>
<td>84.29</td>
</tr>
<tr align="center">
<td>Qwen-VL</td>
<td>89.36</td>
<td>92.26</td>
<td>85.34</td>
<td>83.12</td>
<td>88.25</td>
<td>77.21</td>
<td>85.58</td>
<td>85.48</td>
<td>85.83</td>
</tr>
<tr style="background-color: #ADD8E6;">
<th>Groma</th>
<th>89.53</th>
<th>92.09</th>
<th>86.26</th>
<th>83.90</th>
<th>88.91</th>
<th>78.05</th>
<th>86.37</th>
<th>87.01</th>
<th>86.52</th>
</tr>
</tbody>
</table>
## Installation
Clone the repository
~~~
git clone https://github.com/FoundationVision/Groma.git
cd Groma
~~~
Create the conda environment and install dependencies
~~~
conda create -n groma python=3.9 -y
conda activate groma
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install --upgrade pip # enable PEP 660 support
pip install -e .
cd mmcv
MMCV_WITH_OPS=1 pip installExcerpt of 7,986 characters
Read on GitHub
11
6
Yi Jiang · Bytedance Seed · United States
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:99f562b082af1f70, topic:large-language-models, topic:llm, topic:foundation-models
matched fp:99f562b082af1f70, topic:multimodal, topic:vision-language-model, desc:multimodal