Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
(ECCVW 2025)GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
| Date | Stars |
|---|---|
| 2026-07-24 | 556 |
| 2026-07-25 | 556 |
| 2026-07-28 | 556 |
| 2026-07-30 | 556 |
| 2026-07-31 | 555 |
| 2026-08-06 | 555 |
Today
— stars today
This week
-1 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest :fire: [Demo](http://139.196.83.164:7000/) :fire:
[//]: # (<div id="wrapper" align="center">)
[//]: # (<figure>)
[//]: # ( <img src="figs/demo1.gif" width="45%"> )
[//]: # ( <img src="figs/demo2.gif" width="45%"><br>)
[//]: # ( <p style="font-size:1.2vw;">Left: Single-Region Understanding; Right: Single-Region Understanding</p>)
[//]: # (</figure>)
[//]: # (</div>)
<p align="center">
<img src="figs/demo1.gif" width="80%"> <br>
<p align="center" style="font-size:1.2vw;">Single-Region Understanding</p>
</p>
<p align="center">
<img src="figs/demo2.gif" width="80%"> <br>
<p align="center" style="font-size:1.2vw;">Multiple-Region Understanding</p>
</p>
## Introduction
<p align="center">
<img src="figs/framework.png" width="70%"> <br>
</p>
> [**GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest**](https://arxiv.org/abs/2307.03601)
> [Shilong Zhang*](https://jshilong.github.io/), [Peize Sun*](https://peizesun.github.io/), [Shoufa Chen*](https://www.shoufachen.com/), Min Xiao, Wenqi Shao ,Wenwei Zhang, Kai Chen, Ping Luo</br>
> (*Equal Contribution)
### [[Demo](http://139.196.83.164:7000/)] [[Paper](https://arxiv.org/abs/2307.03601)] [[中文介绍](https://zhuanlan.zhihu.com/p/640283103)]
[//]: # (#:grin::grin::grin:信交流群：xxx (答案：cheems))
## Updates
- [July 25] [GPT4RoI-7B-delta-V0](https://huggingface.co/shilongz/GPT4RoI-7B-delta-V0) has release ! :fire::fire::fire: You need to combine our delta with the original LLaMA weights follow the [GPT4RoI Weights](https://github.com/jshilong/GPT4RoI/tree/main#weights) section.
- [July 7] All training and inference code has been released, you can try demo [here](http://139.196.83.164:7000/) :fire::fire::fire:
## Contents
- [Install](#Install)
- [Data](#Data)
- [GPT4RoI Weights](#Weights)
- [Training](#Training)
- [Gradio](#Gradio)
- [Acknowledge](#Acknowledge)
## Install
1. Clone the `GPT4RoI`
```python
git clone https://github.com/jshilong/gpt4roi.git
cd gpt4roi
```
2. Create the env
```shell
conda create -n gpt4roi python=3.10 -y
conda activate gpt4roi
pip install --upgrade pip # enable PEP 660 support
pip install setuptools_scm
pip install --no-cache-dir -e .
# please use conda re-install the torch, pip may loss some runtime lib
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
```
3. Install the `flash-attn` package
```
pip install ninja
pip install flash-attn --no-build-isolation
```
4. install the `mmcv-1.4.7` package
Make sure that your `nvcc -V` is consistent with cudatookit version of `python -c "import torch;print(torch.version.cuda)`.
```shell
cd mmcv-1.4.7
MMCV_WITH_OPS=1 pip install -e .
```
<!-- ## Data Preparation
| Data file name | Size | original from|
| --- | --- | ---|
| [single_region_caption.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/raw/main/llava_instruct_150k.json) | 229 MB | VC, Refcocog |
| [multi_region_caption.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/raw/main/llava_instruct_80k.json) | 229 MB | flicker30k |
| [spation-instruction21k.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/raw/main/conversation_58k.json) | 126 MB | VCR |
We also use langauge-image multimodal instruction-folllowing dataset [`LLaVA-Instruct-150K`](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K), with we detect objects with EVA-lvis model, you should download the ``eva_lvis_coco.pkl`` when you use this dataset. -->
## Data
Our dataset includes RefCOCO, RefCOCO+, RefCOCOg, Visual Genome, Flickr30K entities, and the VCR dataset. We are sincerely grateful to the creators of these datasets, especially for the VCR dataset, for their forward-thinking in creating these dataset.
The dataset section of this repository may appear somewhat messy, especially the VCR part(still finishing), which may cause GPT4RoIExcerpt of 9,834 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e4b21a44fb13a078, topic:llm, topic:gpt
matched fp:e4b21a44fb13a078, topic:computer-vision
