Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Contextual Object Detection with Multimodal Large Language Models
| Date | Stars |
|---|---|
| 2026-07-31 | 261 |
| 2026-08-05 | 261 |
| 2026-08-06 | 261 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h2 align="center" width="100%">
Contextual Object Detection with Multimodal Large Language Models
</h2>
<div>
<div align="center">
<a href='https://yuhangzang.github.io/' target='_blank'>Yuhang Zang</a> 
<a href='https://weivision.github.io/' target='_blank'>Wei Li</a> 
<a href='https://www.linkedin.com/in/han-jun-581849193/' target='_blank'>Jun Han</a> 
<a href='https://kaiyangzhou.github.io/' target='_blank'>Kaiyang Zhou</a> 
</br>
<a href='https://www.mmlab-ntu.com/person/ccloy/index.html' target='_blank'>Chen Change Loy</a> 
</div>
<div>
<div align="center">
S-Lab, Nanyang Technological University
</div>
<p align="center">
<a href="https://arxiv.org/abs/2305.18279" target='_blank'>
<img src="http://img.shields.io/badge/cs.CV-arXiv%3A2305.18279-B31B1B.svg">
</a>
<a href="https://www.mmlab-ntu.com/project/contextdet/index.html" target='_blank'>
<img src="https://img.shields.io/badge/Project Page-%F0%9F%93%9a-lightblue">
</a>
<a href="https://huggingface.co/spaces/yuhangzang/ContextDet-Demo">
<img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</p>
## 🆕 Updates in Oct 2024
- [x] HuggingFace demo and model checkpoint, see <a href="https://huggingface.co/spaces/yuhangzang/ContextDet-Demo">here</a>
- [x] The CODE dataset for evaluation, see [here](evaluation)
- [ ] ContextDET training scripts, see [here](train_code) (waiting to be cleaned up)
## 🌟 Contextual Object Detection
Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, <i>i.e</i>., object detection. In this work, we address this limitation by introducing a novel research problem of <strong>contextual object detection</strong>--understanding visible objects within different human-AI interactive contexts. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering.
<div style="text-align:center">
<img src="./asset/benchmark.png" width="100%" height="100%">
</div>
### Comparison with Related Works
| Task | Language Input | Output(s) | Remark |
|---|---|---|---|
| Object Detection | ✗ | box, class label | pre-defined class labels |
| Open-Vocabulary Object Detection | (optional) class names for CLIP | box, class label | pre-defined class labels |
| Referring Expression Comprehension | complete referring expression | box that expression refers to | / |
| <b>Contextual Cloze Test</b> (ours) | <b>incomplete</b> expression, object names are masked | {box, <b>name</b>} to complete the mask | <b>name</b> could be most valid English word |
| Image Captioning | ✗ | language caption | / |
| <b>Contextual Captioning</b> (ours) | ✗ | language caption, <b>box</b> | / |
| Visual Question Answering | language question | language answer | / |
| <b>Contextual QA</b> (ours) | language question | language question, <b>box</b> | / |
## 😎 Method
We present ContextDET, a novel <i>generate-then-detect</i> framework, specialized for contextual object detection. ContextDET is end-to-end and consists of three key architectural components:
1) a visual encoder that extracts high-level image representations and computes visual tokens,
2) a pre-trained LLM that decodes multimodal contextual tokens with a task-related multimodal prefix, and
3) a visual decoder that predicts matching scores and bounding boxes for conditional queries linked to contextual object words.
The new <strong>generate-then-detect</strong> framework enables us to detect object words within human vocabulary.
<div style="text-align:center">
<img src="./asset/framework.png" width="100%" height="100%">
</div>
## 🥰 Qualitative Examples
<div style="text-align:center">
<img src="./asset/background.png" width="100%" height="100%">
</div>
## 💻 Try Demo
🤗 You can try our demo on <a href="https://huggingface.cExcerpt of 6,020 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:06fb4a56cfedde06, topic:object-detection, desc:object detection