Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[NeurIPS'24] This repository is the implementation of "SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models"
| Date | Stars |
|---|---|
| 2026-07-24 | 335 |
| 2026-07-25 | 336 |
| 2026-07-28 | 337 |
| 2026-07-30 | 337 |
| 2026-07-31 | 338 |
| 2026-08-06 | 338 |
Today
— stars today
This week
+1 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.30%/day
<p align="center"> <img src="demo_images/logo.png" width="15%"/> </p> # SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models (NeurIPS'24) [](CODE_LICENSE) [](MODEL_LICENSE)   [](https://www.python.org/downloads/release/python-3100/) [arxiv](https://arxiv.org/abs/2406.01584) / [Huggingface](https://huggingface.co/collections/a8cheng/spatialrgpt-66fef10465966adc81819723) ______________________________________________________________________ ## 💡 Introduction [**SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models**](<>) <br> [*An-Chieh Cheng*](https://www.anjiecheng.me/), [*Hongxu (Danny) Yin*](https://hongxu-yin.github.io/), [*Yang Fu*](https://oasisyang.github.io/), [*Qiushan Guo*](https://guoqiushan.github.io/), [*Ruihan Yang*](https://rchalyang.github.io/), [*Jan Kautz*](https://jankautz.com/), [*Xiaolong Wang*](https://xiaolonw.github.io/), [*Sifei Liu*](https://sifeiliu.net/) <br> SpatialRGPT is a powerful vision-language model adept at understanding both 2D and 3D spatial arrangements. It can process any region proposal, such as boxes or masks, and provide answers to complex spatial reasoning questions. ______________________________________________________________________ ## 📢 News - **Oct-07-24**- SpatialRGPT code/dataset/benchmark released! 🔥 - **Sep-25-24**- We're thrilled to share that SpatialRGPT has been accepted to NeurIPS 2024! 🎊 ______________________________________________________________________ ## Installation To build environment for training SpatialRGPT, please run the following: ```bash ./environment_setup.sh srgpt conda activate srgpt ``` ______________________________________________________________________ ## Gradio Demo To run the Gradio demo for SpatialRGPT, please follow these steps. Due to `pydantic` version conflicts, the demo environment is not compatible with the training environment. Therefore, a separate environment will need to be created for the Gradio demo. 1. Build the environment. ```bash ./environment_setup.sh srgpt-demo conda activate srgpt-demo pip install gradio==4.27 deepspeed==0.13.0 gradio_box_promptable_image segment_anything_hq pip install -U 'git+https://github.com/facebookresearch/detectron2.git@ff53992b1985b63bd3262b5a36167098e3dada02' ``` If you run into an error with the detectron2 installation, it could be because `CUDA_HOME` is not set. To fix this, export `CUDA_HOME` to your local CUDA path. See details in [this issue](https://github.com/AnjieCheng/SpatialRGPT/issues/6). 1. Clone the Depth-Anything repository and download the necessary checkpoint: ```bash git clone https://github.com/LiheYoung/Depth-Anything.git wget https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vitl14.pth ``` Place `depth_anything_vitl14.pth` under `Depth-Anything/checkpoints`, and set the path to the environment variable. For example: ```bash export DEPTH_ANYTHING_PATH=/YOUR_OWN_PATH/Depth-Anything ``` 1. Download SAM-HQ checkpoint from [here](https://drive.google.com/file/d/1qobFYrI4eyIANfBSmYcGuWRaSIXfMOQ8/view), and set the path to the environment variable. For example: ```bash export SAM_CKPT_PATH=/YOUR_OWN_PATH/sam_hq_vit_h.pth ``` 1. Launch Gradio server. You can use your own checkpoint, or use `a8cheng/SpatialRGPT-VILA1.5-8B` ```bash cd demo python gradio_web_server_multi.py --model-path PATH_TO_CHECKPOINT ``` ______________________________________________________________________ ## Training SpatialRGPT follows VILA training, whic
Excerpt of 8,407 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:43cc4a54e1d74f95, topic:vision-language-model, desc:vision-language, readme:vision-language