Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ECCV 2024] Improving 2D Feature Representations by 3D-Aware Fine-Tuning
| Date | Stars |
|---|---|
| 2026-07-31 | 329 |
| 2026-08-06 | 329 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Improving 2D Feature Representations by 3D-Aware Fine-Tuning
### ECCV 2024
[Yuanwen Yue](https://ywyue.github.io/) <sup>1</sup>,
[Anurag Das](https://anurag-198.github.io/) <sup>2</sup>,
[Francis Engelmann](https://francisengelmann.github.io/) <sup>1,3</sup>,
[Siyu Tang](https://vlg.inf.ethz.ch/team/Prof-Dr-Siyu-Tang.html) <sup>1</sup>,
[Jan Eric Lenssen](https://janericlenssen.github.io/) <sup>2</sup>
<br>
**<sup>1</sup>ETH Zurich, <sup>2</sup>Max Planck Institute for Informatics, <sup>3</sup>Google**
### [Project Page](https://ywyue.github.io/FiT3D) | [Paper](http://arxiv.org/abs/2407.20229)
<a target="_blank" href="https://colab.research.google.com/github/ywyue/FiT3D/blob/main/FiT3D_demo.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
<a href="https://huggingface.co/spaces/yuanwenyue/FiT3D">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
<img width="1100" src="./assets/teaser.png" />
This is the official repository for the paper Improving 2D Feature Representations by 3D-Aware Fine-Tuning.
## Changelog
- [x] Add Colab Notebook and Hugging Face demo
- [x] Release ScanNet++ preprocessing code
- [x] Release feature Gaussian training code
- [x] Release fine-tuning code
- [x] Release evaluation code
<details open="open" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'>
<summary>Table of Contents</summary>
<ol>
<li>
<a href="#demo">Demo</a>
</li>
<li>
<a href="#preparation">Preparation</a>
</li>
<li>
<a href="#training">Training</a>
</li>
<li>
<a href="#evaluation">Evaluation</a>
</li>
<li>
<a href="#citation">Citation</a>
</li>
</ol>
</details>
## Demo
We provide a [Colab Notebook](https://colab.research.google.com/github/ywyue/FiT3D/blob/main/FiT3D_demo.ipynb) with step-by-step guides to make inference and visualize the PCA features and K-Means clustering of original 2D models and our fine-tuned models.
We also provide an online [Hugging Face demo 🤗](https://huggingface.co/spaces/yuanwenyue/FiT3D) where users can upload their own images and check the visualizations online. Alternatively, to run the demo locally, just try ```python app.py```.
<img width="1100" src="./assets/demo.gif" />
## Preparation
### Environment
* The code has been tested on Linux with Python 3.10.14, torch 1.9.0, and cuda 11.8.
* Create an environment and install pytorch and other required packages:
```shell
git clone https://github.com/ywyue/FiT3D.git
cd FiT3D
conda create -n fit3d python=3.10
conda activate fit3d
pip install torch==2.0.0 torchvision==0.15.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
```
* Compile the feature rasterization modules and the knn module for feature lifting:
```shell
cd submodules/diff-feature-gaussian-rasterization
python setup.py install
cd ../simple-knn/
python setup.py install
```
* Install ```mmcv``` and ```mmsegmentation```, required for downstream evaluation. Note we modifed the source code so please build them from source as follows:
```shell
cd mmcv
MMCV_WITH_OPS=1 pip install -e . -v
cd ../mmsegmentation
pip install -e . -v
```
### Data
We train feature Gaussians and fine-tuning on ScanNet++ scenes. Preprocessing code and instructions are [here](https://github.com/ywyue/FiT3D/tree/main/scannetpp_preprocess). After preprocessing, the ScanNet++ data is expected to be organized as following:
```
FiT3D/
└── db/
└── scannetpp/
├── metadata/
| ├── nvs_sem_train.txt # Training set for NVS and semantic tasks with 230 scenes
| ├── nvs_sem_val.txt # Validation set for NVS and semantic tasks with 50 scenes
| ├── train_samples.txt # Training sample list, formatted as sceneID_imageID
| ├── val_samples.txt # Validation sample list, formatted as Excerpt of 9,631 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:80174850bb29fe83, topic:fine-tuning, desc:fine-tuning, desc:fine tuning
matched fp:80174850bb29fe83, topic:foundation-models
matched fp:80174850bb29fe83, topic:clip
matched fp:80174850bb29fe83, topic:representation-learning