Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICLR 2024] FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators
| Date | Stars |
|---|---|
| 2026-07-31 | 308 |
| 2026-08-04 | 308 |
| 2026-08-06 | 308 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h2>
<a href="https://whu-usi3dv.github.io/FreeReg/" target="_blank">FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators</a>
</h2>
This is the official PyTorch implementation of the following publication:
> **FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators**<br/>
> [Haiping Wang](https://hpwang-whu.github.io/), [Yuan Liu](https://liuyuan-pal.github.io/), [Bing Wang](https://www.polyu.edu.hk/aae/people/academic-staff/dr-wang-bing/), [Yujing Sun](https://yujingsun.github.io/), [Zhen Dong](https://dongzhenwhu.github.io/index.html), [Wenping Wang](https://www.cs.hku.hk/people/academic-staff/wenping), [Bisheng Yang](https://3s.whu.edu.cn/info/1025/1415.htm)<br/>
> *ICLR 2024*<br/>
> [**Paper**](https://arxiv.org/abs/2310.03420) | [**Project-page**](https://whu-usi3dv.github.io/FreeReg/) | [**Video**](https://www.youtube.com/watch?v=N_M5Zsfo1U8)
## 🔭 Introduction
<p align="center">
<strong>TL;DR: FreeReg extracts cross-modality features from pretrained diffusion models and monocular depth estimators for accurate zero-shot image-to-point cloud registration.</strong>
</p>
<img src="utils/media/teaser.png" alt="Network" style="zoom:50%;">
<p align="justify">
<strong>Abstract:</strong> Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. However, due to the modality difference between images and points, it is difficult to learn robust and discriminative cross-modality features by existing metric learning methods for feature matching. Instead of applying metric learning on cross-modality data, we propose to unify the modality between images and point clouds by pretrained large-scale models first, and then establish robust correspondence within the same modality. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds, which enables the building of coarse but robust cross-modality correspondences. We further extract geometric features on depth maps produced by the monocular depth estimator. By matching such geometric features, we significantly improve the accuracy of the coarse correspondences produced by diffusion features. Extensive experiments demonstrate that <strong>without any task-specific training</strong>, direct utilization of both features produces accurate image-to-point cloud registration. On three public indoor and outdoor benchmarks, the proposed method averagely achieves a 20.6 percent improvement in Inlier Ratio, a three-fold higher Inlier Number, and a 48.6 percent improvement in Registration Recall than existing state-of-the-arts.
</p>
## 🆕 News
- 2024-11-08: OpenSource [Free-FreeReg](https://github.com/WHU-USI3DV/FreeReg/tree/FFreeReg), an improved version of FreeReg-G, faster and easy-to-use!
- 2024-01-16: Accepted by ICLR 2024! 🎉🎉🎉
- 2023-10-06: Code, [[project page]](https://whu-usi3dv.github.io/FreeReg/), [[Youtube video]](https://www.youtube.com/watch?v=N_M5Zsfo1U8), and [[Preprint paper]](https://arxiv.org/abs/2310.03420) are aviliable! 🎉
## 💻 Requirements
The code has been tested on:
- Ubuntu 20.04
- CUDA 11.7
- Python 3.8.5
- Pytorch 1.12.1
- GeForce RTX 4090.
## 🔧 Installation
- First, create the conda environment:
```
conda env create -f environment.yaml
conda activate freereg
```
- Second, fetch the latest [MiDaS](https://github.com/isl-org/MiDaS) repo via torch hub for Zoe:
```
import torch
torch.hub.help("intel-isl/MiDaS", "DPT_BEiT_L_384", force_reload=True) # Triggers fresh download of MiDaS repo
```
- Third, install [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine) for FCGF:
```
conda install openblas-devel -c anaconda
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
python setup.py install --blas_include_dirs=$Excerpt of 7,962 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:054a1ac14b092d70, topic:diffusion-models