Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[CVPR2024 Highlight] Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration
| Date | Stars |
|---|---|
| 2026-07-31 | 430 |
| 2026-08-06 | 430 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# ChatSim
Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration
[Arxiv](https://arxiv.org/abs/2402.05746) | [Project Page](https://yifanlu0227.github.io/ChatSim/) | [Video](https://youtu.be/5xWz5YBsE5M)
## News
[06/12/2024] 🔥🔥🔥 **background rendering speed up**! **3D Gaussian splatting** is integrated as a background rendering engine, rendering 50 frames within 30s.
[06/12/2024] 🔥🔥🔥 **foreground rendering speed up**! multiple process for blender rendering in parallel! rendering 50 frames within 5 minutes.
## Requirement
- Ubuntu version >= 20.04 (for using Blender 3.+)
- Python >= 3.8
- Pytorch >= 1.13
- CUDA >= 11.6
- COLMAP or Metashape software (not necessary, we provide recalibrated poses)
- OpenAI API Key (you can also use other models' API from [NVIDIA AI](https://build.nvidia.com/explore/reasoning) for free lunch)
## Installation
First clone this repo recursively.
```bash
git clone https://github.com/yifanlu0227/ChatSim.git --recursive
```
### Step 1: Setup environment
```bash
conda create -n chatsim python=3.9 git-lfs
conda activate chatsim
```
### Step 2: Install background rendering engine
We offer two background rendering methods, one is `McNeRF` in our paper, and another is [`3D Gaussian Splatting`](https://github.com/graphdeco-inria/gaussian-splatting). `McNeRF` encodes the exposure time and achieves brightness-consistent rendering. `3D Gaussian Splatting` is much faster (**about 50 x**) in rendering and has higher PSNR in training views. However, strong perspective shifts result in noticeable artifacts.
**McNeRF**
https://github.com/yifanlu0227/ChatSim/assets/45688237/6e7e4411-31e5-46e3-9ca2-be0d6e813a60
**3D Gaussian Splatting**
https://github.com/yifanlu0227/ChatSim/assets/45688237/e7ac487c-5615-455d-bb38-026aaaabce70
Installing either one is OK! If you want high rendering speed and do not care about brightness inconsistency, choose `3D Gaussian Splatting`.
<details>
<summary><span style="font-weight: bold;"> Install McNeRF (official implement in the paper) </summary>
```bash
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
pip install -r requirements.txt
imageio_download_bin freeimage
```
The installation is the same as [F2-NeRF](https://github.com/totoro97/f2-nerf). Please go through the following steps.
```bash
cd chatsim/background/mcnerf/
# mcnerf use the same data directory.
ln -s ../../../data .
```
#### Step 2.1: Install dependencies
For Debian based Linux distributions:
```
sudo apt install zlib1g-dev
```
For Arch based Linux distributions:
```
sudo pacman -S zlib
```
#### Step 2.2: Download pre-compiled LibTorch
Taking `torch-1.13.1+cu117` for example.
```bash
cd chatsim/background/mcnerf
cd External
# modify the verison if you use a different pytorch installation
wget https://download.pytorch.org/libtorch/cu117/libtorch-cxx11-abi-shared-with-deps-1.13.1%2Bcu117.zip
unzip ./libtorch-cxx11-abi-shared-with-deps-1.13.1+cu117.zip
rm ./libtorch-cxx11-abi-shared-with-deps-1.13.1+cu117.zip
```
#### Step 2.3: Compile
The lowest g++ version is 7.5.0.
```shell
cd ..
cmake . -B build
cmake --build build --target main --config RelWithDebInfo -j
```
If the mcnerf code is modified, the last two lines should always be executed.
</details>
<details>
<summary><span style="font-weight: bold;"> Install 3D Gaussians Splatting </span></summary>
3DGS has much faster inference speed, higher rendering quality. But the HDR sky is not enabled in this case.
Installing 3DGS requires that your CUDA NVCC version matches your pytorch cuda version.
```bash
# make CUDA (nvcc) version consistent with the pytorch CUDA version.
# first check your CUDA (nvcc) version
nvcc -V # for example: Build cuda_11.8.r11.8
# go to https://pytorch.org/get-started/previous-versions/ to find a corresponding one. The version of pytorch itself should >= 1.13.
Excerpt of 23,245 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:483f71d1453d7cc7, llm:Description: 'Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration'. Topics include 3d, 3d-reconstruction, 3d-vision, autonomous-driving, llm, llm-agent, llm-agents.
matched fp:483f71d1453d7cc7, llm:Description: 'Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration'. Topics include 3d, 3d-reconstruction, 3d-vision, autonomous-driving, llm, llm-agent, llm-agents.
matched fp:483f71d1453d7cc7, llm:Description: 'Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration'. Topics include 3d, 3d-reconstruction, 3d-vision, autonomous-driving, llm, llm-agent, llm-agents.
matched fp:483f71d1453d7cc7, llm:Description: 'Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration'. Topics include 3d, 3d-reconstruction, 3d-vision, autonomous-driving, llm, llm-agent, llm-agents.