Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[CVPR 2024] LMDrive: Closed-Loop End-to-End Driving with Large Language Models
| Date | Stars |
|---|---|
| 2026-07-31 | 925 |
| 2026-08-05 | 927 |
| 2026-08-06 | 927 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# LMDrive: Closed-Loop End-to-End Driving with Large Language Models
*An end-to-end, closed-loop, language-based autonomous driving framework, which interacts with the dynamic environment via multi-modal multi-view sensor data and natural language instructions.*
[[Project Page](https://hao-shao.com/projects/lmdrive.html)] [[Paper](https://arxiv.org/abs/2312.07488)] [[Dataset(hugging face)](https://huggingface.co/datasets/OpenDILabCommunity/LMDrive)] [[Model Zoo](https://huggingface.co/collections/OpenDILabCommunity/lmdrive-658aee50ce38d143c4925a98)]
[[Dataset(OpenXlab)](https://openxlab.org.cn/datasets/deepcs233/LMDrive)]
[[Model Zoo(OpenXLab)](https://openxlab.org.cn/models/detail/deepcs233/LMDrive)]
[](https://hits.seeyoufarm.com)
[](https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE)
[](https://github.com/tatsu-lab/stanford_alpaca/blob/main/DATA_LICENSE)
## News
- `[02/27]` [LMDrive](https://arxiv.org/abs/2312.07488) is accepted by CVPR 2024 🎉🎉🎉
- `[01/25]` We uploaded our models to [OpenXLab](https://openxlab.org.cn/models/detail/deepcs233/LMDrive)
- `[01/23]` We gave a talk at [ZhiDongXi (智东西)](https://wqpoq.xetslk.com/sl/3D1aRZ)
- `[01/20]` We uploaded our dataset to [OpenXLab](https://openxlab.org.cn/datasets/deepcs233/LMDrive)
- `[12/21]` We released our project website [here](https://hao-shao.com/projects/lmdrive.html)
****
<div align="center">
<img width="800" src="assets/pipeline.png"></img>
</div>
> [Hao Shao](http://hao-shao.com/), Yuxuan Hu, [Letian Wang](https://letianwang0.wixsite.com/myhome), [Steven L. Waslander](https://www.trailab.utias.utoronto.ca/stevenwaslander), [Yu Liu](https://liuyu.us/), [Hongsheng Li](http://www.ee.cuhk.edu.hk/~hsli/).
This repository contains code for the paper [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://arxiv.org/abs/2312.07488). This work proposes a novel language-guided, end-to-end, closed-loop autonomous driving framework.
## Demo Video
<div align="center">
<video width="800" src="https://github.com/opendilab/LMDrive/assets/17512647/65b2785d-e8bc-4ec1-ac86-e077299a465d"></video>
</div>
## Contents
1. [Setup](#setup)
2. [Model Weights](#lmdrive-Weights)
3. [Dataset](#dataset)
1. [Overview](#overview)
1. [Data Generation](#data-generation)
2. [Data Pre-procession](#data-pre-procession)
3. [Data Parsing](#data-parsing)
4. [Training](#training)
1. [Vision encoder pre-training](#vision-encoder-pre-training)
2. [Instruction finetuning](#instruction-finetuning)
5. [Evaluation](#evaluation)
6. [Citation](#citation)
7. [Acknowledgements](#acknowledgements)
## Setup
Our project is built on three parts: (1) vision encoder (corresponding repo: timm); (2) vision LLM (corresponding repo: LAVIS); (3) data collection, agent controller (corresponding repo: InterFuser, Leaderboard, ScenarioRunner).
Install anaconda
```Shell
wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
bash Anaconda3-2020.11-Linux-x86_64.sh
source ~/.bashrc
```
Clone the repo and build the environment
```Shell
git clone https://github.com/opendilab/LMDrive.git
cd LMDrive
conda create -n lmdrive python=3.8
conda activate lmdrive
cd vision_encoder
pip3 install -r requirements.txt
python setup.py develop # if you have installed timm before, please uninstall it
cd ../LAVIS
pip3 install -r requirements.txt
python setup.py develop # if you have installed LAVIS before, please uninstall it
pip install flash-attn --no-build-isolation # optional
```
Download and setup CARLA 0.9.10.1
```Shell
chmod +x setup_carla.sh
./setup_carla.sh
pip iExcerpt of 20,377 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:16753af99eaa8eb9, llm:Repository title and description: "LMDrive: Closed-Loop End-to-End Driving with Large Language Models" (CVPR 2024). Focus is on end-to-end driving using LLMs (autonomous driving / computer vision + agents).
matched fp:16753af99eaa8eb9, llm:Repository title and description: "LMDrive: Closed-Loop End-to-End Driving with Large Language Models" (CVPR 2024). Focus is on end-to-end driving using LLMs (autonomous driving / computer vision + agents).
matched fp:16753af99eaa8eb9, llm:Repository title and description: "LMDrive: Closed-Loop End-to-End Driving with Large Language Models" (CVPR 2024). Focus is on end-to-end driving using LLMs (autonomous driving / computer vision + agents).
matched fp:16753af99eaa8eb9, llm:Repository title and description: "LMDrive: Closed-Loop End-to-End Driving with Large Language Models" (CVPR 2024). Focus is on end-to-end driving using LLMs (autonomous driving / computer vision + agents).
