Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
PyTorch implementation for the paper "Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving"
| Date | Stars |
|---|---|
| 2026-07-31 | 578 |
| 2026-08-06 | 578 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
https://github.com/user-attachments/assets/82a1993e-5948-4f5a-ad9b-849a21fe9a14
This is the PyTorch implementation for inference and training of the LLM-Driver
described in:
> **Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving**
>
>[Long Chen](https://long.ooo/), [Oleg Sinavski](https://uk.linkedin.com/in/oleg-sinavski), [Jan Hünermann](https://uk.linkedin.com/in/janhuenermann), [Alice Karnsund](https://uk.linkedin.com/in/alice-karnsund), [Andrew James Willmott](https://uk.linkedin.com/in/andrew-willmott-2ba18510), [Danny Birch](https://uk.linkedin.com/in/dannybirch), [Daniel Maund](https://uk.linkedin.com/in/danmaund), [Jamie Shotton](https://jamie.shotton.org/)
>
> *ICRA 2024* </br>
>[[preprint]](https://github.com/wayveai/Driving-with-LLMs/blob/main/assets/preprint.pdf) [[arxiv]](https://arxiv.org/abs/2310.01957)
<p align="center">
<img src="assets/main.png" alt="LLM-Driver" width="500px">
<br/> The LLM-Driver utilises object-level vector input from our driving simulator to predict explanable actions using pretrained Language Models, providing a robust and interpretable solution for autonomous driving.
<img src="assets/main.gif" alt="LLM-Driver">
<br/> The LLM-Driver running in open-loop prediction using the vector inputs (top-left BEV view), with the results of action prediction (steering angles and acceleration/brake pedals), action justification (captions on the rendered video), Driving Question Answering (table at the bottom).
</p>
## News
- **`[2024/01/29]`** Thrilled to share that our paper has been accepted by [ICRA 2024](https://2024.ieee-icra.org/)!
- **`[2023/12/21]`** Please checkout our follow-up work `LingoQA`: [[code]](https://github.com/wayveai/LingoQA) [[arxiv]](https://arxiv.org/abs/2312.14115)
- **`[2023/10/03]`** The paper is now avaliable on [[arxiv]](https://arxiv.org/abs/2310.01957)
- **`[2023/07/06]`** The paper and code have been made available under the [paper_code](https://github.com/wayveai/Driving-with-LLMs/tree/paper_code) branch for anonymous submission.
## Getting Started
### Prerequisites
- Python 3.x
- pip
- Minimum of 20GB VRAM for running evaluations
- Minimum of 40GB VRAM for training (default setting)
### ⚙ Setup
1. **Set up a virtual environment (tested with Python 3.8-3.11)**
```sh
python3 -m venv env
source env/bin/activate
```
2. **Install required dependencies**
```sh
pip install -r requirements.txt.lock
```
Note: `requirements.txt.lock` is generated with `pip-compile` from original `requirements.txt` for reproducibility.
3. **Set up WandB API key**
Set up your [WandB](https://wandb.ai/) API key for training and evaluation logging.
```sh
export WANDB_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```
### 💿 Dataset
- **Training/testing data**:
The datasets have already been checked into the codebase. To unarchive them, use the following commands:
```
tar -xzvf data/vqa_train_10k.tar.gz -C data/
tar -xzvf data/vqa_test_1k.tar.gz -C data/
```
- **Re-collect DrivingQA data**:
While the training and evaluation datasets already include pre-collected DrivingQA data, we also offer a script that illustrates how to collect DrivingQA data using the OpenAI ChatGPT API. If you wish to re-collect the DrivingQA data, simplely run the following command with your OpenAI API key:
```sh
python scripts/collect_vqa.py -i data/vqa_test_1k.pkl -o output_folder/ --openai_api xxxxxxxx
```
### 🏄 Evaluation
1. **Evaluate for Perception and Action Prediction**
Run the following command:
```sh
python train.py \
--mode eval \
--resume_from_checkpoint models/weights/stage2_with_pretrained/ \
--data_path data/vqa_train_10k.pkl \
--val_data_path data/vqa_test_1k.pkl \
--eval_items caption,action \
--vqa
```
2. **Evaluate for DrivingQA**
Run the following Excerpt of 6,664 characters
Read on GitHub
10
8
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:950bc3a4f7513d3f, desc:autonomous driving