Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Official repo of VLABench, a large scale benchmark designed for fairly evaluating VLA, Embodied Agent, and VLMs.
| Date | Stars |
|---|---|
| 2026-07-31 | 456 |
| 2026-08-06 | 460 |
Today
+4 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks
### 🎓 [**Paper**](docs/pdf/paper.pdf) | 🌐 [**Project Website**](https://vlabench.github.io/) ｜ 🤗 [**Hugging Face**](https://huggingface.co/VLABench) | 🐳 [**Quick Start with Docker**](./QuickStart.md)
<img src="docs/images/Figure1_overview.png" width="100%" />
<span style="font-size:16px"> 🚨 <span style="color:#AB4459;">**NOTICE:**</span>Please feel free to start an issue or create a PR! If I do not respond to issues in a timely manner, feel free to send me an email directly. I will do my best to build a more user-friendly community ecosystem for VLABench within this year.</span>
## News
* **Preview** A complete infra framework will be open-sourced alongside new work, including full training pipelines, VLABench evaluation, a new leaderboard, data processing, and real-device deployment. Stay tuned!
* **2025/11/10** Upload several new checkpoints as baselines. Action representation matters.
- [Pi05-ft-primitive](https://huggingface.co/VLABench/pi05-primitive-10task): Track 1 SR 40.6%.
- [Pi0-ft-primitive(10 tasks)](https://huggingface.co/VLABench/pi0-primitive-10task): Track 1 SR 47%.
- [Pifast-ft-primitive(10 tasks)(relative chunk)](https://huggingface.co/VLABench/pi0-fast-primitive-10task): Track 1 SR 29.1%. This is the official transform.
- [Pifast-ft-primitive(10 tasks)(delta chunk)](https://huggingface.co/VLABench/pi0-fast-ft-primitive-10task-deltachunk): Track 1 SR 51.2%. This is the aligned transform.
* **2025/8/06** Update VLABench with:
- parrallel data collection;
- parrallel evaluation example;
- camera augmentation;
- rlds/lerobot data format ft dataset on [hf](https://huggingface.co/VLABench);
- update on [pi0 codebase](https://github.com/Shiduo-zh/openpi).
* **2025/6/26** VLABench has been accepted by ICCV 2025.
* **2025/4/10** Releasing the finetuned pi0 checkpoint(pi0-base and pi0-fast) on [hf](https://huggingface.co/VLABench).
* **2025/3/25** Releasing standard evaluation episodes and primitive task finetune dataset.
* **2025/2/26** Releasing referenced evaluation pipeline.
* **2025/2/14** Releasing the scripts for trajectory generation.
* **2024/12/25** The preview verison of VLABench has been released! The preview version showcases most of the designed tasks and structure, but the functionalities are still being managed and tested.
## Recent Work Todo
- [x] Organize the functional code sections.
- [x] Reconstruct the efficient, user-friendly, and comprehensive evaluation framework.
- [x] Manage the automatic data workflow for existing tasks.
- [x] Improve the DSL of skill libarary.
- [x] Release the trejectory and evaluation scripts.
- [x] Test the interface of humanoid and dual-arm manipulation.
- [x] Release the left few tasks not released in preview version.
- [ ] Leaderboard of VLAs and VLMs in the standard evaluation
- [x] Release standard evaluation datasets/episodes, in different dimension and difficulty level.
- [x] Release standard finetune dataset.
- [ ] Integrate the commonly used VLA models for facilitate replication. (Continously update)
- [] Releasing more datasets, including pretraining version, composite tasks once finalizing testing.
- [] Releasing more checkpoints.
- [] Update the complete training and evaluation community.
## Installation
### Install VLABench
1. Prepare conda environment
```sh
conda create -n vlabench python=3.10
conda activate vlabench
git clone https://github.com/OpenMOSS/VLABench.git
cd VLABench
pip install -r requirements.txt
pip install -e .
```
2. Download the assets
```sh
python scripts/download_assets.py
```
3. (Option) Initialize submodules
```sh
git submodule update --init --recursive
```
This will update other policies repos such openpi.
The script will automatically download the necessary assets and unzip them into the correct directory.
## Data Collection
###Excerpt of 12,102 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a012dd4010b4a1c3, llm:Repository topics: embodied, foundation-models, robotics; description: 'Official repo of VLABench, a large scale benchmark designed for fairly evaluating VLA, Embodied Agent, and VLMs.'
matched fp:a012dd4010b4a1c3, llm:Repository topics: embodied, foundation-models, robotics; description: 'Official repo of VLABench, a large scale benchmark designed for fairly evaluating VLA, Embodied Agent, and VLMs.'
matched fp:a012dd4010b4a1c3, llm:Repository topics: embodied, foundation-models, robotics; description: 'Official repo of VLABench, a large scale benchmark designed for fairly evaluating VLA, Embodied Agent, and VLMs.'
matched fp:a012dd4010b4a1c3, llm:Repository topics: embodied, foundation-models, robotics; description: 'Official repo of VLABench, a large scale benchmark designed for fairly evaluating VLA, Embodied Agent, and VLMs.'