Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
| Date | Stars |
|---|---|
| 2026-07-31 | 825 |
| 2026-08-10 | 828 |
| 2026-08-19 | 829 |
| 2026-08-22 | 831 |
| 2026-08-24 | 832 |
| 2026-08-25 | 833 |
| 2026-08-26 | 832 |
| 2026-08-27 | 833 |
| 2026-09-03 | 834 |
| 2026-09-14 | 835 |
| 2026-09-16 | 836 |
| 2026-09-18 | 837 |
| 2026-09-20 | 839 |
| 2026-09-30 | 838 |
| 2026-10-04 | 838 |
| 2026-10-10 | 838 |
| 2026-10-16 | 836 |
| 2026-10-18 | 836 |
| 2026-10-23 | 837 |
| 2026-10-30 | 838 |
| 2026-11-04 | 839 |
Today
+1 stars today
This week
+2 stars this week
This month
+1 stars this month
Momentum
0.0
growth rate 0.24%/day
## VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
#### [[Project Page]](https://voxposer.github.io/) [[Paper]](https://voxposer.github.io/voxposer.pdf) [[Video]](https://www.youtube.com/watch?v=Yvn4eR05A3M)
[Wenlong Huang](https://wenlong.page)<sup>1</sup>, [Chen Wang](https://www.chenwangjeremy.net/)<sup>1</sup>, [Ruohan Zhang](https://ai.stanford.edu/~zharu/)<sup>1</sup>, [Yunzhu Li](https://yunzhuli.github.io/)<sup>1,2</sup>, [Jiajun Wu](https://jiajunwu.com/)<sup>1</sup>, [Li Fei-Fei](https://profiles.stanford.edu/fei-fei-li)<sup>1</sup>
<sup>1</sup>Stanford University, <sup>2</sup>University of Illinois Urbana-Champaign
<img src="media/teaser.gif" width="550">
This is the official demo code for [VoxPoser](https://voxposer.github.io/), a method that uses large language models and vision-language models to zero-shot synthesize trajectories for manipulation tasks.
In this repo, we provide the implementation of VoxPoser in [RLBench](https://sites.google.com/view/rlbench) as its task diversity best resembles our real-world setup. Note that VoxPoser is a zero-shot method that does not require any training data. Therefore, the main purpose of this repo is to provide a demo implementation rather than an evaluation benchmark.
**Note: This codebase currently does not contain the perception pipeline used in our real-world experiments, which produces a real-time mapping from object names to object masks. Instead, it uses the object masks provided as part of RLBench's `get_observation` function. If you are interested in deploying the code on a real robot, you may find more information in the section [Real World Deployment](#real-world-deployment).**
If you find this work useful in your research, please cite using the following BibTeX:
```bibtex
@article{huang2023voxposer,
title={VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models},
author={Huang, Wenlong and Wang, Chen and Zhang, Ruohan and Li, Yunzhu and Wu, Jiajun and Fei-Fei, Li},
journal={arXiv preprint arXiv:2307.05973},
year={2023}
}
```
## Setup Instructions
Note that this codebase is best run with a display. For running in headless mode, refer to the [instructions in RLBench](https://github.com/stepjam/RLBench#running-headless).
- Create a conda environment:
```Shell
conda create -n voxposer-env python=3.9
conda activate voxposer-env
```
- See [Instructions](https://github.com/stepjam/RLBench#install) to install PyRep and RLBench (Note: install these inside the created conda environment).
- Install other dependencies:
```Shell
pip install -r requirements.txt
```
- Obtain an [OpenAI API](https://openai.com/blog/openai-api) key, and put it inside the first cell of the demo notebook.
## Running Demo
Demo code is at `src/playground.ipynb`. Instructions can be found in the notebook.
## Code Structure
Core to VoxPoser:
- **`playground.ipynb`**: Playground for VoxPoser.
- **`LMP.py`**: Implementation of Language Model Programs (LMPs) that recursively generates code to decompose instructions and compose value maps for each sub-task.
- **`interfaces.py`**: Interface that provides necessary APIs for language models (i.e., LMPs) to operate in voxel space and to invoke motion planner.
- **`planners.py`**: Implementation of a greedy planner that plans a trajectory (represented as a series of waypoints) for an entity/movable given a value map.
- **`controllers.py`**: Given a waypoint for an entity/movable, the controller applies (a series of) robot actions to achieve the waypoint.
- **`dynamics_models.py`**: Environment dynamics model for the case where entity/movable is an object or object part. This is used in `controllers.py` to perform MPC.
- **`prompts/rlbench`**: Prompts used by the different Language Model Programs (LMPs) in VoxPoser.
Environment and utilities:
- **`envs`**:
- **`rlbench_env.py`**: Wrapper of RLBench env to expose useful functions for VoxPoser.
- **`tExcerpt of 6,193 characters
Read on GitHub
5
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:fe0f232586b13bd4, topic:robotics, topic:embodied-ai, desc:manipulation
matched fp:fe0f232586b13bd4, topic:large-language-models, topic:foundation-models
matched fp:fe0f232586b13bd4, topic:vision-language-model, readme:vision-language