VoxPoser
Synthesizes 3D voxel affordance and obstacle value maps from language instructions, then solves for collision-free robot trajectories via MPC.
The loop
- Proposes
An LLM writes Python expressions that compose open-vocabulary vision masks into 3D voxel grids of target attractors, obstacles, and rotation/gripper maps.
- Judged by
A deterministic sampling-based Model Predictive Control (MPC) trajectory optimizer and inverse-kinematics solver compute minimum-cost paths over the 3D field.
- Steers
MPC replans at each step from the current observation; the value map comes from cached model-written code.
Headline result
Zero-shot execution of 25 everyday manipulation tasks across real-world and simulated robot arms with 88% success, plus online adaptation to moving obstacles.
What this does not prove
Voxel grid resolution (e.g. 100^3) limits fine sub-millimeter insertion tasks.
How it generalizes the harness
- The algorithm is the solver
- A domain authority judges
- The output outlives the model
- Continuous-discrete duality
The LLM compiles instructions into a 3D potential field while a deterministic MPC trajectory optimizer solves the continuous control problem.
Classification
| What it proposes | Formal specification |
|---|---|
| Generative model | GPT-4 3D potential-field synthesizer |
| Algorithmic authority | Model Predictive Control (MPC) trajectory planner and IK solver |
| Feedback | Score |
| What gets updated | Prompt context |
| When it runs | At inference time |
| Validity | Empirical (measured) |
| Evidence | Experiments reported |
| Lineage | Successor |