Eureka
An LLM writes robot reward code, simulated RL scores it, and a text summary of training guides the next rewards.
The loop
- Proposes
Reward-function code written by GPT-4 from the simulator's source code and a task description (16 candidates per iteration)
- Judged by
RL (PPO in Isaac Gym) trains a policy on each reward; a ground-truth fitness function scores it
- Steers
The best reward and a text summary of each reward component during training go back into the prompt
Headline result
Across 29 open-source RL environments with 10 robot morphologies, in simulation, the abstract reports Eureka outperforming expert human-written rewards on 83% of tasks, with an average normalized improvement of 52%. It also used its rewards in a curriculum to get a simulated Shadow Hand to perform pen spinning.
What this does not prove
Results are in simulation, with human reward baselines from the benchmark authors and a fixed fitness function. Searching costs 5 independent runs of 5 iterations with 16 samples each, many RL trainings per task.
How it generalizes the harness
A physics simulator and RL training decide reward quality, and the product is a reward function that stays useful without the LLM.
Classification
| What it proposes | Reward function |
|---|---|
| Generative model | GPT-4 (gpt-4-0314), frozen |
| Algorithmic authority | RL training in Isaac Gym with a ground-truth task fitness function |
| Feedback | Written reflection |
| What gets updated | Prompt context |
| When it runs | Offline, at design time |
| Validity | Empirical (measured) |
| Evidence | Experiments reported |
| Lineage | Original system |