Self-Debugging with Execution Feedback
Iterative program synthesis loop driven by unit-test execution traces, compiler diagnostics, and line-by-line rubber-duck code explanation.
The loop
- Proposes
A code LLM synthesizes a candidate program (Python, C++, or SQL) and generates a line-by-line behavioral explanation.
- Judged by
The language runtime and unit-test suite execute the candidate program and return exact assertion failures, stack traces, or syntax errors.
- Steers
Execution error messages paired with the behavioral trace condition the next repair iteration until all tests pass.
Headline result
Improved pass@1 by up to 12 percentage points across TransCoder C++-to-Python translation, Spider text-to-SQL, and MBPP Python synthesis.
What this does not prove
Correctness is bounded by the unit-test suite available during the debugging loop.
How it generalizes the harness
Deterministic compiler and unit-test execution traces gate termination and steer iterative code repair.
Classification
| What it proposes | Program |
|---|---|
| Generative model | Codex / PaLM-2 / StarCoder |
| Algorithmic authority | Interpreter runtime and unit-test execution oracle |
| Feedback | Test results |
| What gets updated | Prompt context |
| When it runs | At inference time |
| Validity | Empirical (measured) |
| Evidence | Experiments reported |
| Lineage | Original system |