Self-Harness
An agent mines its failed runs, edits its own harness, and keeps an edit only if benchmark pass counts do not drop.
The loop
- Proposes
Minimal edits to the agent's harness (its instructions and scaffolding), from the same fixed model
- Judged by
Benchmark verifiers on held-in and held-out splits; an edit passes only if neither split worsens and one improves
- Steers
Failure patterns mined from traces drive the next proposals; accepted edits are merged and rejected ones stay out
Headline result
The paper reports that for three base models (MiniMax M2.5, Qwen3.5-35B-A3B, GLM-5) across three benchmarks (Terminal-Bench-2.0, SWE-bench Verified, AppWorld) every final harness improves both held-in and held-out pass rates, with relative gains of up to 132%. On Terminal-Bench-2.0 held-out, MiniMax M2.5 went 40.5% to 61.9% and Qwen3.5-35B-A3B 23.8% to 38.1%.
What this does not prove
The held-out split also serves as the promotion gate, so it is not an untouched test of generalization. The Terminal-Bench work uses a 64-task subset of 89. The authors say edits may reflect benchmark-specific patterns and that higher-stakes changes need stronger gates. The text reports slightly different figures in two places.
How it generalizes the harness
The product is a harness, which is the thing an agent harness is made of; the loop mostly shows a harness improving itself rather than going beyond one.
Classification
| What it proposes | Code patch |
|---|---|
| Generative model | Fixed base LLM (MiniMax M2.5, Qwen3.5-35B-A3B or GLM-5), no weight updates |
| Algorithmic authority | Benchmark task verifiers and a pass-count acceptance rule |
| Feedback | Test results |
| What gets updated | Prompt context |
| When it runs | Offline, at design time |
| Validity | Empirical (measured) |
| Evidence | Experiments reported |
| Lineage | Original system |