The library

40 AlgoGens. Each shows its loop, its headline result and what it does not prove.

A-CEoH

Algorithmic Contextual EoH

Karlsruhe Institute of Technology and TU Dortmund University, 2026 Optimization · Program evolution

Proposes: Guiding heuristic functions for A* search, as thought plus code
Judged by: Running A* with the heuristic on instances (solved count, fitness, time)
Steers: Fitness drives EoH selection; the A* source code is kept in every prompt

Headline result

For unit-load pre-marshalling, Qwen2.5-Coder:32b-generated heuristics solved 30 of 30 held-out instances (seeds 10-39) against 27 of 30 for the "Optimal A*" baseline. For the sliding puzzle they solved 10 of 10 on seeds 0-9 against 5 of 10 for Hybrid A* and for "Optimal A*".

What it does not prove

Two domains. Where both solve, the LLM heuristics give worse solutions than A* (puzzle fitness 0.281 against 0.252 on shared instances). The paper gives no numeric EoH baseline for the final heuristics, only a qualitative claim of improvement. Accepted at EvoStar.

Active learning for conditional crystal inverse design

The University of Hong Kong, 2025 Materials · Active-learning generation

Proposes: Alloy crystal structures from Con-CDVAE conditioned on a target bulk modulus
Judged by: Three screens: a CGCNN pre-filter, molecular dynamics with MACE-MP-0 (within 20%, stable), then DFT
Steers: Structures passing DFT, with computed bulk moduli, join the training set and the generator is fine-tuned

Headline result

For a 350 GPa bulk modulus target, in a range sparse in the training data, three active learning iterations lowered the mean absolute percentage error of generated crystals from about 0.4 to 0.2 (CGCNN-evaluated). DFT bulk moduli of new structures averaged 343.54 GPa (standard deviation 27.20) after the first iteration and 356.52 GPa (standard deviation 18.61) after the second.

What it does not prove

The error trend is measured with the same kind of GNN used in the screen, and DFT is a calculation rather than a lab result. It is a case study on one property (bulk modulus) of alloys, and no structure was synthesized.

AlphaEvolve

Google DeepMind, 2025 Mathematics · Program evolution

Proposes: Edits to an existing program (algorithms, heuristics, constructions) written by an ensemble of Gemini models
Judged by: One or more user-supplied automated evaluators that run each program and return scores
Steers: Higher-scoring programs are kept in a program database (MAP-Elites plus island populations) and sampled into later prompts

Headline result

Applied to over 50 open math problems, AlphaEvolve matched the best known construction in about 75% of cases and found a better one in about 20%, including a kissing-number configuration of 593 spheres in 11 dimensions (previous record 592). It found a procedure to multiply two 4x4 complex-valued matrices with 48 scalar multiplications (Strassen's recursive method gives 49). At Google it recovered on average 0.7% of fleet-wide compute through a scheduling heuristic and gave a 23% average kernel speedup, about 1% less Gemini training time. Deployment figures as reported by Google DeepMind.

What it does not prove

The 48-multiplication result holds for complex-valued matrices (characteristic 0 fields), not as a general replacement for Strassen. Industry results are developer-reported and not independently replicated. Evaluators must give a machine-checkable score, and a later study found the system sometimes exploited weak evaluators.

AlphaEvolve at scale on 67 math problems

Georgiev, Gomez-Serrano, Tao, Wagner

Google DeepMind, Brown University, Institute for Advanced Study and UCLA (Georgiev, Gomez-Serrano, Tao, Wagner), 2025 Mathematics · Program evolution

Proposes: Search heuristics (programs) that look for a mathematical construction, or programs that give a formula for any n
Judged by: User-written deterministic evaluators that score the best object found, sometimes checked afterwards by proof tools
Steers: Best heuristics are reused in later rounds; Kakeya's program went to proof tools

Headline result

The paper reports 67 problems across analysis, combinatorics, geometry and number theory. AlphaEvolve rediscovered the best known solutions in most cases and found improved ones in several, for example an autocorrelation upper bound reduced from 1.5708 to 1.5032 and a finite field Kakeya construction of size 1/4 p^3 + 7/8 p^2 - 1/8 for primes p = 1 mod 4.

What it does not prove

The authors state it did not match previous results in all cases, struggles without a smooth score, and some gains could have been matched by traditional methods. It exploited weak verifiers in some problems (for example numerical integration in Problem 6.6), and results depended on expert prompt advice.

AlphaGeometry

Google DeepMind, 2024 Proofs · Proof-state search

Proposes: Auxiliary constructions (new points and lines) for an Olympiad geometry problem
Judged by: A symbolic deduction engine that derives all consequences and decides whether the goal is proved
Steers: The enlarged figure goes back to the language model for the next construction, under beam search

Headline result

In the paper's evaluation on a test set of 30 olympiad-level geometry problems (IMO-AG-30), AlphaGeometry solves 25, versus ten for the earlier Wu's method baseline and 18 for the strongest baseline (DD + AR + human-designed heuristics). Symbolic deduction with algebra (DD + AR) alone solves 14.

What it does not prove

The 30-problem set is geometry only, translated into the system's own formal language. The language model is trained once on synthetic data before the search runs; it is not retrained by the search results. A later study (Sinha et al., arXiv:2404.06405) reports that Wu's method alone solves 15 of the same 30 problems, so the ten-problem baseline is contested.

AlphaGeometry2

Google DeepMind, 2025 Proofs · Proof-state search

Proposes: Auxiliary constructions for geometry problems, from a Gemini-architecture language model
Judged by: An improved symbolic deduction engine
Steers: Parallel search trees share facts (SKEST), so one tree's findings guide the others

Headline result

Solves 42 of 50 IMO geometry problems from 2000 to 2024 (IMO-AG-50); the paper states an overall solving rate of 84% on geometry problems from the last 25 years versus 54% for the original, and says AG2 surpasses an average gold medalist. The formal language now covers 88% of those problems, up from 66%.

What it does not prove

Geometry only. The paper reports progress toward, not completion of, a fully automated pipeline from natural-language problem statements. The AG2 system was also part of the IMO 2024 silver-medal-standard result.

AlphaProof

Google DeepMind, 2025 Proofs · Proof-state search

Proposes: Lean proofs, built step by step, for millions of auto-formalized problems
Judged by: The Lean proof assistant, which verifies every proof
Steers: Verified outcomes train the model; for hard problems, test-time RL learns from millions of related variants

Headline result

At IMO 2024 the system with AlphaProof as its core engine solved three of the five non-geometry problems, including the hardest one; combined with AlphaGeometry 2 this reached a score equivalent to a silver medallist, with multi-day computation.

What it does not prove

Problems were formalized into Lean before the system worked on them, and the paper reports multi-day computation, so this is far from a competition-time-limit result. The Nature paper's claim is the RL method; the medal-level score is the combined system.

AlphaVerus

Carnegie Mellon University, 2024 Code verification · Verifier-guided repair

Proposes: Verus translations of programs (from Dafny) and repaired versions, from a Llama 3.1 70B model
Judged by: The Verus verifier, plus a filter that rejects translations whose specification or program is misaligned
Steers: Treefinement tree search fixes errors from verifier feedback; verified, filtered programs become prompt examples

Headline result

With Llama 3.1 70B and no model fine-tuning, verified-code generation reaches 32.9% on HumanEval-Verus and 65.7% on MBPP-verified (pass@256), versus 27.1% and 35.9% for GPT-4o and 11.8% and 26.9% for Llama 3.1 70B.

What it does not prove

HumanEval-Verus and MBPP-verified are small, hand-adapted benchmarks, and the task is generating code plus proofs from specifications. Verification proves code matches the specification, so the misalignment filter matters.

ATLANTIS

Team Atlanta, AIxCC winner

Team Atlanta, 2025 Security · Verifier-guided repair

Proposes: Fuzz seeds, payload generator scripts and source patches, from LLM agents
Judged by: Running the code: coverage, sanitizer crashes as proof of a bug, patch checks against crash and tests
Steers: Coverage feedback refines scripts (up to four rounds); failed patches guide the next try; seeds join a corpus

Headline result

Won the DARPA AIxCC final at DEF CON 33 (August 2025) with 392.76 points; the team's paper reports 43 of 70 vulnerabilities found. DARPA's overall figures (all teams): 54 of 63 synthetic vulnerabilities found and 43 patched, 18 real-world bugs found, about $152 average per task.

What it does not prove

Scores and counts come from the team's own paper plus DARPA's page, which differ (70 versus 63 challenges after a DARPA correction). The competition used synthetic bugs inserted into open-source projects. Real-world finds are small in number.

Augmented Hill-Climb

University of Cambridge, 2022 Chemistry · Oracle-trained generation

Proposes: SMILES strings from an RNN (also tested on a transformer) molecule generator
Judged by: A scoring function evaluated on each sampled batch; the paper's tasks include docking objectives, with diversity filters
Steers: RL updates the generator with a hybrid of REINVENT's loss and Hill-Climb's top-scorer selection

Headline result

On the authors' benchmark tasks, compared with REINVENT, optimization ability improved about 1.5-fold and sample efficiency about 45-fold, and Augmented Hill-Climb outperformed the other reinforcement learning strategies tested on six tasks.

What it does not prove

The gains are measured on the authors' benchmark tasks and docking objectives, not on lab-measured activity. It is a training rule inside the REINVENT-style setup rather than a separate system.

AutoVerus

Microsoft Research, University of Illinois Urbana-Champaign, Columbia University and UC Irvine, 2024 Code verification · Verifier-guided repair

Proposes: Proof annotations (loop invariants, assertions) for Rust code with Verus specifications, from GPT-4o agents
Judged by: The Verus verifier, which either proves the specification or returns errors
Steers: Three phases: initial proof, refinement with generic tips, then agents that fix the specific errors Verus reports

Headline result

Proves 137 of 150 tasks (91.3%) on Verus-Bench, with at most 3 tries per task and GPT-4o; 122 were proved on the first attempt. A baseline that repeatedly calls GPT-4o for up to 10 minutes proved 67 of 150. More than half were done in under 30 seconds or 3 LLM calls.

What it does not prove

Verus-Bench is a 150-task suite the authors built from existing code and verification benchmarks, and the task gives the code and the specification. The tool proves what the spec says; a wrong spec would still pass.

BFS-Prover

ByteDance Seed, 2025 Proofs · Proof-state search

Proposes: Lean 4 tactics for the current proof state, from a language model
Judged by: The Lean 4 compiler through LeanDojo, which runs each tactic and reports success or an error
Steers: Best-first search ranks states by log probability; expert iteration retrains on proofs (SFT) and error pairs (DPO)

Headline result

Reaches 72.95% on the miniF2F test set as an accumulative score; the single-run figure is 70.83% +/- 0.89 at a budget of 2048 x 2 x 600 (Lean 4.7.0). The authors say this shows plain best-first search can compete when scaled.

What it does not prove

The 72.95% is a cumulative (union across runs and settings) score, not one fixed-budget run. Its claims date from early 2025 and are compared with provers that used different budgets.

CALM

Co-evolution of Algorithms and Language Model

City University of Hong Kong, Southeast University, University of Victoria, Hon Hai Research Institute, 2025 Optimization · Program evolution

Proposes: Heuristic code from a small LLM inside an evolutionary search
Judged by: Running the heuristic on problem instances to get a quality score that becomes the RL reward
Steers: Verbal guidance through prompts plus numerical guidance through GRPO fine-tuning of the LLM weights

Headline result

A Qwen2.5-7B-Instruct model (INT4, on a single 24GB GPU) with GRPO fine-tuning reached an average 0.71% gap on online bin packing, against 0.89% for MCTS-AHD and 2.40% for Best Fit, and beat the best listed competitor on CVRP (3.95% vs 4.70%) and OP (12.58% vs 16.34%) at N=200 in the paper's tables.

What it does not prove

The advantage is not uniform: on TSP at N=50 MCTS-AHD (9.69%) beats CALM (10.04%). Numbers come from the paper's own tables and baselines, which mostly ran on GPT-4o-mini.

CEoH

Contextual Evolution of Heuristics

Karlsruhe Institute of Technology and TU Dortmund University, 2025 Optimization · Program evolution

Proposes: Tree-search scoring heuristics for the unit-load pre-marshalling problem, as thought plus code
Judged by: Running the heuristic and measuring the average relative gap of its move count to a lower bound
Steers: Fitness drives EoH selection; the added problem description is kept in every prompt

Headline result

For unit-load pre-marshalling, Qwen2.5-Coder:32b with CEoH reached a fitness of 12.5% (best overall), against 14.3% for the best EoH run (GPT-4o 2024-08-06). The "Optimal A*" baseline scored about 8.1%. CEoH outperformed EoH for Gemma2:27b, Qwen2.5-Coder:32b and DeepSeekV3:685b.

What it does not prove

One niche problem. On unseen instances fitness rose to 30.8 (Qwen CEoH) and 33.9 (GPT-4o EoH), which the authors read as overfitting. For GPT-4o EoH was better than CEoH, and none solved the hardest case (80% fill).

ChemSpaceAL

Yale University (Batista group), 2023 Chemistry · Active-learning generation

Proposes: SMILES strings from a GPT-based molecular generator (100,000 unique molecules per iteration)
Judged by: About 1,000 molecules sampled across clusters are docked with DiffDock and scored by contact-based interactions
Steers: Clusters are weighted by mean score into a training set, and the GPT is fine-tuned on it

Headline result

In the paper's computational study on c-Abl kinase, the model learned to generate molecules similar to FDA-approved inhibitors without being shown them and reproduced two exactly (imatinib and bosutinib after five iterations). The method was also applied to the HNH domain of Cas9, which has no commercial small-molecule inhibitors.

What it does not prove

The judge is a learned docking model (DiffDock) plus a simple contact-count score, so this is predicted binding only. Matching known drugs is a retrospective sanity check, and no molecule was tested in a lab.

CogFunSearch

Google DeepMind with Janelia Research Campus, Rockefeller University, Max Planck Institute for Biological Cybernetics, Princeton University, UCL and Columbia University, 2025 Science · Program evolution

Proposes: Python programs that predict an animal's next choice from its past choices and rewards
Judged by: Scoring each program on how well it predicts recorded rat choices
Steers: The best-scoring programs are fed back into the prompt, as in FunSearch

Headline result

In a bioRxiv preprint, on rat choices in a 2-armed drifting bandit task (data from Miller et al. 2021), CogFunSearch reliably found programs that beat the best existing cognitive model the authors compared against, both when improving supplied programs and when starting from scratch.

What it does not prove

A simple unconstrained neural network predicted significantly better than both the classic models and the discovered programs, so interpretability is gained at some cost in accuracy. Preprint, one task family.

CoupleEvo

Karlsruhe Institute of Technology, 2026 Optimization · Program evolution

Proposes: Heuristics for each sub-problem, written by an LLM inside an evolutionary loop
Judged by: Running the combined heuristics on instances of the coupled problem
Steers: Sequential, iterative or integrated schedules decide which sub-problem's heuristic evolves in each generation

Headline result

On two coupled problems (inventory routing, and multi-robot multi-bay pre-marshalling) with Qwen3-coder:30b, the sequential and iterative strategies converged more stably than the integrated one. On inventory routing the mean relative gap was 0.0429 (sequential), 0.0433 (iterative), 0.0537 (integrated).

What it does not prove

Two problems and one model. The paper's own text says iterative has the smallest median on one problem while its table shows sequential lower, so read the tables, not the prose. Accepted to a GECCO 2026 workshop.

Darwin Godel Machine

Sakana AI and University of British Columbia (Jeff Clune lab), 2025 ML systems · Program evolution

Proposes: A modified version of the agent's own code (tools, prompts, workflows), written by a frozen foundation model
Judged by: Running the new agent on coding benchmarks (SWE-bench Verified, Polyglot) and recording pass rates
Steers: Parents come from an archive of agents, weighted by score and few children; children join it

Headline result

Over 80 iterations, the best agent rose from 20.0% to 50.0% on SWE-bench (Verified) and from 14.2% to 30.7% on Polyglot (abstract figures; the paper's Section 4.4 gives 14.0% as the Polyglot start).

What it does not prove

It stays below the best closed-source systems on SWE-bench and is limited by the underlying model. The run took about two weeks with significant API cost. The authors report that the agent sometimes faked test results, and that when rewarded for avoiding this it sometimes removed the detection markers (per Sakana's blog). Archive maintenance and parent selection are fixed, not self-modified.

DeepSeek-Prover-V1.5 with RMaxTS

DeepSeek, 2024 Proofs · Proof-state search

Proposes: Lean 4 proofs and proof steps from a language model
Judged by: The Lean 4 proof assistant, which accepts or rejects each proof and each step
Steers: Lean outcomes are the RL reward; at inference, RMaxTS tree search rewards new states to pick branches

Headline result

With RL plus RMaxTS at a sample budget of 32 x 6400 (mixture of two guiding prompts) it reaches 63.5% on miniF2F-test; on the ProofNet test set it reaches 25.3% at a budget of 4 x 6400. The paper calls both new state of the art at publication.

What it does not prove

Numbers depend heavily on sample budget: the same model gets 50.0% to 54.9% at 32 to 3200 samples without search. The ProofNet validation set was used for expert iteration, so the test column is the clean one.

DrEureka

University of Pennsylvania, NVIDIA and UT Austin, 2024 Robotics · Program evolution

Proposes: Reward code (as in Eureka, with safety instructions) and then domain randomization configurations from GPT-4
Judged by: RL training in simulation, plus a physics-prior step (RAPP) that bounds each randomized parameter
Steers: Reward stage: best result feeds back into the prompt. Randomization stage: open loop, no real-world feedback

Headline result

On a real quadruped forward-locomotion test (2 m/s target, 5 m track), the best DrEureka policy reached 1.83 m/s and 5.00 m versus 1.32 m/s and 4.17 m for the human-designed setup. The robot also walked atop a real yoga ball for 15.4 seconds on average.

What it does not prove

Only the reward stage closes a loop with simulation results, and the randomization stage is a one-shot proposal. The real-world numbers come from short hardware trials, and the yoga-ball runs are about 15 seconds, not minutes.

EoH

Evolution of Heuristics

City University of Hong Kong, Huawei Noah's Ark Lab, Southern University of Science and Technology, 2024 Optimization · Program evolution

Proposes: Heuristics as a natural-language 'thought' plus code, made by five LLM prompt operators
Judged by: Running the code on benchmark instances and measuring fitness
Steers: Fitness drives selection of parents for the next generation of a fixed-size population

Headline result

On online bin packing, EoH with about 2,000 LLM queries (GPT-3.5-turbo) averaged a 1.18% gap to the lower bound over six test sets, against 2.23% for FunSearch (which the paper says needs around 1 million queries), 4.13% for best fit and 4.51% for first fit. It also reached the best-known tour on three TSPLib instances (pr124, kroA150, u159) and led on Taillard flow-shop sets.

What it does not prove

Main experiments use GPT-3.5-turbo on three classic problems. Comparisons are on the authors' own set-ups, and FunSearch's query count is quoted from its published runs, not re-run.

Eureka

NVIDIA, UPenn, Caltech, UT Austin, 2023 Robotics · Program evolution

Proposes: Reward-function code written by GPT-4 from the simulator's source code and a task description (16 candidates per iteration)
Judged by: RL (PPO in Isaac Gym) trains a policy on each reward; a ground-truth fitness function scores it
Steers: The best reward and a text summary of each reward component during training go back into the prompt

Headline result

Across 29 open-source RL environments with 10 robot morphologies, in simulation, the abstract reports Eureka outperforming expert human-written rewards on 83% of tasks, with an average normalized improvement of 52%. It also used its rewards in a curriculum to get a simulated Shadow Hand to perform pen spinning.

What it does not prove

Results are in simulation, with human reward baselines from the benchmark authors and a fixed fitness function. Searching costs 5 independent runs of 5 iterations with 16 samples each, many RL trainings per task.

FunSearch

Google DeepMind, 2023 Mathematics · Program evolution

Proposes: Python functions (inside a fixed skeleton) that construct cap sets or pack bins
Judged by: An automated evaluator that runs each program and scores its output
Steers: Best programs are shown in later prompts; islands of programs are periodically reseeded

Headline result

On the cap set problem, FunSearch found a cap set of size 512 in 8 dimensions, larger than any previously known, and raised the lower bound on the asymptotic cap set capacity to 2.2202 (previous best 2.2180). On online bin packing it evolved heuristics that beat first fit and best fit on the OR-Library sets OR1-OR4 and on Weibull instances.

What it does not prove

Results are for two problem families with a cheap, graded score. The paper notes that the method needs such a score and that sampling is costly. The public repo is a reference implementation only: it has no LLM, no sandbox and no distributed system.

Goedel-Prover

Princeton Language and Intelligence (Princeton University), with Tsinghua University, Amazon, Meta FAIR and Numina, 2025 Proofs · Oracle-trained generation

Proposes: Lean 4 proofs for 1.64 million formalized statements
Judged by: The Lean compiler, which checks each candidate proof
Steers: Verified proofs join the training set and the next, stronger prover is trained on them (expert iteration)

Headline result

Goedel-Prover-SFT reaches 57.6% Pass@32 on miniF2F, 7.6 points above DeepSeek-Prover-V1.5-RL; the paper's abstract reports over 60% Pass@32 after RL including DPO, 7 PutnamBench problems at Pass@512, and proofs for 29.7K Lean Workbook problems versus 15.7K for prior provers.

What it does not prove

Its best-result claims are dated April 2025 and use Pass@32 or Pass@512, so budgets differ from some rivals. The paper text says 8 iterations while a table lists Iter-9. The abstract does not describe the compiler signal beyond verification.

HSEvo

Hanoi University of Science and Technology and George Mason University, 2024 Optimization · Program evolution

Proposes: Heuristic code from LLM crossover and mutation, plus harmony-search refinement of numeric parameters
Judged by: Running the heuristic on benchmark instances for an objective value, plus two diversity metrics
Steers: Objective and diversity scores steer selection to balance exploration and convergence

Headline result

The authors propose two diversity metrics and use them to compare FunSearch, EoH and ReEvo. They report EoH has higher diversity but unstable scores, ReEvo has good scores but poor diversity, and that HSEvo reaches high diversity and good objective scores while staying cost-effective.

What it does not prove

The abstract gives no numeric gains, and results are on the authors' black-box heuristic-design benchmarks. 'Cost-effective' is the authors' wording.

InvDesFlow-AL

Renmin University of China, 2025 Materials · Active-learning generation

Proposes: Crystal structures from a diffusion model (EGNN backbone) pretrained on Alex-MP-20 and GNoME crystals
Judged by: Relaxation with DPA-2, then a committee score on formation energy and Ehull or on superconductor Tc
Steers: The top-scored generated crystals become the next fine-tuning set, repeated over multiple rounds

Headline result

Generated average formation energy fell over five iterations (means -1.14, -2.03, -2.93, -3.56, -3.77; units not stated in the text). The authors report 1,598,551 materials with Ehull below 50 meV, and Li2AuH6 as a conventional superconductor with a transition temperature of about 140 K from first-principles electron-phonon calculations. Crystal structure prediction RMSE is 0.0423 angstrom, reported as a 32.96% improvement over existing generative models.

What it does not prove

The paper gives two counts for low-Ehull materials (1,598,551 in text, 1,610,600 in a figure caption) and does not say how Ehull is queried. The 140 K value is a calculation, not a measurement, and not every evaluation is DFT.

Lib-INVENT with docking and RFXscore

Sanofi, 2022 Chemistry · Oracle-trained generation

Proposes: Decorations (substituents) added to a factor Xa scaffold, written as SMILES by Lib-INVENT, a REINVENT variant
Judged by: Glide XP docking, ligand efficiency, the RFXscore rescoring model, QED and a factor Xa QSAR model
Steers: Reinforcement learning updates the generator's agent network; a diversity filter penalises repeats

Headline result

In a factor Xa case study, docking followed by RFXscore gave the best results, especially when diverse structures fitting a binding site were wanted; RFXscore plus the factor Xa model left the most compounds after MM/GBSA and other filters. RFXscore had a test r2 of 0.494 and MAE of 1.086.

What it does not prove

MM/GBSA was applied afterwards as an independent check, not as a training reward. Everything is computed binding, with no lab measurements, and the target is a single well-studied protein (factor Xa).

LLM-SR

Virginia Tech, Carnegie Mellon University and Allen Institute for AI, 2025 Science · Program evolution

Proposes: Equation skeletons as Python functions with placeholder parameters
Judged by: Fitting parameters with BFGS or Adam and scoring negative mean squared error against data
Steers: An experience buffer of 10 islands feeds the best equations (k=2) into the next prompt

Headline result

Across four problems (two oscillators, E. coli growth, aluminum stress-strain) with about 2.5K iterations, LLM-SR beat symbolic regression baselines on normalized MSE, most clearly out of domain: for E. coli growth 0.0037 against 1.0141 for the best baseline.

What it does not prove

Four benchmarks, two of them custom-built. Baselines (e.g. PySR) ran with far more evaluations (2M+). Results use GPT-3.5-turbo or Mixtral-8x7B; the best model differs by problem.

OpenEvolve

algorithmicsuperintelligence (open-source community project), 2025 Optimization · Program evolution

Proposes: Code changes in marked EVOLVE-BLOCK regions, written by an ensemble of LLMs from any compatible API
Judged by: A user-written evaluator that scores each program; errors and other evaluator output are kept as artifacts
Steers: Scores place programs in a MAP-Elites grid across island populations; errors and evaluator output feed the next prompt

Headline result

The project reports 2-3x speedups on GPU kernels (MLX Metal) and a circle packing for n=26 that its README calls state of the art, plus sorting and signal-processing examples. These are the project's own claims; the README gives no numeric circle-packing score.

What it does not prove

All results are project-reported and not independently replicated here. The README gives no numbers for most examples. It is an engineering framework, not a paper.

OSS-Fuzz-Gen

LLM fuzz target generation

Google (OSS-Fuzz), 2023 Security · Verifier-guided repair

Proposes: Fuzz targets (C/C++ entry points) for functions that Fuzz Introspector flags as under-fuzzed
Judged by: OSS-Fuzz build, a check that the target calls the function, a crash-on-start check, and new line coverage
Steers: Build errors and a not-calling-the-function error are fed back in a code-fixer prompt; repeated rounds fix errors

Headline result

Google's page reports 14 of 31 tested OSS-Fuzz projects compiled new targets and increased coverage (gains 0 to 31%; tinyxml2 line coverage 38% to 69%), and a similar target in OpenSSL rediscovered CVE-2022-3602.

What it does not prove

The page calls this a preliminary investigation on a small set of C/C++ benchmarks with an unnamed Google model. The 14 are the projects that succeeded and the selection of the 31 is not fully explained. Crash and coverage results are checks; the page does not say they feed back into the model.

ReEvo

Reflective Evolution

Peking University, KAIST, Southeast University, Singapore Management University (AI4CO), 2024 Optimization · Program evolution

Proposes: Heuristic code produced by LLM crossover and elitist mutation
Judged by: Running the heuristic on problem instances to get an objective value
Steers: Short-term reflection compares parents; long-term reflection accumulates hints; both feed later prompts ('verbal gradient')

Headline result

Across six combinatorial problems (TSP, CVRP, OP, MKP, BPP, DPP), with a budget of 100 heuristic evaluations and GPT-3.5 Turbo, ReEvo beats EoH on sample efficiency in five problems, and designed heuristics improved neural solvers, for example POMO's TSP1000 gap from about 52% to 29%. Its genetic algorithm for decap placement scored 12.98 vs 12.41 for the expert-designed one.

What it does not prove

Authors' benchmark set-ups with GPT-3.5 Turbo as the main model; 'state of the art' refers to the compared methods in the paper.

ReEvo-designed memetic algorithm for secure fluid antenna systems

Harbin Institute of Technology, Shenzhen, 2026 Engineering · Program evolution

Proposes: Crossover, mutation and local-search operator code for fluid antenna port selection, written by DeepSeek Reasoner
Judged by: Best secure sum-rate an inner genetic algorithm reaches with the operators on training channels, via secure WMMSE
Steers: ReEvo's reflective evolution feeds performance comparisons back into the next round of operator code

Headline result

At equal generation counts (100 generations), the evolved memetic algorithm reached 11.423 bit/s/Hz at 1 W (7.6% over a conventional GA) and 23.611 bit/s/Hz at 5 W (8.2% over a conventional GA), and about 4 to 6% above the two LLM-assisted GA baselines.

What it does not prove

Equal generations are not equal evaluation budgets, because local search adds evaluations, and error bars overlap with the LLM-GA baselines at some operating points. Results are simulated (10 users, one eavesdropper, 20x20 grid) in a preprint accepted by WCSP 2026.

REINVENT 4

with DockStream docking

AstraZeneca (Molecular AI), 2024 Chemistry · Oracle-trained generation

Proposes: SMILES strings for new small molecules, sampled from an RNN or transformer generator
Judged by: A user-defined scoring function; here a DockStream docking score (Glide) plus drug-likeness (QED)
Steers: Reinforcement learning, run as staged (curriculum) learning, raises the probability of high-scoring molecules in the generator

Headline result

In the paper's demonstration on PDK1 (PDB 2XCH), 50 epochs of staged learning with batch size 128 produced 119 hits out of 6,400 generated molecules (1.9%), where a hit means a docking score of -8 kcal/mol or better and QED of 0.7 or higher; the hits spanned 103 generic scaffolds.

What it does not prove

The PDK1 run is described by the authors as a hypothetical virtual screening exercise, and a good docking score is a computed estimate, not measured activity. Docking is one optional scoring component; DockStream alone is only a docking wrapper, not an AlgoGen, and its repository is no longer maintained.

ReProver

LeanDojo

LeanDojo team (Kaiyu Yang et al.), 2023 Proofs · Proof-state search

Proposes: Lean tactics, from a language model that first retrieves relevant premises from the math library
Judged by: The Lean environment, driven through LeanDojo, which runs each tactic and reports the new state or an error
Steers: Best-first search expands the most promising proof states and feeds each new state back to the tactic generator

Headline result

On the LeanDojo Benchmark test set, pass@1 is 51.2% (random split) and 26.3% (novel_premises split) with retrieval, versus 47.6% and 23.2% without retrieval and 29.0% and 7.4% for zero-shot GPT-4. Also 26.5% on miniF2F and 13.8% on ProofNet (pass@1, 10 minute wall-time limit per theorem).

What it does not prove

Main experiments use Lean 3 (Lean 4 results are in an appendix not covered here). The tactic generator is trained once, offline, on extracted proofs; search results do not update its weights. The paper's claim is about open, cheap training (one GPU week), not top accuracy.

Sakana AI CUDA Engineer

robust-kbench pipeline

Sakana AI, 2025 ML systems · Program evolution

Proposes: CUDA kernels written by an ensemble of frontier LLMs
Judged by: Compile and numeric check against PyTorch, LLM pre-filters, then GPU timing and profiling
Steers: Up to five fast, correct past kernels plus a profiling summary go into the next prompt

Headline result

On 200 Level 1 and 2 KernelBench tasks the average speedup was 3.13x, falling to 1.49x after excluding contaminated tasks; 95% of tasks translated successfully. On twelve robust-kbench tasks forward kernels reached up to 2.5x over PyTorch eager.

What it does not prove

The earlier Feb 2025 release claimed 10-100x and Sakana acknowledged the system exploited evaluation code (a memory-reuse flaw), and walked those claims back. In this paper two kernels with 51x and 123x speedups came from benchmark loopholes. Part of the filtering uses LLM verifiers, which are not exact judges. Cost was about $5 per kernel.

Seed-Prover

ByteDance Seed (AI4Math team), 2025 Proofs · Verifier-guided repair

Proposes: Lean proofs built from lemmas, from a language model
Judged by: The Lean proof assistant
Steers: Lean feedback, proved lemmas and self-summaries of earlier attempts go into the next try, with three test-time strategies

Headline result

The abstract reports 78.1% of formalized past IMO problems proved, MiniF2F saturated, above 50% on PutnamBench, and 5 of 6 IMO 2025 problems fully proved; a separate Seed-Geometry engine handles geometry.

What it does not prove

This card rests on the abstract only. Budgets, training details and who formalized the IMO 2025 statements are not stated there.

Self-Harness

Shanghai Artificial Intelligence Laboratory, 2026 ML systems · Verifier-guided repair

Proposes: Minimal edits to the agent's harness (its instructions and scaffolding), from the same fixed model
Judged by: Benchmark verifiers on held-in and held-out splits; an edit passes only if neither split worsens and one improves
Steers: Failure patterns mined from traces drive the next proposals; accepted edits are merged and rejected ones stay out

Headline result

The paper reports that for three base models (MiniMax M2.5, Qwen3.5-35B-A3B, GLM-5) across three benchmarks (Terminal-Bench-2.0, SWE-bench Verified, AppWorld) every final harness improves both held-in and held-out pass rates, with relative gains of up to 132%. On Terminal-Bench-2.0 held-out, MiniMax M2.5 went 40.5% to 61.9% and Qwen3.5-35B-A3B 23.8% to 38.1%.

What it does not prove

The held-out split also serves as the promotion gate, so it is not an untouched test of generalization. The Terminal-Bench work uses a 64-task subset of 89. The authors say edits may reflect benchmark-specific patterns and that higher-stakes changes need stronger gates. The text reports slightly different figures in two places.

ShinkaEvolve

Sakana AI, 2025 Optimization · Program evolution

Proposes: Diff edits, rewrites or crossovers of programs, written by an ensemble of LLMs
Judged by: Task-specific evaluators that score each program and return text feedback
Steers: Parents mix score and novelty across islands; an embedding filter rejects repeats; a UCB1 bandit picks the LLM

Headline result

ShinkaEvolve found a circle-packing (n=26) solution with sum of radii 2.63598 under its relaxed verifier, beating AlphaEvolve's reported solution within 150 program evaluations. It also improved ALE-Agent solutions on ALE-Bench by about 2.3% on average over 10 tasks, designed an AIME agent scaffold (gpt-4.1-nano, 7 LLM calls), and found a new mixture-of-experts load-balancing loss.

What it does not prove

The paper uses both '150 samples' and '150 generations' for the circle-packing run. The MoE loss was evolved at small scale (556M parameters, 2B tokens) and the authors note limited budget and similar architecture between search and test. AIME gains were smaller on 2023 problems, which the authors link to possible contamination.

T-GADE

Osaka Metropolitan University, 2026 Optimization · Program evolution

Proposes: Structured artifacts (here, bin packing scoring heuristics) from LLM crossover, mutation and a self-repair step
Judged by: Evaluation on online bin packing, measured as excess bins over the volume lower bound
Steers: Survival selection minimizes free energy F_T = <E> - T*H, with temperature T controlling the quality-diversity balance

Headline result

In a preregistered seed-paired comparison at 400 total LLM calls on online bin packing, the HoxLM configuration won 9 of 10 runs against EoH, with a median sealed excess of 0.836% against 3.912%.

What it does not prove

Preprint (arXiv v2, 14 Oct 2026), one task, 10 seed pairs, and an EoH-aligned implementation. Not independently replicated.

T3 Planner

Jia Li et al., 2025 Robotics · Verifier-guided repair

Proposes: Subgoal waypoints, timed waypoints and a Python controller script, written by an LLM in three stages
Judged by: Signal Temporal Logic robustness checks, a Python syntax check, and simulation of the controller on annotated cases
Steers: A failed output goes back to the LLM as the previous failed result, until all checks pass

Headline result

In simulated household, chip-design and navigation scenarios, success rates with GPT-4o were 91.65%, 91.60% and 83.69%, against 56.67%, 61.29% and 50.00% for AutoTAMP with GPT-4o. The authors claim a 30 to 40% improvement.

What it does not prove

All scenarios are in a simulated 2D environment with no real-robot tests, and the paper gives no retry limit. The checker is a temporal-logic verifier, not model predictive control, and this is a single preprint with no venue listed.