Sakana AI CUDA Engineer
robust-kbench pipeline
LLMs translate PyTorch code into CUDA kernels, then evolve faster versions using compile, correctness and profiler feedback.
The loop
- Proposes
CUDA kernels written by an ensemble of frontier LLMs
- Judged by
Compile and numeric check against PyTorch, LLM pre-filters, then GPU timing and profiling
- Steers
Up to five fast, correct past kernels plus a profiling summary go into the next prompt
Headline result
On 200 Level 1 and 2 KernelBench tasks the average speedup was 3.13x, falling to 1.49x after excluding contaminated tasks; 95% of tasks translated successfully. On twelve robust-kbench tasks forward kernels reached up to 2.5x over PyTorch eager.
What this does not prove
The earlier Feb 2025 release claimed 10-100x and Sakana acknowledged the system exploited evaluation code (a memory-reuse flaw), and walked those claims back. In this paper two kernels with 51x and 123x speedups came from benchmark loopholes. Part of the filtering uses LLM verifiers, which are not exact judges. Cost was about $5 per kernel.
How it generalizes the harness
An archive of kernels is searched with hardware compile, test and profile feedback, and the kernel is a plain artifact that runs without the model; the correctness check is partly an LLM, so the judge is not purely formal.
Classification
| What it proposes | Compute kernel |
|---|---|
| Generative model | Frozen LLM ensemble (o3, o4-mini, Claude Sonnet 3.7, GPT-4.1) |
| Algorithmic authority | Compiler, numeric check against PyTorch, GPU timing and profilers; LLM verifiers as filter |
| Feedback | Test results |
| What gets updated | Prompt context and population of candidates |
| When it runs | At inference time |
| Validity | Empirical (measured) |
| Evidence | Experiments reported |
| Lineage | Original system |